2008/9/12 Aaron Griffin <aaronmgriffin@gmail.com>:
On Fri, Sep 12, 2008 at 3:23 PM, Dusty Phillips <buchuki@gmail.com> wrote:
2008/9/12 Eric Belanger <belanger@astro.umontreal.ca>:
Hi,
I don't know if you remember but a while ago a huge part of extra i686 (IIRC it was all packages from L to Z) were orphaned and erroneouly showing up as recently updated on the web site. This just happened again with packages in extra x86_64. I don't know what could caused that but it's very annoying as we has to readopt all our packages back.
Fuck.
I remember Judd telling me not to swear at users but its ok to swear at scripts right?
This has to be happening in reporead.py. Fucking reporead.py. To the best of my knowledge, no other script updates the web database in anyway, am I wrong?
The actual db_update script splits the packages into those that are in the database and those that are not and processes them separately. Packages that are not currently in the database get added as orphans because apparently its hard to interrogate the maintainer from the db.tar.gz. At first, I assumed that it is doing an add when it should be doing an update, which would add new packages with orphan maintainer. But this doesn't appear to be the case because there are not currently any duplicate x86_64 packages (that aren't in testing).
My second more likely hypothesis is race conditions. I don't know how the db scripts update exactly, but I suspect reporead is reading a db.tar.gz file that is either broken or not yet fully uploaded. It sees this broken db file and drops all the packages in the web interface that are not in that file. Then x minutes later (crontab), it runs again on a proper db and sees the missing packages again. It adds them to the database and sets the maintainer to orphan.
Are such broken dbs possible/likely/happening? If its a race condition, we need to put a lock on the database (maybe dbtools does this already) so that reporead isn't accessing it at the same time as dbtools. If its just that when the database gets updated it sometimes breaks the database well.. that just needs to be fixed.
Hmmm, the DBs are constructed in /tmp and then moved live to /home/ftp/whatever it's possible that reporead may be opening it mid-move, but that doesn't seem right. It's gzipped. Wouldn't that balk if you took half of a DB file, and tried to gunzip it?
I think so... not sure if this is a proper test of it but it fails: dusty:x86_64 $ head -c 10000 extra.db.tar.gz | tar -xz gzip: stdin: unexpected end of file tar: Unexpected EOF in archive tar: Unexpected EOF in archive tar: Error is not recoverable: exiting now reporead does some great stuff with logger (debug and info). Do you know if any of those logged messages are saved? I haven't checked this time, but IIRC last time, it was all packages after the letter L that got orphaned or something. This indicates that for some reason reporead is not processing all the packages in the file. Either the db does not contain all the files because a half full db got uploaded or it is reading part of the db and then exiting for some reason. Why either of these would occur is beyond me. Dusty