[aur-dev] AUR slow queries, stale database columns, etc.
LocationID, DummyPkg- are these used anymore on the Packages table? If not, getting rid of these would do wonders toward actually allowing MySQL to use indexes for some queries, which it has been unable to do for a long time. I've pasted some of the slow query log analysis below from stuff this month, it is pretty obvious where work should be focused. Killing these two fields and then replacing the SQL_CALC_FOUND_ROWS with a second count query should take care of nearly every slow query that currently happens. Also potentially worth attacking is the use of GREATEST(SubmittedTS,ModifiedTS) - since the wonderful MySQL has no functional indexes using this in a query as an order by clause results in full table scans. -Dan $ mysqldumpslow -s c sigurd-slow.log | head -n 100 Reading mysql slow query log from sigurd-slow.log Count: 3550 Time=2.08s (7390s) Lock=0.00s (3s) Rows=26.7 (94644), aur[aur]@localhost SELECT SQL_CALC_FOUND_ROWS Users.Username AS Maintainer, PackageCategories.Category, PackageLocations.Location, Packages.Name, Packages.Version, Packages.Description, Packages.NumVotes, Packages.ID, Packages.OutOfDate FROM Packages LEFT JOIN Users ON (Packages.MaintainerUID = Users.ID) LEFT JOIN PackageCategories ON (Packages.CategoryID = PackageCategories.ID) LEFT JOIN PackageLocations ON (Packages.LocationID = PackageLocations.ID) WHERE Packages.DummyPkg = N ORDER BY NumVotes DESC, Name ASC, CategoryID DESC LIMIT N, N Count: 2275 Time=1.38s (3143s) Lock=0.00s (0s) Rows=25.4 (57680), aur[aur]@localhost SELECT SQL_CALC_FOUND_ROWS CommentNotify.UserID AS Notify, PackageVotes.UsersID AS Voted, Users.Username AS Maintainer, PackageCategories.Category, PackageLocations.Location, Packages.Name, Packages.Version, Packages.Description, Packages.NumVotes, Packages.ID, Packages.OutOfDate FROM Packages LEFT JOIN Users ON (Packages.MaintainerUID = Users.ID) LEFT JOIN PackageVotes ON (Packages.ID = PackageVotes.PackageID AND PackageVotes.UsersID = N) LEFT JOIN CommentNotify ON (Packages.ID = CommentNotify.PkgID AND CommentNotify.UserID = N) LEFT JOIN PackageCategories ON (Packages.CategoryID = PackageCategories.ID) LEFT JOIN PackageLocations ON (Packages.LocationID = PackageLocations.ID) WHERE Packages.DummyPkg = N ORDER BY NumVotes DESC, Name ASC, CategoryID DESC LIMIT N, N Count: 350 Time=1.62s (566s) Lock=0.01s (1s) Rows=27.4 (9591), aur[aur]@localhost SELECT SQL_CALC_FOUND_ROWS Users.Username AS Maintainer, PackageCategories.Category, PackageLocations.Location, Packages.Name, Packages.Version, Packages.Description, Packages.NumVotes, Packages.ID, Packages.OutOfDate FROM Packages LEFT JOIN Users ON (Packages.MaintainerUID = Users.ID) LEFT JOIN PackageCategories ON (Packages.CategoryID = PackageCategories.ID) LEFT JOIN PackageLocations ON (Packages.LocationID = PackageLocations.ID) WHERE Packages.DummyPkg = N ORDER BY NumVotes ASC, Name ASC, CategoryID DESC LIMIT N, N Count: 307 Time=1.20s (368s) Lock=0.00s (0s) Rows=65.8 (20200), aur[aur]@localhost SELECT SQL_CALC_FOUND_ROWS CommentNotify.UserID AS Notify, PackageVotes.UsersID AS Voted, Users.Username AS Maintainer, PackageCategories.Category, PackageLocations.Location, Packages.Name, Packages.Version, Packages.Description, Packages.NumVotes, Packages.ID, Packages.OutOfDate FROM Packages LEFT JOIN Users ON (Packages.MaintainerUID = Users.ID) LEFT JOIN PackageVotes ON (Packages.ID = PackageVotes.PackageID AND PackageVotes.UsersID = N) LEFT JOIN CommentNotify ON (Packages.ID = CommentNotify.PkgID AND CommentNotify.UserID = N) LEFT JOIN PackageCategories ON (Packages.CategoryID = PackageCategories.ID) LEFT JOIN PackageLocations ON (Packages.LocationID = PackageLocations.ID) WHERE Packages.DummyPkg = N ORDER BY GREATEST(SubmittedTS,ModifiedTS) DESC, Name ASC, LocationID ASC LIMIT N, N Count: 220 Time=1.76s (386s) Lock=0.00s (0s) Rows=29.9 (6585), aur[aur]@localhost SELECT SQL_CALC_FOUND_ROWS Users.Username AS Maintainer, PackageCategories.Category, PackageLocations.Location, Packages.Name, Packages.Version, Packages.Description, Packages.NumVotes, Packages.ID, Packages.OutOfDate FROM Packages LEFT JOIN Users ON (Packages.MaintainerUID = Users.ID) LEFT JOIN PackageCategories ON (Packages.CategoryID = PackageCategories.ID) LEFT JOIN PackageLocations ON (Packages.LocationID = PackageLocations.ID) WHERE Packages.DummyPkg = N ORDER BY Name DESC, LocationID ASC, CategoryID DESC LIMIT N, N Count: 218 Time=1.31s (286s) Lock=0.00s (0s) Rows=28.1 (6122), aur[aur]@localhost SELECT SQL_CALC_FOUND_ROWS Users.Username AS Maintainer, PackageCategories.Category, PackageLocations.Location, Packages.Name, Packages.Version, Packages.Description, Packages.NumVotes, Packages.ID, Packages.OutOfDate FROM Packages LEFT JOIN Users ON (Packages.MaintainerUID = Users.ID) LEFT JOIN PackageCategories ON (Packages.CategoryID = PackageCategories.ID) LEFT JOIN PackageLocations ON (Packages.LocationID = PackageLocations.ID) WHERE Packages.DummyPkg = N ORDER BY Name ASC, LocationID ASC, CategoryID DESC LIMIT N, N Count: 187 Time=2.21s (413s) Lock=0.00s (0s) Rows=26.9 (5034), aur[aur]@localhost SELECT SQL_CALC_FOUND_ROWS Users.Username AS Maintainer, PackageCategories.Category, PackageLocations.Location, Packages.Name, Packages.Version, Packages.Description, Packages.NumVotes, Packages.ID, Packages.OutOfDate FROM Packages LEFT JOIN Users ON (Packages.MaintainerUID = Users.ID) LEFT JOIN PackageCategories ON (Packages.CategoryID = PackageCategories.ID) LEFT JOIN PackageLocations ON (Packages.LocationID = PackageLocations.ID) WHERE Packages.DummyPkg = N ORDER BY LocationID ASC, Name ASC, CategoryID DESC LIMIT N, N
We don't need this anymore since all packages managed here are well...managed here. Rip out all of the places we were using this field, many of which depended on the magic value '2' anyway. On the display side of things, we had a column that was always showing 'unsupported' that is now gone, and you can no longer sort by this column. Signed-off-by: Dan McGee <dan@archlinux.org> --- Completely untested, but might as well put up or shut up as far as providing code to clean things up. Let me know what I can do to get this included. I may have time to test it in the next few days locally. -Dan UPGRADING | 3 +- scripts/cleanup | 2 +- support/schema/aur-schema.sql | 20 +------ support/schema/gendummydata.py | 106 +++++++++------------------------- web/html/pkgsubmit.php | 5 +- web/lib/aurjson.class.php | 2 +- web/lib/pkgfuncs.inc | 77 ++++--------------------- web/lib/stats.inc | 6 +- web/template/pkg_details.php | 22 +++---- web/template/pkg_search_form.php | 2 +- web/template/pkg_search_results.php | 4 - 11 files changed, 61 insertions(+), 188 deletions(-) diff --git a/UPGRADING b/UPGRADING index 2d3434f..61b1462 100644 --- a/UPGRADING +++ b/UPGRADING @@ -9,7 +9,8 @@ From 1.7.0 to 1.8.0 ---- ALTER TABLE Packages ADD OutOfDateTS BIGINT UNSIGNED NULL DEFAULT NULL; UPDATE Packages SET OutOfDateTS = UNIX_TIMESTAMP() WHERE OutOfDate = 1; -ALTER TABLE Packages DROP OutOfDate, DROP URLPath; +ALTER TABLE Packages DROP OutOfDate, DROP URLPath, LocationID; +DROP TABLE PackageLocations; ---- 2. You will need to update all packages which are stored in the incoming dir as diff --git a/scripts/cleanup b/scripts/cleanup index 21c913a..4fc9ea2 100755 --- a/scripts/cleanup +++ b/scripts/cleanup @@ -25,7 +25,7 @@ exec('ls ' . INCOMING_DIR, $files); $count = 0; foreach ($files as $pkgname) { - if (package_location($pkgname) != 'unsupported') { + if (!package_exists($pkgname)) { echo 'Removing ' . INCOMING_DIR . "$pkgname\n"; system('rm -r ' . INCOMING_DIR . $pkgname); $count++; diff --git a/support/schema/aur-schema.sql b/support/schema/aur-schema.sql index cd24f38..6af8167 100644 --- a/support/schema/aur-schema.sql +++ b/support/schema/aur-schema.sql @@ -1,5 +1,5 @@ -- The MySQL database layout for the AUR. Certain data --- is also included such as AccountTypes, PackageLocations, etc. +-- is also included such as AccountTypes, etc. -- DROP DATABASE AUR; CREATE DATABASE AUR; @@ -89,21 +89,6 @@ INSERT INTO PackageCategories (Category) VALUES ('x11'); INSERT INTO PackageCategories (Category) VALUES ('xfce'); --- The various repositories that a package could live in. --- -CREATE TABLE PackageLocations ( - ID TINYINT UNSIGNED NOT NULL AUTO_INCREMENT, - Location CHAR(32) NOT NULL, - PRIMARY KEY (ID) -); -INSERT INTO PackageLocations (Location) VALUES ('none'); -INSERT INTO PackageLocations (Location) VALUES ('unsupported'); -INSERT INTO PackageLocations (Location) VALUES ('community'); -INSERT INTO PackageLocations (Location) VALUES ('current'); -INSERT INTO PackageLocations (Location) VALUES ('extra'); -INSERT INTO PackageLocations (Location) VALUES ('unstable'); - - -- Information about the actual packages -- CREATE TABLE Packages ( @@ -116,7 +101,6 @@ CREATE TABLE Packages ( DummyPkg TINYINT UNSIGNED NOT NULL DEFAULT 0, -- 1=>dummy FSPath CHAR(255) NOT NULL DEFAULT '', License CHAR(40) NOT NULL DEFAULT '', - LocationID TINYINT UNSIGNED NOT NULL DEFAULT 1, NumVotes INTEGER UNSIGNED NOT NULL DEFAULT 0, OutOfDateTS BIGINT UNSIGNED NULL DEFAULT NULL, SubmittedTS BIGINT UNSIGNED NOT NULL, @@ -127,13 +111,11 @@ CREATE TABLE Packages ( PRIMARY KEY (ID), UNIQUE (Name), INDEX (CategoryID), - INDEX (LocationID), INDEX (DummyPkg), INDEX (NumVotes), INDEX (SubmitterUID), INDEX (MaintainerUID), FOREIGN KEY (CategoryID) REFERENCES PackageCategories(ID) ON DELETE NO ACTION, - FOREIGN KEY (LocationID) REFERENCES PackageLocations(ID) ON DELETE NO ACTION, FOREIGN KEY (SubmitterUID) REFERENCES Users(ID) ON DELETE NO ACTION, FOREIGN KEY (MaintainerUID) REFERENCES Users(ID) ON DELETE NO ACTION ); diff --git a/support/schema/gendummydata.py b/support/schema/gendummydata.py index 99f8195..a03b3e6 100755 --- a/support/schema/gendummydata.py +++ b/support/schema/gendummydata.py @@ -85,28 +85,21 @@ esc = db.escape_string # seen_users = {} seen_pkgs = {} -locations = {} categories = {} -location_keys = [] category_keys = [] user_keys = [] # some functions to generate random data # -def genVersion(location_id=0): +def genVersion(): ver = [] ver.append("%d" % random.randrange(0,10)) ver.append("%d" % random.randrange(0,20)) if random.randrange(0,2) == 0: ver.append("%d" % random.randrange(0,100)) - if location_id == 2: # the package is in the AUR - return ".".join(ver) + "-u%d" % random.randrange(1,11) - else: - return ".".join(ver) + "%d" % random.randrange(1,11) + return ".".join(ver) + "-u%d" % random.randrange(1,11) def genCategory(): return categories[category_keys[random.randrange(0,len(category_keys))]] -def genLocation(): - return locations[location_keys[random.randrange(0,len(location_keys))]] def genUID(): return seen_users[user_keys[random.randrange(0,len(user_keys))]] @@ -162,7 +155,7 @@ contents = None # Load package categories from database # -if DBUG: print "Loading package categories/locations..." +if DBUG: print "Loading package categories..." q = "SELECT * FROM PackageCategories" dbc.execute(q) row = dbc.fetchone() @@ -171,16 +164,6 @@ while row: row = dbc.fetchone() category_keys = categories.keys() -# Load package locations from database -# -q = "SELECT * FROM PackageLocations" -dbc.execute(q) -row = dbc.fetchone() -while row: - locations[row[1]] = row[0] - row = dbc.fetchone() -location_keys = locations.keys() - # done with the database # dbc.close() @@ -243,14 +226,10 @@ for p in seen_pkgs.keys(): if count % 20 == 0: # every so often, there are orphans... muid = 0 - location_id = genLocation() - if location_id == 1: # unsupported pkgs don't have a maintainer - muid = 0 - uuid = genUID() # the submitter/user - s = "INSERT INTO Packages (ID, Name, Version, CategoryID, LocationID, SubmittedTS, SubmitterUID, MaintainerUID) VALUES (%d, '%s', '%s', %d, %d, %d, %d, %d);\n" % (seen_pkgs[p], p, genVersion(location_id), - genCategory(), location_id, NOW, uuid, muid) + s = "INSERT INTO Packages (ID, Name, Version, CategoryID, SubmittedTS, SubmitterUID, MaintainerUID) VALUES (%d, '%s', '%s', %d, %d, %d, %d);\n" % (seen_pkgs[p], p, genVersion(), + genCategory(), NOW, uuid, muid) out.write(s) if count % 100 == 0: if DBUG: print ".", @@ -265,63 +244,34 @@ for p in seen_pkgs.keys(): s = "INSERT INTO PackageComments (PackageID, UsersID, Comments, CommentTS) VALUES (%d, %d, '%s', %d);\n" % (seen_pkgs[p], genUID(), fortune, now) out.write(s) - if location_id == 1: # Unsupported - just a PKGBUILD and maybe other stuff - others = random.randrange(0,3) - s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/PKGBUILD" % p, + others = random.randrange(0,3) + s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/PKGBUILD" % p, + random.randrange(0,999)) + out.write(s) + if others == 0: + s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/%s.patch" % (p,p), random.randrange(0,999)) out.write(s) - if others == 0: - s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/%s.patch" % (p,p), - random.randrange(0,999)) - out.write(s) - elif others == 1: - s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/%s.patch" % (p,p), - random.randrange(0,999)) - out.write(s) - s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/arch.patch" % p, - random.randrange(0,999)) - out.write(s) + elif others == 1: + s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/%s.patch" % (p,p), + random.randrange(0,999)) + out.write(s) + s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/arch.patch" % p, + random.randrange(0,999)) + out.write(s) - elif others == 2: - s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/%s.patch" % (p,p), - random.randrange(0,999)) - out.write(s) - s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/arch.patch" % p, - random.randrange(0,999)) - out.write(s) - s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/%s.install" % (p,p), - random.randrange(0,999)) - out.write(s) + elif others == 2: + s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/%s.patch" % (p,p), + random.randrange(0,999)) + out.write(s) + s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/arch.patch" % p, + random.randrange(0,999)) + out.write(s) + s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/%s.install" % (p,p), + random.randrange(0,999)) + out.write(s) - else: - # Create package contents - # - num_files = random.randrange(PKG_FILES[0], PKG_FILES[1]) - files = {} - for f in range(num_files): - loc = RANDOM_PATHS[random.randrange(len(RANDOM_PATHS))] - if "lib" in loc: - path = loc + "/lib" + p + ".so" - elif "man" in loc: - path = loc + "/" + p + "." + loc[-1] + ".gz" - elif "share" in loc: - path = loc + "/" + p + "/sounds/" + p + ".wav" - elif "profile" in loc: - path = loc + "/" + p + ".sh" - elif "rc.d" in loc: - path = loc + "/" + p - elif "etc" in loc: - path = loc + "/" + p + ".conf" - elif "opt" in loc: - path = loc + "/" + p + "/bin/" + p - else: - path = loc + "/" + p - if not files.has_key(path): - files[path] = 1 - s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], path, - random.randrange(0,99999999)) - out.write(s) if DBUG: print "." # Cast votes diff --git a/web/html/pkgsubmit.php b/web/html/pkgsubmit.php index c39e2f9..3d77f72 100644 --- a/web/html/pkgsubmit.php +++ b/web/html/pkgsubmit.php @@ -277,7 +277,7 @@ if ($_COOKIE["AURSID"]): } # Update package data - $q = sprintf("UPDATE Packages SET ModifiedTS = UNIX_TIMESTAMP(), Name = '%s', Version = '%s-%s', License = '%s', Description = '%s', URL = '%s', LocationID = 2, FSPath = '%s', OutOfDateTS = NULL WHERE ID = %d", + $q = sprintf("UPDATE Packages SET ModifiedTS = UNIX_TIMESTAMP(), Name = '%s', Version = '%s-%s', License = '%s', Description = '%s', URL = '%s', FSPath = '%s', OutOfDateTS = NULL WHERE ID = %d", mysql_real_escape_string($new_pkgbuild['pkgname']), mysql_real_escape_string($new_pkgbuild['pkgver']), mysql_real_escape_string($new_pkgbuild['pkgrel']), @@ -322,7 +322,7 @@ if ($_COOKIE["AURSID"]): $uid = uid_from_sid($_COOKIE["AURSID"]); # This is a brand new package - $q = sprintf("INSERT INTO Packages (Name, License, Version, CategoryID, Description, URL, LocationID, SubmittedTS, SubmitterUID, MaintainerUID, FSPath) VALUES ('%s', '%s', '%s-%s', %d, '%s', '%s', 2, UNIX_TIMESTAMP(), %d, %d, '%s')", + $q = sprintf("INSERT INTO Packages (Name, License, Version, CategoryID, Description, URL, SubmittedTS, SubmitterUID, MaintainerUID, FSPath) VALUES ('%s', '%s', '%s-%s', %d, '%s', '%s', UNIX_TIMESTAMP(), %d, %d, '%s')", mysql_real_escape_string($new_pkgbuild['pkgname']), mysql_real_escape_string($new_pkgbuild['license']), mysql_real_escape_string($new_pkgbuild['pkgver']), @@ -398,7 +398,6 @@ html_header("Submit"); if (ini_get("file_uploads")): $pkg_categories = pkgCategories(); - $pkg_locations = pkgLocations(); ?> <form action='pkgsubmit.php' method='post' enctype='multipart/form-data'> diff --git a/web/lib/aurjson.class.php b/web/lib/aurjson.class.php index 0868cb2..285aad7 100644 --- a/web/lib/aurjson.class.php +++ b/web/lib/aurjson.class.php @@ -29,7 +29,7 @@ class AurJSON { $this->fields = array( 'Packages.ID', 'Name', 'Version', 'CategoryID', - 'Description', 'LocationID', 'URL', 'CONCAT("' . + 'Description', 'URL', 'CONCAT("' . mysql_real_escape_string(URL_DIR) . '", Name, "/", Name, ".tar.gz") AS URLPath', 'License', 'NumVotes', '(OutOfDateTS IS NOT NULL) AS OutOfDate' diff --git a/web/lib/pkgfuncs.inc b/web/lib/pkgfuncs.inc index c701348..66331d6 100644 --- a/web/lib/pkgfuncs.inc +++ b/web/lib/pkgfuncs.inc @@ -80,37 +80,6 @@ function pkgCategories() { return $cats; } -# grab the current list of PackageLocations -# -function pkgLocations() { - $locs = array(); - $dbh = db_connect(); - $q = "SELECT * FROM PackageLocations WHERE ID != 1 AND ID < 4 "; - $q.= "ORDER BY Location ASC"; - $result = db_query($q, $dbh); - if ($result) { - while ($row = mysql_fetch_row($result)) { - $locs[$row[0]] = $row[1]; - } - } - return $locs; -} - -# Return the repository name for a particular package. -function package_location($name="") { - if (!$name) {return NULL;} - $dbh = db_connect(); - $q = "SELECT PackageLocations.Location FROM Packages "; - $q.= "LEFT JOIN PackageLocations ON "; - $q.= "Packages.LocationID = PackageLocations.ID "; - $q.= "WHERE Name = '".mysql_real_escape_string($name)."' "; - $q.= "AND DummyPkg = 0"; - $result = db_query($q, $dbh); - if (!$result) {return NULL;} - $row = mysql_fetch_row($result); - return $row[0]; -} - # check to see if the package name exists # function package_exists($name="") { @@ -324,9 +293,8 @@ function package_details($id=0, $SID="") { $atype = account_from_sid($SID); $uid = uid_from_sid($SID); - $q = "SELECT Packages.*,Location,Category "; - $q.= "FROM Packages,PackageLocations,PackageCategories "; - $q.= "WHERE Packages.LocationID = PackageLocations.ID "; + $q = "SELECT Packages.*,Category "; + $q.= "FROM Packages,PackageCategories "; $q.= "AND Packages.CategoryID = PackageCategories.ID "; $q.= "AND Packages.ID = " . intval($id); $dbh = db_connect(); @@ -373,15 +341,13 @@ function package_details($id=0, $SID="") { * request vars: * O - starting result number * PP - number of search hits per page - * L - package location ID number * C - package category ID number * K - package search string * SO - search hit sort order: * values: a - ascending * d - descending * SB - sort search hits by: - * values: l - package location - * c - package category + * values: c - package category * n - package name * v - number of votes * m - maintainer username @@ -415,8 +381,6 @@ function pkg_search_page($SID="") { // grab info for user if they're logged in if ($SID) $myuid = uid_from_sid($SID); - // get a list of package locations - $locs = pkgLocations(); // get a list of package categories $cats = pkgCategories(); //meow @@ -455,7 +419,6 @@ function pkg_search_page($SID="") { } $q .= "Users.Username AS Maintainer, PackageCategories.Category, - PackageLocations.Location, Packages.Name, Packages.Version, Packages.Description, Packages.NumVotes, Packages.ID, Packages.OutOfDateTS @@ -469,15 +432,10 @@ function pkg_search_page($SID="") { } $q .= "LEFT JOIN PackageCategories ON (Packages.CategoryID = PackageCategories.ID) - LEFT JOIN PackageLocations - ON (Packages.LocationID = PackageLocations.ID) WHERE Packages.DummyPkg = 0 "; - // TODO: possibly do string matching on category and - // location to make request variable values more sensible - if (intval($_GET["L"])) { - $q .= "AND Packages.LocationID = ".intval($_GET["L"])." "; - } + // TODO: possibly do string matching on category + // to make request variable values more sensible if (intval($_GET["C"])) { $q.= "AND Packages.CategoryID = ".intval($_GET["C"])." "; } @@ -515,16 +473,12 @@ function pkg_search_page($SID="") { $order = $_GET["SO"] == 'd' ? 'DESC' : 'ASC'; - $q_sort = "ORDER BY Name ".$order.", LocationID ASC, CategoryID DESC "; + $q_sort = "ORDER BY Name ".$order.", CategoryID DESC "; switch ($_GET["SB"]) { case 'c': - $q_sort = "ORDER BY CategoryID ".$order.", Name ASC, LocationID ASC "; + $q_sort = "ORDER BY CategoryID ".$order.", Name ASC "; $_GET["SB"] = 'c'; break; - case 'l': - $q_sort = "ORDER BY LocationID ".$order.", Name ASC, CategoryID DESC "; - $_GET["SB"] = 'l'; - break; case 'v': $q_sort = "ORDER BY NumVotes ".$order.", Name ASC, CategoryID DESC "; $_GET["SB"] = 'v'; @@ -542,11 +496,11 @@ function pkg_search_page($SID="") { $_GET["SB"] = 'o'; break; case 'm': - $q_sort = "ORDER BY Maintainer ".$order.", Name ASC, LocationID ASC "; + $q_sort = "ORDER BY Maintainer ".$order.", Name ASC "; $_GET["SB"] = 'm'; break; case 'a': - $q_sort = "ORDER BY GREATEST(SubmittedTS,ModifiedTS) ".$order.", Name ASC, LocationID ASC "; + $q_sort = "ORDER BY GREATEST(SubmittedTS,ModifiedTS) ".$order.", Name ASC "; $_GET["SB"] = 'a'; break; default: @@ -719,10 +673,8 @@ function pkg_delete ($atype, $ids) { # Only grab Unsupported packages that "we" own or are not owned at all $ids_to_delete = array(); - $q = "SELECT Packages.ID FROM Packages, PackageLocations "; + $q = "SELECT Packages.ID FROM Packages "; $q.= "WHERE Packages.ID IN (" . $delete . ") "; - $q.= "AND Packages.LocationID = PackageLocations.ID "; - $q.= "AND PackageLocations.Location = 'unsupported' "; # If they're a TU or dev, can delete if ($atype == "Trusted User" || $atype == "Developer") { @@ -816,7 +768,6 @@ function pkg_adopt ($atype, $ids, $action = True) { if ($action && $atype == "User") { # Regular users may only adopt orphan packages from unsupported $q.= "AND $field = 0 "; - $q.= "AND LocationID = 2 "; } else if ($atype == "User") { $q.= "AND $field = " . uid_from_sid($_COOKIE["AURSID"]); } @@ -1047,20 +998,18 @@ function pkg_change_category($atype) { return __("Missing package ID."); } - # Verify package ownership and location + # Verify package ownership $dbh = db_connect(); $q = "SELECT Packages.MaintainerUID,"; - $q.= "PackageLocations.Location "; $q.= "FROM Packages "; - $q.= "LEFT JOIN PackageLocations ON Packages.LocationID = PackageLocations.ID "; $q.= "WHERE Packages.ID = ".$pid; $result = db_query($q, $dbh); echo mysql_error(); $pkg = mysql_fetch_assoc($result); $uid = uid_from_sid($_COOKIE["AURSID"]); - if ($pkg["Location"] == "unsupported" and ($uid == $pkg["MaintainerUID"] or - ($atype == "Developer" or $atype == "Trusted User"))) { + if ($uid == $pkg["MaintainerUID"] or + ($atype == "Developer" or $atype == "Trusted User")) { $q = "UPDATE Packages "; $q.= "SET CategoryID = ".intval($category_id)." "; $q.= "WHERE ID = ".intval($pid); diff --git a/web/lib/stats.inc b/web/lib/stats.inc index 6a76558..f42e417 100644 --- a/web/lib/stats.inc +++ b/web/lib/stats.inc @@ -54,9 +54,9 @@ function user_table($user, $dbh) { global $apc_prefix; $escuser = mysql_real_escape_string($user); - $base_q = "SELECT count(*) FROM Packages,PackageLocations,Users WHERE Packages.MaintainerUID = Users.ID AND Packages.LocationID = PackageLocations.ID AND PackageLocations.Location = '%s' AND Users.Username='" . $escuser . "'"; + $base_q = "SELECT count(*) FROM Packages,Users WHERE Packages.MaintainerUID = Users.ID AND Users.Username='" . $escuser . "'"; - $maintainer_unsupported_count = db_cache_value(sprintf($base_q, 'unsupported'), $dbh, + $maintainer_unsupported_count = db_cache_value($base_q, $dbh, $apc_prefix . 'user_unsupported_count:' . $escuser); $q = "SELECT count(*) FROM Packages,Users WHERE Packages.OutOfDateTS IS NOT NULL AND Packages.MaintainerUID = Users.ID AND Users.Username='" . $escuser . "'"; @@ -74,7 +74,7 @@ function general_stats_table($dbh) { global $apc_prefix; # AUR statistics - $q = "SELECT count(*) FROM Packages,PackageLocations WHERE Packages.LocationID = PackageLocations.ID AND PackageLocations.Location = 'unsupported'"; + $q = "SELECT count(*) FROM Packages WHERE DummyPkg = 0"; $unsupported_count = db_cache_value($q, $dbh, $apc_prefix . 'unsupported_count'); $q = "SELECT count(*) from Users"; diff --git a/web/template/pkg_details.php b/web/template/pkg_details.php index 8aba3af..c1c3e96 100644 --- a/web/template/pkg_details.php +++ b/web/template/pkg_details.php @@ -1,13 +1,13 @@ <?php $pkgid = intval($_REQUEST['ID']); -if ($row["Location"] == "unsupported" and ($uid == $row["MaintainerUID"] or - ($atype == "Developer" or $atype == "Trusted User"))) { +if ($uid == $row["MaintainerUID"] or + ($atype == "Developer" or $atype == "Trusted User")) { $catarr = pkgCategories(); $edit_cat = "<form method='POST' action='packages.php?ID=".$pkgid."'>\n"; $edit_cat.= "<input type='hidden' name='action' value='do_ChangeCategory'>"; - $edit_cat.= $row['Location']." :: "; + $edit_cat.= "Category: "; $edit_cat.= "<select name='category_id'>\n"; foreach ($catarr as $cid => $catname) { $edit_cat.= "<option value='$cid'"; @@ -21,7 +21,7 @@ if ($row["Location"] == "unsupported" and ($uid == $row["MaintainerUID"] or } else { - $edit_cat = $row['Location']." :: ".$row['Category']; + $edit_cat = "Category: ".$row['Category']; } if ($row["MaintainerUID"]) { @@ -76,11 +76,9 @@ $out_of_date_time = ($row["OutOfDateTS"] == 0) ? $msg : gmdate("r", intval($row[ <p><span class='f3'> <?php - if ($row['LocationID'] == 2) { - $urlpath = URL_DIR . $row['Name']; - print "<a href='$urlpath/" . $row['Name'] . ".tar.gz'>".__("Tarball")."</a> :: "; - print "<a href='$urlpath/PKGBUILD'>".__("PKGBUILD")."</a></span>"; - } + $urlpath = URL_DIR . $row['Name']; + print "<a href='$urlpath/" . $row['Name'] . ".tar.gz'>".__("Tarball")."</a> :: "; + print "<a href='$urlpath/PKGBUILD'>".__("PKGBUILD")."</a></span>"; if ($row["OutOfDateTS"] !== NULL) { echo "<br /><span class='f6'>".__("This package has been flagged out of date.")." (${out_of_date_time})</span>"; @@ -172,10 +170,8 @@ $out_of_date_time = ($row["OutOfDateTS"] == 0) ? $msg : gmdate("r", intval($row[ else { $src = $src[0]; # It is presumably an internal source - if ($row["LocationID"] == 2) { - echo "<span class='f8'>$src</span>"; - echo "<br />\n"; - } + echo "<span class='f8'>$src</span>"; + echo "<br />\n"; } } ?> diff --git a/web/template/pkg_search_form.php b/web/template/pkg_search_form.php index c616c7b..fdeb01b 100644 --- a/web/template/pkg_search_form.php +++ b/web/template/pkg_search_form.php @@ -56,7 +56,7 @@ <label><?php print __("Sort by"); ?></label> <select name='SB'> <?php - $sortby = array('n' => __('Name'), 'c' => __('Category'), 'l' => __('Location'), 'v' => __('Votes'), 'w' => __('Voted'), 'o' => __('Notify'), 'm' => __('Maintainer'), 'a' => __('Age')); + $sortby = array('n' => __('Name'), 'c' => __('Category'), 'v' => __('Votes'), 'w' => __('Voted'), 'o' => __('Notify'), 'm' => __('Maintainer'), 'a' => __('Age')); foreach ($sortby as $k => $v): if ($_REQUEST['SB'] == $k): ?> diff --git a/web/template/pkg_search_results.php b/web/template/pkg_search_results.php index b8f2e14..4830ca8 100644 --- a/web/template/pkg_search_results.php +++ b/web/template/pkg_search_results.php @@ -16,9 +16,6 @@ <?php endif; ?> <th style='border-bottom: #666 1px solid; vertical-align: bottom'><span class='f2'> - <a href='?<?php print mkurl('SB=l&SO=' . $SO_next) ?>'><?php print __("Location") ?></a> - </span></th> - <th style='border-bottom: #666 1px solid; vertical-align: bottom'><span class='f2'> <a href='?<?php print mkurl('SB=c&SO=' . $SO_next) ?>'><?php print __("Category") ?></a> </span></th> <th style='border-bottom: #666 1px solid; vertical-align: bottom; text-align: center;'><span class='f2'> @@ -52,7 +49,6 @@ for ($i = 0; $row = mysql_fetch_assoc($result); $i++) { <?php if ($SID): ?> <td class='<?php print $c ?>'><input type='checkbox' name='IDs[<?php print $row["ID"] ?>]' value='1'></td> <?php endif; ?> - <td class='<?php print $c ?>'><span class='f5'><span class='blue'><?php print $row["Location"] ?></span></span></td> <td class='<?php print $c ?>'><span class='f5'><span class='blue'><?php print $row["Category"] ?></span></span></td> <td class='<?php print $c ?>'><span class='f4'><a href='packages.php?ID=<?php print $row["ID"] ?>'><span class='black'><?php print $row["Name"] ?> <?php print $row["Version"] ?></span></a></span></td> <td class='<?php print $c ?>' style="text-align: right"><span class='f5'><span class='blue'><?php print $row["NumVotes"] ?></span></span></td> -- 1.7.3.5
On Sat, Jan 29, 2011 at 04:20:06PM -0600, Dan McGee wrote:
We don't need this anymore since all packages managed here are well...managed here. Rip out all of the places we were using this field, many of which depended on the magic value '2' anyway.
On the display side of things, we had a column that was always showing 'unsupported' that is now gone, and you can no longer sort by this column.
Sounds like a good idea to me. I just applied it to my working tree and it basically looks fine. As far as I can see by now, you only broke one SQL query:
@@ -324,9 +293,8 @@ function package_details($id=0, $SID="") { $atype = account_from_sid($SID); $uid = uid_from_sid($SID);
- $q = "SELECT Packages.*,Location,Category "; - $q.= "FROM Packages,PackageLocations,PackageCategories "; - $q.= "WHERE Packages.LocationID = PackageLocations.ID "; + $q = "SELECT Packages.*,Category "; + $q.= "FROM Packages,PackageCategories "; $q.= "AND Packages.CategoryID = PackageCategories.ID ";
... should be patched to use "WHERE" instead of "AND" of course.
$q.= "AND Packages.ID = " . intval($id); $dbh = db_connect();
Apart from this, I have no objections against pushing this. I already commited that to my working tree. Just let me do some more testing :)
We don't need this anymore since all packages managed here are well...managed here. Rip out all of the places we were using this field, many of which depended on the magic value '2' anyway. On the display side of things, we had a column that was always showing 'unsupported' that is now gone, and you can no longer sort by this column. Signed-off-by: Dan McGee <dan@archlinux.org> --- Added Lukas' small correction as well as one of my own in UPGRADING. UPGRADING | 3 +- scripts/cleanup | 2 +- support/schema/aur-schema.sql | 20 +------ support/schema/gendummydata.py | 106 +++++++++------------------------- web/html/pkgsubmit.php | 5 +- web/lib/aurjson.class.php | 2 +- web/lib/pkgfuncs.inc | 77 ++++--------------------- web/lib/stats.inc | 6 +- web/template/pkg_details.php | 22 +++---- web/template/pkg_search_form.php | 2 +- web/template/pkg_search_results.php | 4 - 11 files changed, 61 insertions(+), 188 deletions(-) diff --git a/UPGRADING b/UPGRADING index 2d3434f..61b1462 100644 --- a/UPGRADING +++ b/UPGRADING @@ -9,7 +9,8 @@ From 1.7.0 to 1.8.0 ---- ALTER TABLE Packages ADD OutOfDateTS BIGINT UNSIGNED NULL DEFAULT NULL; UPDATE Packages SET OutOfDateTS = UNIX_TIMESTAMP() WHERE OutOfDate = 1; -ALTER TABLE Packages DROP OutOfDate, DROP URLPath; +ALTER TABLE Packages DROP OutOfDate, DROP URLPath, LocationID; +DROP TABLE PackageLocations; ---- 2. You will need to update all packages which are stored in the incoming dir as diff --git a/scripts/cleanup b/scripts/cleanup index 21c913a..4fc9ea2 100755 --- a/scripts/cleanup +++ b/scripts/cleanup @@ -25,7 +25,7 @@ exec('ls ' . INCOMING_DIR, $files); $count = 0; foreach ($files as $pkgname) { - if (package_location($pkgname) != 'unsupported') { + if (!package_exists($pkgname)) { echo 'Removing ' . INCOMING_DIR . "$pkgname\n"; system('rm -r ' . INCOMING_DIR . $pkgname); $count++; diff --git a/support/schema/aur-schema.sql b/support/schema/aur-schema.sql index cd24f38..6af8167 100644 --- a/support/schema/aur-schema.sql +++ b/support/schema/aur-schema.sql @@ -1,5 +1,5 @@ -- The MySQL database layout for the AUR. Certain data --- is also included such as AccountTypes, PackageLocations, etc. +-- is also included such as AccountTypes, etc. -- DROP DATABASE AUR; CREATE DATABASE AUR; @@ -89,21 +89,6 @@ INSERT INTO PackageCategories (Category) VALUES ('x11'); INSERT INTO PackageCategories (Category) VALUES ('xfce'); --- The various repositories that a package could live in. --- -CREATE TABLE PackageLocations ( - ID TINYINT UNSIGNED NOT NULL AUTO_INCREMENT, - Location CHAR(32) NOT NULL, - PRIMARY KEY (ID) -); -INSERT INTO PackageLocations (Location) VALUES ('none'); -INSERT INTO PackageLocations (Location) VALUES ('unsupported'); -INSERT INTO PackageLocations (Location) VALUES ('community'); -INSERT INTO PackageLocations (Location) VALUES ('current'); -INSERT INTO PackageLocations (Location) VALUES ('extra'); -INSERT INTO PackageLocations (Location) VALUES ('unstable'); - - -- Information about the actual packages -- CREATE TABLE Packages ( @@ -116,7 +101,6 @@ CREATE TABLE Packages ( DummyPkg TINYINT UNSIGNED NOT NULL DEFAULT 0, -- 1=>dummy FSPath CHAR(255) NOT NULL DEFAULT '', License CHAR(40) NOT NULL DEFAULT '', - LocationID TINYINT UNSIGNED NOT NULL DEFAULT 1, NumVotes INTEGER UNSIGNED NOT NULL DEFAULT 0, OutOfDateTS BIGINT UNSIGNED NULL DEFAULT NULL, SubmittedTS BIGINT UNSIGNED NOT NULL, @@ -127,13 +111,11 @@ CREATE TABLE Packages ( PRIMARY KEY (ID), UNIQUE (Name), INDEX (CategoryID), - INDEX (LocationID), INDEX (DummyPkg), INDEX (NumVotes), INDEX (SubmitterUID), INDEX (MaintainerUID), FOREIGN KEY (CategoryID) REFERENCES PackageCategories(ID) ON DELETE NO ACTION, - FOREIGN KEY (LocationID) REFERENCES PackageLocations(ID) ON DELETE NO ACTION, FOREIGN KEY (SubmitterUID) REFERENCES Users(ID) ON DELETE NO ACTION, FOREIGN KEY (MaintainerUID) REFERENCES Users(ID) ON DELETE NO ACTION ); diff --git a/support/schema/gendummydata.py b/support/schema/gendummydata.py index 99f8195..a03b3e6 100755 --- a/support/schema/gendummydata.py +++ b/support/schema/gendummydata.py @@ -85,28 +85,21 @@ esc = db.escape_string # seen_users = {} seen_pkgs = {} -locations = {} categories = {} -location_keys = [] category_keys = [] user_keys = [] # some functions to generate random data # -def genVersion(location_id=0): +def genVersion(): ver = [] ver.append("%d" % random.randrange(0,10)) ver.append("%d" % random.randrange(0,20)) if random.randrange(0,2) == 0: ver.append("%d" % random.randrange(0,100)) - if location_id == 2: # the package is in the AUR - return ".".join(ver) + "-u%d" % random.randrange(1,11) - else: - return ".".join(ver) + "%d" % random.randrange(1,11) + return ".".join(ver) + "-u%d" % random.randrange(1,11) def genCategory(): return categories[category_keys[random.randrange(0,len(category_keys))]] -def genLocation(): - return locations[location_keys[random.randrange(0,len(location_keys))]] def genUID(): return seen_users[user_keys[random.randrange(0,len(user_keys))]] @@ -162,7 +155,7 @@ contents = None # Load package categories from database # -if DBUG: print "Loading package categories/locations..." +if DBUG: print "Loading package categories..." q = "SELECT * FROM PackageCategories" dbc.execute(q) row = dbc.fetchone() @@ -171,16 +164,6 @@ while row: row = dbc.fetchone() category_keys = categories.keys() -# Load package locations from database -# -q = "SELECT * FROM PackageLocations" -dbc.execute(q) -row = dbc.fetchone() -while row: - locations[row[1]] = row[0] - row = dbc.fetchone() -location_keys = locations.keys() - # done with the database # dbc.close() @@ -243,14 +226,10 @@ for p in seen_pkgs.keys(): if count % 20 == 0: # every so often, there are orphans... muid = 0 - location_id = genLocation() - if location_id == 1: # unsupported pkgs don't have a maintainer - muid = 0 - uuid = genUID() # the submitter/user - s = "INSERT INTO Packages (ID, Name, Version, CategoryID, LocationID, SubmittedTS, SubmitterUID, MaintainerUID) VALUES (%d, '%s', '%s', %d, %d, %d, %d, %d);\n" % (seen_pkgs[p], p, genVersion(location_id), - genCategory(), location_id, NOW, uuid, muid) + s = "INSERT INTO Packages (ID, Name, Version, CategoryID, SubmittedTS, SubmitterUID, MaintainerUID) VALUES (%d, '%s', '%s', %d, %d, %d, %d);\n" % (seen_pkgs[p], p, genVersion(), + genCategory(), NOW, uuid, muid) out.write(s) if count % 100 == 0: if DBUG: print ".", @@ -265,63 +244,34 @@ for p in seen_pkgs.keys(): s = "INSERT INTO PackageComments (PackageID, UsersID, Comments, CommentTS) VALUES (%d, %d, '%s', %d);\n" % (seen_pkgs[p], genUID(), fortune, now) out.write(s) - if location_id == 1: # Unsupported - just a PKGBUILD and maybe other stuff - others = random.randrange(0,3) - s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/PKGBUILD" % p, + others = random.randrange(0,3) + s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/PKGBUILD" % p, + random.randrange(0,999)) + out.write(s) + if others == 0: + s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/%s.patch" % (p,p), random.randrange(0,999)) out.write(s) - if others == 0: - s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/%s.patch" % (p,p), - random.randrange(0,999)) - out.write(s) - elif others == 1: - s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/%s.patch" % (p,p), - random.randrange(0,999)) - out.write(s) - s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/arch.patch" % p, - random.randrange(0,999)) - out.write(s) + elif others == 1: + s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/%s.patch" % (p,p), + random.randrange(0,999)) + out.write(s) + s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/arch.patch" % p, + random.randrange(0,999)) + out.write(s) - elif others == 2: - s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/%s.patch" % (p,p), - random.randrange(0,999)) - out.write(s) - s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/arch.patch" % p, - random.randrange(0,999)) - out.write(s) - s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/%s.install" % (p,p), - random.randrange(0,999)) - out.write(s) + elif others == 2: + s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/%s.patch" % (p,p), + random.randrange(0,999)) + out.write(s) + s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/arch.patch" % p, + random.randrange(0,999)) + out.write(s) + s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/%s.install" % (p,p), + random.randrange(0,999)) + out.write(s) - else: - # Create package contents - # - num_files = random.randrange(PKG_FILES[0], PKG_FILES[1]) - files = {} - for f in range(num_files): - loc = RANDOM_PATHS[random.randrange(len(RANDOM_PATHS))] - if "lib" in loc: - path = loc + "/lib" + p + ".so" - elif "man" in loc: - path = loc + "/" + p + "." + loc[-1] + ".gz" - elif "share" in loc: - path = loc + "/" + p + "/sounds/" + p + ".wav" - elif "profile" in loc: - path = loc + "/" + p + ".sh" - elif "rc.d" in loc: - path = loc + "/" + p - elif "etc" in loc: - path = loc + "/" + p + ".conf" - elif "opt" in loc: - path = loc + "/" + p + "/bin/" + p - else: - path = loc + "/" + p - if not files.has_key(path): - files[path] = 1 - s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], path, - random.randrange(0,99999999)) - out.write(s) if DBUG: print "." # Cast votes diff --git a/web/html/pkgsubmit.php b/web/html/pkgsubmit.php index c39e2f9..3d77f72 100644 --- a/web/html/pkgsubmit.php +++ b/web/html/pkgsubmit.php @@ -277,7 +277,7 @@ if ($_COOKIE["AURSID"]): } # Update package data - $q = sprintf("UPDATE Packages SET ModifiedTS = UNIX_TIMESTAMP(), Name = '%s', Version = '%s-%s', License = '%s', Description = '%s', URL = '%s', LocationID = 2, FSPath = '%s', OutOfDateTS = NULL WHERE ID = %d", + $q = sprintf("UPDATE Packages SET ModifiedTS = UNIX_TIMESTAMP(), Name = '%s', Version = '%s-%s', License = '%s', Description = '%s', URL = '%s', FSPath = '%s', OutOfDateTS = NULL WHERE ID = %d", mysql_real_escape_string($new_pkgbuild['pkgname']), mysql_real_escape_string($new_pkgbuild['pkgver']), mysql_real_escape_string($new_pkgbuild['pkgrel']), @@ -322,7 +322,7 @@ if ($_COOKIE["AURSID"]): $uid = uid_from_sid($_COOKIE["AURSID"]); # This is a brand new package - $q = sprintf("INSERT INTO Packages (Name, License, Version, CategoryID, Description, URL, LocationID, SubmittedTS, SubmitterUID, MaintainerUID, FSPath) VALUES ('%s', '%s', '%s-%s', %d, '%s', '%s', 2, UNIX_TIMESTAMP(), %d, %d, '%s')", + $q = sprintf("INSERT INTO Packages (Name, License, Version, CategoryID, Description, URL, SubmittedTS, SubmitterUID, MaintainerUID, FSPath) VALUES ('%s', '%s', '%s-%s', %d, '%s', '%s', UNIX_TIMESTAMP(), %d, %d, '%s')", mysql_real_escape_string($new_pkgbuild['pkgname']), mysql_real_escape_string($new_pkgbuild['license']), mysql_real_escape_string($new_pkgbuild['pkgver']), @@ -398,7 +398,6 @@ html_header("Submit"); if (ini_get("file_uploads")): $pkg_categories = pkgCategories(); - $pkg_locations = pkgLocations(); ?> <form action='pkgsubmit.php' method='post' enctype='multipart/form-data'> diff --git a/web/lib/aurjson.class.php b/web/lib/aurjson.class.php index 0868cb2..285aad7 100644 --- a/web/lib/aurjson.class.php +++ b/web/lib/aurjson.class.php @@ -29,7 +29,7 @@ class AurJSON { $this->fields = array( 'Packages.ID', 'Name', 'Version', 'CategoryID', - 'Description', 'LocationID', 'URL', 'CONCAT("' . + 'Description', 'URL', 'CONCAT("' . mysql_real_escape_string(URL_DIR) . '", Name, "/", Name, ".tar.gz") AS URLPath', 'License', 'NumVotes', '(OutOfDateTS IS NOT NULL) AS OutOfDate' diff --git a/web/lib/pkgfuncs.inc b/web/lib/pkgfuncs.inc index c701348..66331d6 100644 --- a/web/lib/pkgfuncs.inc +++ b/web/lib/pkgfuncs.inc @@ -80,37 +80,6 @@ function pkgCategories() { return $cats; } -# grab the current list of PackageLocations -# -function pkgLocations() { - $locs = array(); - $dbh = db_connect(); - $q = "SELECT * FROM PackageLocations WHERE ID != 1 AND ID < 4 "; - $q.= "ORDER BY Location ASC"; - $result = db_query($q, $dbh); - if ($result) { - while ($row = mysql_fetch_row($result)) { - $locs[$row[0]] = $row[1]; - } - } - return $locs; -} - -# Return the repository name for a particular package. -function package_location($name="") { - if (!$name) {return NULL;} - $dbh = db_connect(); - $q = "SELECT PackageLocations.Location FROM Packages "; - $q.= "LEFT JOIN PackageLocations ON "; - $q.= "Packages.LocationID = PackageLocations.ID "; - $q.= "WHERE Name = '".mysql_real_escape_string($name)."' "; - $q.= "AND DummyPkg = 0"; - $result = db_query($q, $dbh); - if (!$result) {return NULL;} - $row = mysql_fetch_row($result); - return $row[0]; -} - # check to see if the package name exists # function package_exists($name="") { @@ -324,9 +293,8 @@ function package_details($id=0, $SID="") { $atype = account_from_sid($SID); $uid = uid_from_sid($SID); - $q = "SELECT Packages.*,Location,Category "; - $q.= "FROM Packages,PackageLocations,PackageCategories "; - $q.= "WHERE Packages.LocationID = PackageLocations.ID "; + $q = "SELECT Packages.*,Category "; + $q.= "FROM Packages,PackageCategories "; $q.= "AND Packages.CategoryID = PackageCategories.ID "; $q.= "AND Packages.ID = " . intval($id); $dbh = db_connect(); @@ -373,15 +341,13 @@ function package_details($id=0, $SID="") { * request vars: * O - starting result number * PP - number of search hits per page - * L - package location ID number * C - package category ID number * K - package search string * SO - search hit sort order: * values: a - ascending * d - descending * SB - sort search hits by: - * values: l - package location - * c - package category + * values: c - package category * n - package name * v - number of votes * m - maintainer username @@ -415,8 +381,6 @@ function pkg_search_page($SID="") { // grab info for user if they're logged in if ($SID) $myuid = uid_from_sid($SID); - // get a list of package locations - $locs = pkgLocations(); // get a list of package categories $cats = pkgCategories(); //meow @@ -455,7 +419,6 @@ function pkg_search_page($SID="") { } $q .= "Users.Username AS Maintainer, PackageCategories.Category, - PackageLocations.Location, Packages.Name, Packages.Version, Packages.Description, Packages.NumVotes, Packages.ID, Packages.OutOfDateTS @@ -469,15 +432,10 @@ function pkg_search_page($SID="") { } $q .= "LEFT JOIN PackageCategories ON (Packages.CategoryID = PackageCategories.ID) - LEFT JOIN PackageLocations - ON (Packages.LocationID = PackageLocations.ID) WHERE Packages.DummyPkg = 0 "; - // TODO: possibly do string matching on category and - // location to make request variable values more sensible - if (intval($_GET["L"])) { - $q .= "AND Packages.LocationID = ".intval($_GET["L"])." "; - } + // TODO: possibly do string matching on category + // to make request variable values more sensible if (intval($_GET["C"])) { $q.= "AND Packages.CategoryID = ".intval($_GET["C"])." "; } @@ -515,16 +473,12 @@ function pkg_search_page($SID="") { $order = $_GET["SO"] == 'd' ? 'DESC' : 'ASC'; - $q_sort = "ORDER BY Name ".$order.", LocationID ASC, CategoryID DESC "; + $q_sort = "ORDER BY Name ".$order.", CategoryID DESC "; switch ($_GET["SB"]) { case 'c': - $q_sort = "ORDER BY CategoryID ".$order.", Name ASC, LocationID ASC "; + $q_sort = "ORDER BY CategoryID ".$order.", Name ASC "; $_GET["SB"] = 'c'; break; - case 'l': - $q_sort = "ORDER BY LocationID ".$order.", Name ASC, CategoryID DESC "; - $_GET["SB"] = 'l'; - break; case 'v': $q_sort = "ORDER BY NumVotes ".$order.", Name ASC, CategoryID DESC "; $_GET["SB"] = 'v'; @@ -542,11 +496,11 @@ function pkg_search_page($SID="") { $_GET["SB"] = 'o'; break; case 'm': - $q_sort = "ORDER BY Maintainer ".$order.", Name ASC, LocationID ASC "; + $q_sort = "ORDER BY Maintainer ".$order.", Name ASC "; $_GET["SB"] = 'm'; break; case 'a': - $q_sort = "ORDER BY GREATEST(SubmittedTS,ModifiedTS) ".$order.", Name ASC, LocationID ASC "; + $q_sort = "ORDER BY GREATEST(SubmittedTS,ModifiedTS) ".$order.", Name ASC "; $_GET["SB"] = 'a'; break; default: @@ -719,10 +673,8 @@ function pkg_delete ($atype, $ids) { # Only grab Unsupported packages that "we" own or are not owned at all $ids_to_delete = array(); - $q = "SELECT Packages.ID FROM Packages, PackageLocations "; + $q = "SELECT Packages.ID FROM Packages "; $q.= "WHERE Packages.ID IN (" . $delete . ") "; - $q.= "AND Packages.LocationID = PackageLocations.ID "; - $q.= "AND PackageLocations.Location = 'unsupported' "; # If they're a TU or dev, can delete if ($atype == "Trusted User" || $atype == "Developer") { @@ -816,7 +768,6 @@ function pkg_adopt ($atype, $ids, $action = True) { if ($action && $atype == "User") { # Regular users may only adopt orphan packages from unsupported $q.= "AND $field = 0 "; - $q.= "AND LocationID = 2 "; } else if ($atype == "User") { $q.= "AND $field = " . uid_from_sid($_COOKIE["AURSID"]); } @@ -1047,20 +998,18 @@ function pkg_change_category($atype) { return __("Missing package ID."); } - # Verify package ownership and location + # Verify package ownership $dbh = db_connect(); $q = "SELECT Packages.MaintainerUID,"; - $q.= "PackageLocations.Location "; $q.= "FROM Packages "; - $q.= "LEFT JOIN PackageLocations ON Packages.LocationID = PackageLocations.ID "; $q.= "WHERE Packages.ID = ".$pid; $result = db_query($q, $dbh); echo mysql_error(); $pkg = mysql_fetch_assoc($result); $uid = uid_from_sid($_COOKIE["AURSID"]); - if ($pkg["Location"] == "unsupported" and ($uid == $pkg["MaintainerUID"] or - ($atype == "Developer" or $atype == "Trusted User"))) { + if ($uid == $pkg["MaintainerUID"] or + ($atype == "Developer" or $atype == "Trusted User")) { $q = "UPDATE Packages "; $q.= "SET CategoryID = ".intval($category_id)." "; $q.= "WHERE ID = ".intval($pid); diff --git a/web/lib/stats.inc b/web/lib/stats.inc index 6a76558..f42e417 100644 --- a/web/lib/stats.inc +++ b/web/lib/stats.inc @@ -54,9 +54,9 @@ function user_table($user, $dbh) { global $apc_prefix; $escuser = mysql_real_escape_string($user); - $base_q = "SELECT count(*) FROM Packages,PackageLocations,Users WHERE Packages.MaintainerUID = Users.ID AND Packages.LocationID = PackageLocations.ID AND PackageLocations.Location = '%s' AND Users.Username='" . $escuser . "'"; + $base_q = "SELECT count(*) FROM Packages,Users WHERE Packages.MaintainerUID = Users.ID AND Users.Username='" . $escuser . "'"; - $maintainer_unsupported_count = db_cache_value(sprintf($base_q, 'unsupported'), $dbh, + $maintainer_unsupported_count = db_cache_value($base_q, $dbh, $apc_prefix . 'user_unsupported_count:' . $escuser); $q = "SELECT count(*) FROM Packages,Users WHERE Packages.OutOfDateTS IS NOT NULL AND Packages.MaintainerUID = Users.ID AND Users.Username='" . $escuser . "'"; @@ -74,7 +74,7 @@ function general_stats_table($dbh) { global $apc_prefix; # AUR statistics - $q = "SELECT count(*) FROM Packages,PackageLocations WHERE Packages.LocationID = PackageLocations.ID AND PackageLocations.Location = 'unsupported'"; + $q = "SELECT count(*) FROM Packages WHERE DummyPkg = 0"; $unsupported_count = db_cache_value($q, $dbh, $apc_prefix . 'unsupported_count'); $q = "SELECT count(*) from Users"; diff --git a/web/template/pkg_details.php b/web/template/pkg_details.php index 8aba3af..c1c3e96 100644 --- a/web/template/pkg_details.php +++ b/web/template/pkg_details.php @@ -1,13 +1,13 @@ <?php $pkgid = intval($_REQUEST['ID']); -if ($row["Location"] == "unsupported" and ($uid == $row["MaintainerUID"] or - ($atype == "Developer" or $atype == "Trusted User"))) { +if ($uid == $row["MaintainerUID"] or + ($atype == "Developer" or $atype == "Trusted User")) { $catarr = pkgCategories(); $edit_cat = "<form method='POST' action='packages.php?ID=".$pkgid."'>\n"; $edit_cat.= "<input type='hidden' name='action' value='do_ChangeCategory'>"; - $edit_cat.= $row['Location']." :: "; + $edit_cat.= "Category: "; $edit_cat.= "<select name='category_id'>\n"; foreach ($catarr as $cid => $catname) { $edit_cat.= "<option value='$cid'"; @@ -21,7 +21,7 @@ if ($row["Location"] == "unsupported" and ($uid == $row["MaintainerUID"] or } else { - $edit_cat = $row['Location']." :: ".$row['Category']; + $edit_cat = "Category: ".$row['Category']; } if ($row["MaintainerUID"]) { @@ -76,11 +76,9 @@ $out_of_date_time = ($row["OutOfDateTS"] == 0) ? $msg : gmdate("r", intval($row[ <p><span class='f3'> <?php - if ($row['LocationID'] == 2) { - $urlpath = URL_DIR . $row['Name']; - print "<a href='$urlpath/" . $row['Name'] . ".tar.gz'>".__("Tarball")."</a> :: "; - print "<a href='$urlpath/PKGBUILD'>".__("PKGBUILD")."</a></span>"; - } + $urlpath = URL_DIR . $row['Name']; + print "<a href='$urlpath/" . $row['Name'] . ".tar.gz'>".__("Tarball")."</a> :: "; + print "<a href='$urlpath/PKGBUILD'>".__("PKGBUILD")."</a></span>"; if ($row["OutOfDateTS"] !== NULL) { echo "<br /><span class='f6'>".__("This package has been flagged out of date.")." (${out_of_date_time})</span>"; @@ -172,10 +170,8 @@ $out_of_date_time = ($row["OutOfDateTS"] == 0) ? $msg : gmdate("r", intval($row[ else { $src = $src[0]; # It is presumably an internal source - if ($row["LocationID"] == 2) { - echo "<span class='f8'>$src</span>"; - echo "<br />\n"; - } + echo "<span class='f8'>$src</span>"; + echo "<br />\n"; } } ?> diff --git a/web/template/pkg_search_form.php b/web/template/pkg_search_form.php index c616c7b..fdeb01b 100644 --- a/web/template/pkg_search_form.php +++ b/web/template/pkg_search_form.php @@ -56,7 +56,7 @@ <label><?php print __("Sort by"); ?></label> <select name='SB'> <?php - $sortby = array('n' => __('Name'), 'c' => __('Category'), 'l' => __('Location'), 'v' => __('Votes'), 'w' => __('Voted'), 'o' => __('Notify'), 'm' => __('Maintainer'), 'a' => __('Age')); + $sortby = array('n' => __('Name'), 'c' => __('Category'), 'v' => __('Votes'), 'w' => __('Voted'), 'o' => __('Notify'), 'm' => __('Maintainer'), 'a' => __('Age')); foreach ($sortby as $k => $v): if ($_REQUEST['SB'] == $k): ?> diff --git a/web/template/pkg_search_results.php b/web/template/pkg_search_results.php index b8f2e14..4830ca8 100644 --- a/web/template/pkg_search_results.php +++ b/web/template/pkg_search_results.php @@ -16,9 +16,6 @@ <?php endif; ?> <th style='border-bottom: #666 1px solid; vertical-align: bottom'><span class='f2'> - <a href='?<?php print mkurl('SB=l&SO=' . $SO_next) ?>'><?php print __("Location") ?></a> - </span></th> - <th style='border-bottom: #666 1px solid; vertical-align: bottom'><span class='f2'> <a href='?<?php print mkurl('SB=c&SO=' . $SO_next) ?>'><?php print __("Category") ?></a> </span></th> <th style='border-bottom: #666 1px solid; vertical-align: bottom; text-align: center;'><span class='f2'> @@ -52,7 +49,6 @@ for ($i = 0; $row = mysql_fetch_assoc($result); $i++) { <?php if ($SID): ?> <td class='<?php print $c ?>'><input type='checkbox' name='IDs[<?php print $row["ID"] ?>]' value='1'></td> <?php endif; ?> - <td class='<?php print $c ?>'><span class='f5'><span class='blue'><?php print $row["Location"] ?></span></span></td> <td class='<?php print $c ?>'><span class='f5'><span class='blue'><?php print $row["Category"] ?></span></span></td> <td class='<?php print $c ?>'><span class='f4'><a href='packages.php?ID=<?php print $row["ID"] ?>'><span class='black'><?php print $row["Name"] ?> <?php print $row["Version"] ?></span></a></span></td> <td class='<?php print $c ?>' style="text-align: right"><span class='f5'><span class='blue'><?php print $row["NumVotes"] ?></span></span></td> -- 1.7.3.5
We don't need this anymore since all packages managed here are well...managed here. Rip out all of the places we were using this field, many of which depended on the magic value '2' anyway. On the display side of things, we had a column that was always showing 'unsupported' that is now gone, and you can no longer sort by this column. Signed-off-by: Dan McGee <dan@archlinux.org> --- Totally embarassing there; this is the correct patch with the necessary corrections. UPGRADING | 3 +- scripts/cleanup | 2 +- support/schema/aur-schema.sql | 20 +------ support/schema/gendummydata.py | 106 +++++++++------------------------- web/html/pkgsubmit.php | 5 +- web/lib/aurjson.class.php | 2 +- web/lib/pkgfuncs.inc | 79 +++++--------------------- web/lib/stats.inc | 6 +- web/template/pkg_details.php | 22 +++---- web/template/pkg_search_form.php | 2 +- web/template/pkg_search_results.php | 4 - 11 files changed, 62 insertions(+), 189 deletions(-) diff --git a/UPGRADING b/UPGRADING index 2d3434f..e5dc527 100644 --- a/UPGRADING +++ b/UPGRADING @@ -9,7 +9,8 @@ From 1.7.0 to 1.8.0 ---- ALTER TABLE Packages ADD OutOfDateTS BIGINT UNSIGNED NULL DEFAULT NULL; UPDATE Packages SET OutOfDateTS = UNIX_TIMESTAMP() WHERE OutOfDate = 1; -ALTER TABLE Packages DROP OutOfDate, DROP URLPath; +ALTER TABLE Packages DROP OutOfDate, DROP URLPath, DROP LocationID; +DROP TABLE PackageLocations; ---- 2. You will need to update all packages which are stored in the incoming dir as diff --git a/scripts/cleanup b/scripts/cleanup index 21c913a..4fc9ea2 100755 --- a/scripts/cleanup +++ b/scripts/cleanup @@ -25,7 +25,7 @@ exec('ls ' . INCOMING_DIR, $files); $count = 0; foreach ($files as $pkgname) { - if (package_location($pkgname) != 'unsupported') { + if (!package_exists($pkgname)) { echo 'Removing ' . INCOMING_DIR . "$pkgname\n"; system('rm -r ' . INCOMING_DIR . $pkgname); $count++; diff --git a/support/schema/aur-schema.sql b/support/schema/aur-schema.sql index cd24f38..6af8167 100644 --- a/support/schema/aur-schema.sql +++ b/support/schema/aur-schema.sql @@ -1,5 +1,5 @@ -- The MySQL database layout for the AUR. Certain data --- is also included such as AccountTypes, PackageLocations, etc. +-- is also included such as AccountTypes, etc. -- DROP DATABASE AUR; CREATE DATABASE AUR; @@ -89,21 +89,6 @@ INSERT INTO PackageCategories (Category) VALUES ('x11'); INSERT INTO PackageCategories (Category) VALUES ('xfce'); --- The various repositories that a package could live in. --- -CREATE TABLE PackageLocations ( - ID TINYINT UNSIGNED NOT NULL AUTO_INCREMENT, - Location CHAR(32) NOT NULL, - PRIMARY KEY (ID) -); -INSERT INTO PackageLocations (Location) VALUES ('none'); -INSERT INTO PackageLocations (Location) VALUES ('unsupported'); -INSERT INTO PackageLocations (Location) VALUES ('community'); -INSERT INTO PackageLocations (Location) VALUES ('current'); -INSERT INTO PackageLocations (Location) VALUES ('extra'); -INSERT INTO PackageLocations (Location) VALUES ('unstable'); - - -- Information about the actual packages -- CREATE TABLE Packages ( @@ -116,7 +101,6 @@ CREATE TABLE Packages ( DummyPkg TINYINT UNSIGNED NOT NULL DEFAULT 0, -- 1=>dummy FSPath CHAR(255) NOT NULL DEFAULT '', License CHAR(40) NOT NULL DEFAULT '', - LocationID TINYINT UNSIGNED NOT NULL DEFAULT 1, NumVotes INTEGER UNSIGNED NOT NULL DEFAULT 0, OutOfDateTS BIGINT UNSIGNED NULL DEFAULT NULL, SubmittedTS BIGINT UNSIGNED NOT NULL, @@ -127,13 +111,11 @@ CREATE TABLE Packages ( PRIMARY KEY (ID), UNIQUE (Name), INDEX (CategoryID), - INDEX (LocationID), INDEX (DummyPkg), INDEX (NumVotes), INDEX (SubmitterUID), INDEX (MaintainerUID), FOREIGN KEY (CategoryID) REFERENCES PackageCategories(ID) ON DELETE NO ACTION, - FOREIGN KEY (LocationID) REFERENCES PackageLocations(ID) ON DELETE NO ACTION, FOREIGN KEY (SubmitterUID) REFERENCES Users(ID) ON DELETE NO ACTION, FOREIGN KEY (MaintainerUID) REFERENCES Users(ID) ON DELETE NO ACTION ); diff --git a/support/schema/gendummydata.py b/support/schema/gendummydata.py index 99f8195..a03b3e6 100755 --- a/support/schema/gendummydata.py +++ b/support/schema/gendummydata.py @@ -85,28 +85,21 @@ esc = db.escape_string # seen_users = {} seen_pkgs = {} -locations = {} categories = {} -location_keys = [] category_keys = [] user_keys = [] # some functions to generate random data # -def genVersion(location_id=0): +def genVersion(): ver = [] ver.append("%d" % random.randrange(0,10)) ver.append("%d" % random.randrange(0,20)) if random.randrange(0,2) == 0: ver.append("%d" % random.randrange(0,100)) - if location_id == 2: # the package is in the AUR - return ".".join(ver) + "-u%d" % random.randrange(1,11) - else: - return ".".join(ver) + "%d" % random.randrange(1,11) + return ".".join(ver) + "-u%d" % random.randrange(1,11) def genCategory(): return categories[category_keys[random.randrange(0,len(category_keys))]] -def genLocation(): - return locations[location_keys[random.randrange(0,len(location_keys))]] def genUID(): return seen_users[user_keys[random.randrange(0,len(user_keys))]] @@ -162,7 +155,7 @@ contents = None # Load package categories from database # -if DBUG: print "Loading package categories/locations..." +if DBUG: print "Loading package categories..." q = "SELECT * FROM PackageCategories" dbc.execute(q) row = dbc.fetchone() @@ -171,16 +164,6 @@ while row: row = dbc.fetchone() category_keys = categories.keys() -# Load package locations from database -# -q = "SELECT * FROM PackageLocations" -dbc.execute(q) -row = dbc.fetchone() -while row: - locations[row[1]] = row[0] - row = dbc.fetchone() -location_keys = locations.keys() - # done with the database # dbc.close() @@ -243,14 +226,10 @@ for p in seen_pkgs.keys(): if count % 20 == 0: # every so often, there are orphans... muid = 0 - location_id = genLocation() - if location_id == 1: # unsupported pkgs don't have a maintainer - muid = 0 - uuid = genUID() # the submitter/user - s = "INSERT INTO Packages (ID, Name, Version, CategoryID, LocationID, SubmittedTS, SubmitterUID, MaintainerUID) VALUES (%d, '%s', '%s', %d, %d, %d, %d, %d);\n" % (seen_pkgs[p], p, genVersion(location_id), - genCategory(), location_id, NOW, uuid, muid) + s = "INSERT INTO Packages (ID, Name, Version, CategoryID, SubmittedTS, SubmitterUID, MaintainerUID) VALUES (%d, '%s', '%s', %d, %d, %d, %d);\n" % (seen_pkgs[p], p, genVersion(), + genCategory(), NOW, uuid, muid) out.write(s) if count % 100 == 0: if DBUG: print ".", @@ -265,63 +244,34 @@ for p in seen_pkgs.keys(): s = "INSERT INTO PackageComments (PackageID, UsersID, Comments, CommentTS) VALUES (%d, %d, '%s', %d);\n" % (seen_pkgs[p], genUID(), fortune, now) out.write(s) - if location_id == 1: # Unsupported - just a PKGBUILD and maybe other stuff - others = random.randrange(0,3) - s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/PKGBUILD" % p, + others = random.randrange(0,3) + s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/PKGBUILD" % p, + random.randrange(0,999)) + out.write(s) + if others == 0: + s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/%s.patch" % (p,p), random.randrange(0,999)) out.write(s) - if others == 0: - s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/%s.patch" % (p,p), - random.randrange(0,999)) - out.write(s) - elif others == 1: - s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/%s.patch" % (p,p), - random.randrange(0,999)) - out.write(s) - s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/arch.patch" % p, - random.randrange(0,999)) - out.write(s) + elif others == 1: + s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/%s.patch" % (p,p), + random.randrange(0,999)) + out.write(s) + s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/arch.patch" % p, + random.randrange(0,999)) + out.write(s) - elif others == 2: - s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/%s.patch" % (p,p), - random.randrange(0,999)) - out.write(s) - s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/arch.patch" % p, - random.randrange(0,999)) - out.write(s) - s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/%s.install" % (p,p), - random.randrange(0,999)) - out.write(s) + elif others == 2: + s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/%s.patch" % (p,p), + random.randrange(0,999)) + out.write(s) + s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/arch.patch" % p, + random.randrange(0,999)) + out.write(s) + s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], "/home/aur/incoming/%s/%s.install" % (p,p), + random.randrange(0,999)) + out.write(s) - else: - # Create package contents - # - num_files = random.randrange(PKG_FILES[0], PKG_FILES[1]) - files = {} - for f in range(num_files): - loc = RANDOM_PATHS[random.randrange(len(RANDOM_PATHS))] - if "lib" in loc: - path = loc + "/lib" + p + ".so" - elif "man" in loc: - path = loc + "/" + p + "." + loc[-1] + ".gz" - elif "share" in loc: - path = loc + "/" + p + "/sounds/" + p + ".wav" - elif "profile" in loc: - path = loc + "/" + p + ".sh" - elif "rc.d" in loc: - path = loc + "/" + p - elif "etc" in loc: - path = loc + "/" + p + ".conf" - elif "opt" in loc: - path = loc + "/" + p + "/bin/" + p - else: - path = loc + "/" + p - if not files.has_key(path): - files[path] = 1 - s = "INSERT INTO PackageContents (PackageID, FSPath, FileSize) VALUES (%d, '%s', %d);\n" % (seen_pkgs[p], path, - random.randrange(0,99999999)) - out.write(s) if DBUG: print "." # Cast votes diff --git a/web/html/pkgsubmit.php b/web/html/pkgsubmit.php index c39e2f9..3d77f72 100644 --- a/web/html/pkgsubmit.php +++ b/web/html/pkgsubmit.php @@ -277,7 +277,7 @@ if ($_COOKIE["AURSID"]): } # Update package data - $q = sprintf("UPDATE Packages SET ModifiedTS = UNIX_TIMESTAMP(), Name = '%s', Version = '%s-%s', License = '%s', Description = '%s', URL = '%s', LocationID = 2, FSPath = '%s', OutOfDateTS = NULL WHERE ID = %d", + $q = sprintf("UPDATE Packages SET ModifiedTS = UNIX_TIMESTAMP(), Name = '%s', Version = '%s-%s', License = '%s', Description = '%s', URL = '%s', FSPath = '%s', OutOfDateTS = NULL WHERE ID = %d", mysql_real_escape_string($new_pkgbuild['pkgname']), mysql_real_escape_string($new_pkgbuild['pkgver']), mysql_real_escape_string($new_pkgbuild['pkgrel']), @@ -322,7 +322,7 @@ if ($_COOKIE["AURSID"]): $uid = uid_from_sid($_COOKIE["AURSID"]); # This is a brand new package - $q = sprintf("INSERT INTO Packages (Name, License, Version, CategoryID, Description, URL, LocationID, SubmittedTS, SubmitterUID, MaintainerUID, FSPath) VALUES ('%s', '%s', '%s-%s', %d, '%s', '%s', 2, UNIX_TIMESTAMP(), %d, %d, '%s')", + $q = sprintf("INSERT INTO Packages (Name, License, Version, CategoryID, Description, URL, SubmittedTS, SubmitterUID, MaintainerUID, FSPath) VALUES ('%s', '%s', '%s-%s', %d, '%s', '%s', UNIX_TIMESTAMP(), %d, %d, '%s')", mysql_real_escape_string($new_pkgbuild['pkgname']), mysql_real_escape_string($new_pkgbuild['license']), mysql_real_escape_string($new_pkgbuild['pkgver']), @@ -398,7 +398,6 @@ html_header("Submit"); if (ini_get("file_uploads")): $pkg_categories = pkgCategories(); - $pkg_locations = pkgLocations(); ?> <form action='pkgsubmit.php' method='post' enctype='multipart/form-data'> diff --git a/web/lib/aurjson.class.php b/web/lib/aurjson.class.php index 0868cb2..285aad7 100644 --- a/web/lib/aurjson.class.php +++ b/web/lib/aurjson.class.php @@ -29,7 +29,7 @@ class AurJSON { $this->fields = array( 'Packages.ID', 'Name', 'Version', 'CategoryID', - 'Description', 'LocationID', 'URL', 'CONCAT("' . + 'Description', 'URL', 'CONCAT("' . mysql_real_escape_string(URL_DIR) . '", Name, "/", Name, ".tar.gz") AS URLPath', 'License', 'NumVotes', '(OutOfDateTS IS NOT NULL) AS OutOfDate' diff --git a/web/lib/pkgfuncs.inc b/web/lib/pkgfuncs.inc index c701348..997c453 100644 --- a/web/lib/pkgfuncs.inc +++ b/web/lib/pkgfuncs.inc @@ -80,37 +80,6 @@ function pkgCategories() { return $cats; } -# grab the current list of PackageLocations -# -function pkgLocations() { - $locs = array(); - $dbh = db_connect(); - $q = "SELECT * FROM PackageLocations WHERE ID != 1 AND ID < 4 "; - $q.= "ORDER BY Location ASC"; - $result = db_query($q, $dbh); - if ($result) { - while ($row = mysql_fetch_row($result)) { - $locs[$row[0]] = $row[1]; - } - } - return $locs; -} - -# Return the repository name for a particular package. -function package_location($name="") { - if (!$name) {return NULL;} - $dbh = db_connect(); - $q = "SELECT PackageLocations.Location FROM Packages "; - $q.= "LEFT JOIN PackageLocations ON "; - $q.= "Packages.LocationID = PackageLocations.ID "; - $q.= "WHERE Name = '".mysql_real_escape_string($name)."' "; - $q.= "AND DummyPkg = 0"; - $result = db_query($q, $dbh); - if (!$result) {return NULL;} - $row = mysql_fetch_row($result); - return $row[0]; -} - # check to see if the package name exists # function package_exists($name="") { @@ -324,10 +293,9 @@ function package_details($id=0, $SID="") { $atype = account_from_sid($SID); $uid = uid_from_sid($SID); - $q = "SELECT Packages.*,Location,Category "; - $q.= "FROM Packages,PackageLocations,PackageCategories "; - $q.= "WHERE Packages.LocationID = PackageLocations.ID "; - $q.= "AND Packages.CategoryID = PackageCategories.ID "; + $q = "SELECT Packages.*,Category "; + $q.= "FROM Packages,PackageCategories "; + $q.= "WHERE Packages.CategoryID = PackageCategories.ID "; $q.= "AND Packages.ID = " . intval($id); $dbh = db_connect(); $results = db_query($q, $dbh); @@ -373,15 +341,13 @@ function package_details($id=0, $SID="") { * request vars: * O - starting result number * PP - number of search hits per page - * L - package location ID number * C - package category ID number * K - package search string * SO - search hit sort order: * values: a - ascending * d - descending * SB - sort search hits by: - * values: l - package location - * c - package category + * values: c - package category * n - package name * v - number of votes * m - maintainer username @@ -415,8 +381,6 @@ function pkg_search_page($SID="") { // grab info for user if they're logged in if ($SID) $myuid = uid_from_sid($SID); - // get a list of package locations - $locs = pkgLocations(); // get a list of package categories $cats = pkgCategories(); //meow @@ -455,7 +419,6 @@ function pkg_search_page($SID="") { } $q .= "Users.Username AS Maintainer, PackageCategories.Category, - PackageLocations.Location, Packages.Name, Packages.Version, Packages.Description, Packages.NumVotes, Packages.ID, Packages.OutOfDateTS @@ -469,15 +432,10 @@ function pkg_search_page($SID="") { } $q .= "LEFT JOIN PackageCategories ON (Packages.CategoryID = PackageCategories.ID) - LEFT JOIN PackageLocations - ON (Packages.LocationID = PackageLocations.ID) WHERE Packages.DummyPkg = 0 "; - // TODO: possibly do string matching on category and - // location to make request variable values more sensible - if (intval($_GET["L"])) { - $q .= "AND Packages.LocationID = ".intval($_GET["L"])." "; - } + // TODO: possibly do string matching on category + // to make request variable values more sensible if (intval($_GET["C"])) { $q.= "AND Packages.CategoryID = ".intval($_GET["C"])." "; } @@ -515,16 +473,12 @@ function pkg_search_page($SID="") { $order = $_GET["SO"] == 'd' ? 'DESC' : 'ASC'; - $q_sort = "ORDER BY Name ".$order.", LocationID ASC, CategoryID DESC "; + $q_sort = "ORDER BY Name ".$order.", CategoryID DESC "; switch ($_GET["SB"]) { case 'c': - $q_sort = "ORDER BY CategoryID ".$order.", Name ASC, LocationID ASC "; + $q_sort = "ORDER BY CategoryID ".$order.", Name ASC "; $_GET["SB"] = 'c'; break; - case 'l': - $q_sort = "ORDER BY LocationID ".$order.", Name ASC, CategoryID DESC "; - $_GET["SB"] = 'l'; - break; case 'v': $q_sort = "ORDER BY NumVotes ".$order.", Name ASC, CategoryID DESC "; $_GET["SB"] = 'v'; @@ -542,11 +496,11 @@ function pkg_search_page($SID="") { $_GET["SB"] = 'o'; break; case 'm': - $q_sort = "ORDER BY Maintainer ".$order.", Name ASC, LocationID ASC "; + $q_sort = "ORDER BY Maintainer ".$order.", Name ASC "; $_GET["SB"] = 'm'; break; case 'a': - $q_sort = "ORDER BY GREATEST(SubmittedTS,ModifiedTS) ".$order.", Name ASC, LocationID ASC "; + $q_sort = "ORDER BY GREATEST(SubmittedTS,ModifiedTS) ".$order.", Name ASC "; $_GET["SB"] = 'a'; break; default: @@ -719,10 +673,8 @@ function pkg_delete ($atype, $ids) { # Only grab Unsupported packages that "we" own or are not owned at all $ids_to_delete = array(); - $q = "SELECT Packages.ID FROM Packages, PackageLocations "; + $q = "SELECT Packages.ID FROM Packages "; $q.= "WHERE Packages.ID IN (" . $delete . ") "; - $q.= "AND Packages.LocationID = PackageLocations.ID "; - $q.= "AND PackageLocations.Location = 'unsupported' "; # If they're a TU or dev, can delete if ($atype == "Trusted User" || $atype == "Developer") { @@ -816,7 +768,6 @@ function pkg_adopt ($atype, $ids, $action = True) { if ($action && $atype == "User") { # Regular users may only adopt orphan packages from unsupported $q.= "AND $field = 0 "; - $q.= "AND LocationID = 2 "; } else if ($atype == "User") { $q.= "AND $field = " . uid_from_sid($_COOKIE["AURSID"]); } @@ -1047,20 +998,18 @@ function pkg_change_category($atype) { return __("Missing package ID."); } - # Verify package ownership and location + # Verify package ownership $dbh = db_connect(); $q = "SELECT Packages.MaintainerUID,"; - $q.= "PackageLocations.Location "; $q.= "FROM Packages "; - $q.= "LEFT JOIN PackageLocations ON Packages.LocationID = PackageLocations.ID "; $q.= "WHERE Packages.ID = ".$pid; $result = db_query($q, $dbh); echo mysql_error(); $pkg = mysql_fetch_assoc($result); $uid = uid_from_sid($_COOKIE["AURSID"]); - if ($pkg["Location"] == "unsupported" and ($uid == $pkg["MaintainerUID"] or - ($atype == "Developer" or $atype == "Trusted User"))) { + if ($uid == $pkg["MaintainerUID"] or + ($atype == "Developer" or $atype == "Trusted User")) { $q = "UPDATE Packages "; $q.= "SET CategoryID = ".intval($category_id)." "; $q.= "WHERE ID = ".intval($pid); diff --git a/web/lib/stats.inc b/web/lib/stats.inc index 6a76558..f42e417 100644 --- a/web/lib/stats.inc +++ b/web/lib/stats.inc @@ -54,9 +54,9 @@ function user_table($user, $dbh) { global $apc_prefix; $escuser = mysql_real_escape_string($user); - $base_q = "SELECT count(*) FROM Packages,PackageLocations,Users WHERE Packages.MaintainerUID = Users.ID AND Packages.LocationID = PackageLocations.ID AND PackageLocations.Location = '%s' AND Users.Username='" . $escuser . "'"; + $base_q = "SELECT count(*) FROM Packages,Users WHERE Packages.MaintainerUID = Users.ID AND Users.Username='" . $escuser . "'"; - $maintainer_unsupported_count = db_cache_value(sprintf($base_q, 'unsupported'), $dbh, + $maintainer_unsupported_count = db_cache_value($base_q, $dbh, $apc_prefix . 'user_unsupported_count:' . $escuser); $q = "SELECT count(*) FROM Packages,Users WHERE Packages.OutOfDateTS IS NOT NULL AND Packages.MaintainerUID = Users.ID AND Users.Username='" . $escuser . "'"; @@ -74,7 +74,7 @@ function general_stats_table($dbh) { global $apc_prefix; # AUR statistics - $q = "SELECT count(*) FROM Packages,PackageLocations WHERE Packages.LocationID = PackageLocations.ID AND PackageLocations.Location = 'unsupported'"; + $q = "SELECT count(*) FROM Packages WHERE DummyPkg = 0"; $unsupported_count = db_cache_value($q, $dbh, $apc_prefix . 'unsupported_count'); $q = "SELECT count(*) from Users"; diff --git a/web/template/pkg_details.php b/web/template/pkg_details.php index 8aba3af..c1c3e96 100644 --- a/web/template/pkg_details.php +++ b/web/template/pkg_details.php @@ -1,13 +1,13 @@ <?php $pkgid = intval($_REQUEST['ID']); -if ($row["Location"] == "unsupported" and ($uid == $row["MaintainerUID"] or - ($atype == "Developer" or $atype == "Trusted User"))) { +if ($uid == $row["MaintainerUID"] or + ($atype == "Developer" or $atype == "Trusted User")) { $catarr = pkgCategories(); $edit_cat = "<form method='POST' action='packages.php?ID=".$pkgid."'>\n"; $edit_cat.= "<input type='hidden' name='action' value='do_ChangeCategory'>"; - $edit_cat.= $row['Location']." :: "; + $edit_cat.= "Category: "; $edit_cat.= "<select name='category_id'>\n"; foreach ($catarr as $cid => $catname) { $edit_cat.= "<option value='$cid'"; @@ -21,7 +21,7 @@ if ($row["Location"] == "unsupported" and ($uid == $row["MaintainerUID"] or } else { - $edit_cat = $row['Location']." :: ".$row['Category']; + $edit_cat = "Category: ".$row['Category']; } if ($row["MaintainerUID"]) { @@ -76,11 +76,9 @@ $out_of_date_time = ($row["OutOfDateTS"] == 0) ? $msg : gmdate("r", intval($row[ <p><span class='f3'> <?php - if ($row['LocationID'] == 2) { - $urlpath = URL_DIR . $row['Name']; - print "<a href='$urlpath/" . $row['Name'] . ".tar.gz'>".__("Tarball")."</a> :: "; - print "<a href='$urlpath/PKGBUILD'>".__("PKGBUILD")."</a></span>"; - } + $urlpath = URL_DIR . $row['Name']; + print "<a href='$urlpath/" . $row['Name'] . ".tar.gz'>".__("Tarball")."</a> :: "; + print "<a href='$urlpath/PKGBUILD'>".__("PKGBUILD")."</a></span>"; if ($row["OutOfDateTS"] !== NULL) { echo "<br /><span class='f6'>".__("This package has been flagged out of date.")." (${out_of_date_time})</span>"; @@ -172,10 +170,8 @@ $out_of_date_time = ($row["OutOfDateTS"] == 0) ? $msg : gmdate("r", intval($row[ else { $src = $src[0]; # It is presumably an internal source - if ($row["LocationID"] == 2) { - echo "<span class='f8'>$src</span>"; - echo "<br />\n"; - } + echo "<span class='f8'>$src</span>"; + echo "<br />\n"; } } ?> diff --git a/web/template/pkg_search_form.php b/web/template/pkg_search_form.php index c616c7b..fdeb01b 100644 --- a/web/template/pkg_search_form.php +++ b/web/template/pkg_search_form.php @@ -56,7 +56,7 @@ <label><?php print __("Sort by"); ?></label> <select name='SB'> <?php - $sortby = array('n' => __('Name'), 'c' => __('Category'), 'l' => __('Location'), 'v' => __('Votes'), 'w' => __('Voted'), 'o' => __('Notify'), 'm' => __('Maintainer'), 'a' => __('Age')); + $sortby = array('n' => __('Name'), 'c' => __('Category'), 'v' => __('Votes'), 'w' => __('Voted'), 'o' => __('Notify'), 'm' => __('Maintainer'), 'a' => __('Age')); foreach ($sortby as $k => $v): if ($_REQUEST['SB'] == $k): ?> diff --git a/web/template/pkg_search_results.php b/web/template/pkg_search_results.php index b8f2e14..4830ca8 100644 --- a/web/template/pkg_search_results.php +++ b/web/template/pkg_search_results.php @@ -16,9 +16,6 @@ <?php endif; ?> <th style='border-bottom: #666 1px solid; vertical-align: bottom'><span class='f2'> - <a href='?<?php print mkurl('SB=l&SO=' . $SO_next) ?>'><?php print __("Location") ?></a> - </span></th> - <th style='border-bottom: #666 1px solid; vertical-align: bottom'><span class='f2'> <a href='?<?php print mkurl('SB=c&SO=' . $SO_next) ?>'><?php print __("Category") ?></a> </span></th> <th style='border-bottom: #666 1px solid; vertical-align: bottom; text-align: center;'><span class='f2'> @@ -52,7 +49,6 @@ for ($i = 0; $row = mysql_fetch_assoc($result); $i++) { <?php if ($SID): ?> <td class='<?php print $c ?>'><input type='checkbox' name='IDs[<?php print $row["ID"] ?>]' value='1'></td> <?php endif; ?> - <td class='<?php print $c ?>'><span class='f5'><span class='blue'><?php print $row["Location"] ?></span></span></td> <td class='<?php print $c ?>'><span class='f5'><span class='blue'><?php print $row["Category"] ?></span></span></td> <td class='<?php print $c ?>'><span class='f4'><a href='packages.php?ID=<?php print $row["ID"] ?>'><span class='black'><?php print $row["Name"] ?> <?php print $row["Version"] ?></span></a></span></td> <td class='<?php print $c ?>' style="text-align: right"><span class='f5'><span class='blue'><?php print $row["NumVotes"] ?></span></span></td> -- 1.7.3.5
On Mon, Jan 31, 2011 at 11:18:15AM -0600, Dan McGee wrote:
We don't need this anymore since all packages managed here are well...managed here. Rip out all of the places we were using this field, many of which depended on the magic value '2' anyway.
On the display side of things, we had a column that was always showing 'unsupported' that is now gone, and you can no longer sort by this column.
Signed-off-by: Dan McGee <dan@archlinux.org> ---
Totally embarassing there; this is the correct patch with the necessary corrections.
Nothing to be ashamed of :p Rebased my working tree, looks fine now. Thanks! :)
On Sat 29 Jan 2011 15:50 -0600, Dan McGee wrote:
LocationID, DummyPkg- are these used anymore on the Packages table? If not, getting rid of these would do wonders toward actually allowing MySQL to use indexes for some queries, which it has been unable to do for a long time. I've pasted some of the slow query log analysis below from stuff this month, it is pretty obvious where work should be focused.
Killing these two fields and then replacing the SQL_CALC_FOUND_ROWS with a second count query should take care of nearly every slow query that currently happens.
Also potentially worth attacking is the use of GREATEST(SubmittedTS,ModifiedTS) - since the wonderful MySQL has no functional indexes using this in a query as an order by clause results in full table scans.
LocationID doesn't really have much use anymore. DummyPkg is still used to track dependencies, but perhaps another strategy could be used to do that. Thanks for looking at that, and the patches. :D
participants (3)
-
Dan McGee
-
Loui Chang
-
Lukas Fleischer