[aur-dev] [PATCH] Cache all front page stats in APC if available
Use the APC cache to store all of the counts and the recently updated package list in a cache, which cuts down on the number of database queries needed. If the data isn't perfectly up to date we will survive. This version of the patch will also cache the relevant counts for individual logged-in users and is more careful about checking whether the value actually exists in the cache by using the status reference to apc_fetch(). Signed-off-by: Dan McGee <dan@archlinux.org> --- Sorry I keep tweaking this patch, but I've extensively tested this version locally with APC both on and off and it seems to work quite well. We even cache some user-specific stuff now for what its worth. Since our cache expires relatively quickly, this isn't going to be a used a lot but when it is it will save some database load. -Dan web/lib/stats.inc | 53 +++++++++++++++++++++++----------- web/template/stats/updates_table.php | 5 +-- 2 files changed, 38 insertions(+), 20 deletions(-) diff --git a/web/lib/stats.inc b/web/lib/stats.inc index 6fbc033..e2b7f90 100644 --- a/web/lib/stats.inc +++ b/web/lib/stats.inc @@ -2,10 +2,39 @@ include_once('aur.inc'); +# Check if APC extension is loaded +if (!defined('EXTENSION_LOADED_APC')) + define('EXTENSION_LOADED_APC', extension_loaded('apc')); +$apc_prefix = 'aur:'; + +# run a simple db query, retrieving and/or caching the value if APC +# is available for use +# +function db_cache_value($dbq, $dbh, $key) +{ + if(!(EXTENSION_LOADED_APC && ($ret = apc_fetch($key)))) { + $result = db_query($dbq, $dbh); + $row = mysql_fetch_row($result); + $ret = $row[0]; + # set the TTL here in seconds: 300 seconds = 5 minutes + apc_store($key, $ret, 300); + } + return $ret; +} + function updates_table($dbh) { - $q = 'SELECT * FROM Packages WHERE DummyPkg != 1 ORDER BY GREATEST(SubmittedTS,ModifiedTS) DESC LIMIT 0 , 10'; - $newest_packages = db_query($q, $dbh); + $key = $apc_prefix . 'recent_updates'; + if(!(EXTENSION_LOADED_APC && ($newest_packages = apc_fetch($key)))) { + $q = 'SELECT * FROM Packages WHERE DummyPkg != 1 ORDER BY GREATEST(SubmittedTS,ModifiedTS) DESC LIMIT 0 , 10'; + $result = db_query($q, $dbh); + + $newest_packages = new ArrayObject(); + while ($row = mysql_fetch_assoc($result)) { + $newest_packages->append($row); + } + apc_store($key, $newest_packages, 300); + } include('stats/updates_table.php'); } @@ -42,30 +71,20 @@ function general_stats_table($dbh) { # AUR statistics $q = "SELECT count(*) FROM Packages,PackageLocations WHERE Packages.LocationID = PackageLocations.ID AND PackageLocations.Location = 'unsupported'"; - $result = db_query($q, $dbh); - $row = mysql_fetch_row($result); - $unsupported_count = $row[0]; + $unsupported_count = db_cache_value($q, $dbh, $apc_prefix . 'unsupported_count'); $q = "SELECT count(*) FROM Packages,PackageLocations WHERE Packages.LocationID = PackageLocations.ID AND PackageLocations.Location = 'community'"; - $result = db_query($q, $dbh); - $row = mysql_fetch_row($result); - $community_count = $row[0]; + $community_count = db_cache_value($q, $dbh, $apc_prefix . 'community_count'); $q = "SELECT count(*) from Users"; - $result = db_query($q, $dbh); - $row = mysql_fetch_row($result); - $user_count = $row[0]; + $user_count = db_cache_value($q, $dbh, $apc_prefix . 'user_count'); $q = "SELECT count(*) from Users,AccountTypes WHERE Users.AccountTypeID = AccountTypes.ID AND AccountTypes.AccountType = 'Trusted User'"; - $result = db_query($q, $dbh); - $row = mysql_fetch_row($result); - $tu_count = $row[0]; + $tu_count = db_cache_value($q, $dbh, $apc_prefix . 'tu_count'); $targstamp = intval(strtotime("-7 days")); $q = "SELECT count(*) from Packages WHERE (Packages.SubmittedTS >= $targstamp OR Packages.ModifiedTS >= $targstamp)"; - $result = db_query($q, $dbh); - $row = mysql_fetch_row($result); - $update_count = $row[0]; + $update_count = db_cache_value($q, $dbh, $apc_prefix . 'update_count'); include('stats/general_stats_table.php'); } diff --git a/web/template/stats/updates_table.php b/web/template/stats/updates_table.php index e1eb888..9d1af01 100644 --- a/web/template/stats/updates_table.php +++ b/web/template/stats/updates_table.php @@ -6,8 +6,7 @@ </th> </tr> -<?php while ($row = mysql_fetch_assoc($newest_packages)): ?> - +<?php foreach ($newest_packages->getIterator() as $row): ?> <tr> <td class="boxSoft"> <span class="f4"><span class="blue"> @@ -34,7 +33,7 @@ endif; </td> </tr> -<?php endwhile; ?> +<?php endforeach; ?> </table> -- 1.6.1
On Sun, Jan 04, 2009 at 01:43:58PM -0600, Dan McGee wrote:
Use the APC cache to store all of the counts and the recently updated package list in a cache, which cuts down on the number of database queries needed. If the data isn't perfectly up to date we will survive.
This version of the patch will also cache the relevant counts for individual logged-in users and is more careful about checking whether the value actually exists in the cache by using the status reference to apc_fetch().
Signed-off-by: Dan McGee <dan@archlinux.org> ---
Sorry I keep tweaking this patch, but I've extensively tested this version locally with APC both on and off and it seems to work quite well.
We even cache some user-specific stuff now for what its worth. Since our cache expires relatively quickly, this isn't going to be a used a lot but when it is it will save some database load.
No problem. I'd definitely like to understand and test this before pushing it though, so anything that reduces problems is great. The stats aren't exactly a critical feature though, so I guess there's no need to worry too much.
On Sun, Jan 04, 2009 at 01:43:58PM -0600, Dan McGee wrote:
Use the APC cache to store all of the counts and the recently updated package list in a cache, which cuts down on the number of database queries needed. If the data isn't perfectly up to date we will survive.
This version of the patch will also cache the relevant counts for individual logged-in users and is more careful about checking whether the value actually exists in the cache by using the status reference to apc_fetch().
Signed-off-by: Dan McGee <dan@archlinux.org> ---
Sorry I keep tweaking this patch, but I've extensively tested this version locally with APC both on and off and it seems to work quite well.
I pushed this as is, but I had to commit a fix because it was causing this error: PHP Fatal error: Call to undefined function apc_store() in /home/louipc/devel/git/aur/web/lib/stats.inc on line 36 Are you actually testing your patches?
On Thu, Jan 8, 2009 at 12:14 PM, Loui Chang <louipc.ist@gmail.com> wrote:
On Sun, Jan 04, 2009 at 01:43:58PM -0600, Dan McGee wrote:
Use the APC cache to store all of the counts and the recently updated package list in a cache, which cuts down on the number of database queries needed. If the data isn't perfectly up to date we will survive.
This version of the patch will also cache the relevant counts for individual logged-in users and is more careful about checking whether the value actually exists in the cache by using the status reference to apc_fetch().
Signed-off-by: Dan McGee <dan@archlinux.org> ---
Sorry I keep tweaking this patch, but I've extensively tested this version locally with APC both on and off and it seems to work quite well.
I pushed this as is, but I had to commit a fix because it was causing this error:
PHP Fatal error: Call to undefined function apc_store() in /home/louipc/devel/git/aur/web/lib/stats.inc on line 36
Are you actually testing your patches?
Definitely- I'm running the whole setup locally. I swear I had wrapped these calls as you ended up doing, and I wonder if I lost work during one of my iterations. Thanks for making the fixes. Since we only run the AUR in one place and I know it has APC available, I was a bit less intensive about testing the not-installed case although I tried to accommodate it. -Dan
On Fri, Jan 09, 2009 at 06:26:00AM -0600, Dan McGee wrote:
On Thu, Jan 8, 2009 at 12:14 PM, Loui Chang <louipc.ist@gmail.com> wrote:
Are you actually testing your patches?
Definitely- I'm running the whole setup locally. I swear I had wrapped these calls as you ended up doing, and I wonder if I lost work during one of my iterations. Thanks for making the fixes.
Since we only run the AUR in one place and I know it has APC available, I was a bit less intensive about testing the not-installed case although I tried to accommodate it.
Alright. Good to hear. I was just concerned because I was altering a lot of these these patches because they were introducing regressions. Cheers.
participants (3)
-
Dan McGee
-
Dan McGee
-
Loui Chang