Hi all,
as you should know I have prepared the update to the new version of FluxBB. This is now mostly done except some minor adjustments to the configurable main menu.
I have put a rough plan on my user page at http://wiki.archlinux.org/index.php/User:Pierre I have tested this locally and it is more or less painless. The only problem I see is that the update script is quite inefficient; so the forums will be down for an hour or longer.
Should we pick a special date here? I'd also need someone who is more familiar with the apache config to alter the permission for some files.
Some additional notes: * I didn't get a response from the guy with the duplicate account yet * Obviously some previous migrations failed so I need to cleanup the database schema. (there are duplicate or unused keys)
-- Pierre Schmitz, https://users.archlinux.de/~pierre
On 07/20/2010 06:15 PM, Pierre Schmitz wrote:
Hi all,
as you should know I have prepared the update to the new version of FluxBB. This is now mostly done except some minor adjustments to the configurable main menu.
I have put a rough plan on my user page at http://wiki.archlinux.org/index.php/User:Pierre I have tested this locally and it is more or less painless. The only problem I see is that the update script is quite inefficient; so the forums will be down for an hour or longer.
Should we pick a special date here? I'd also need someone who is more familiar with the apache config to alter the permission for some files.
i say to do it when you have time. just add a pretty lolcat and i'll be happy
Some additional notes:
- I didn't get a response from the guy with the duplicate account yet
delete it if is problematic?
- Obviously some previous migrations failed so I need to cleanup the
database schema. (there are duplicate or unused keys)
-- Pierre Schmitz, https://users.archlinux.de/~pierre
On Tue, 20 Jul 2010 18:50:48 +0300, Ionuț Bîru ibiru@archlinux.org wrote:
On 07/20/2010 06:15 PM, Pierre Schmitz wrote:
- I didn't get a response from the guy with the duplicate account yet
delete it if is problematic?
No, I'll just rename the user and send him a mail with the new login. It's no problem to rename again if needed.
On Tue, Jul 20, 2010 at 10:15 AM, Pierre Schmitz pierre@archlinux.de wrote:
Hi all,
as you should know I have prepared the update to the new version of FluxBB. This is now mostly done except some minor adjustments to the configurable main menu.
I have put a rough plan on my user page at http://wiki.archlinux.org/index.php/User:Pierre I have tested this locally and it is more or less painless. The only problem I see is that the update script is quite inefficient; so the forums will be down for an hour or longer.
Should we pick a special date here? I'd also need someone who is more familiar with the apache config to alter the permission for some files.
Some additional notes:
- I didn't get a response from the guy with the duplicate account yet
- Obviously some previous migrations failed so I need to cleanup the
database schema. (there are duplicate or unused keys)
Want to do it sometime Thursday or Friday while some of us are at ArchCon? I'm willing to help out and I have sudo rights and such to edit everything necessary.
We also do not have enough room to dump and reload a second copy of the DB. Any ideas on this, anyone? We didn't think very straight when we were sizing /var on a DB machine...
-Dan
On Tue, 20 Jul 2010 10:59:35 -0500, Dan McGee dpmcgee@gmail.com wrote:
Want to do it sometime Thursday or Friday while some of us are at ArchCon? I'm willing to help out and I have sudo rights and such to edit everything necessary.
I need to see if I have time at thursday. I also have sudo rights but not that much experience with the apache config. But I could look it up hopefully.
We also do not have enough room to dump and reload a second copy of the DB. Any ideas on this, anyone? We didn't think very straight when we were sizing /var on a DB machine...
Good catch. But it should just fit. There is 3.2GB free disc space and the bbs database takes 1.2GB. Once everything is running the phpbb database can be removed. (I can keep a local dump file in my home dir for some time though...just in case)
On Tue, Jul 20, 2010 at 11:51 AM, Pierre Schmitz pierre@archlinux.de wrote:
On Tue, 20 Jul 2010 10:59:35 -0500, Dan McGee dpmcgee@gmail.com wrote:
Want to do it sometime Thursday or Friday while some of us are at ArchCon? I'm willing to help out and I have sudo rights and such to edit everything necessary.
I need to see if I have time at thursday. I also have sudo rights but not that much experience with the apache config. But I could look it up hopefully.
We also do not have enough room to dump and reload a second copy of the DB. Any ideas on this, anyone? We didn't think very straight when we were sizing /var on a DB machine...
Good catch. But it should just fit. There is 3.2GB free disc space and the bbs database takes 1.2GB. Once everything is running the phpbb database can be removed. (I can keep a local dump file in my home dir for some time though...just in case)
No, that is not how InnoDB works which we recently switched to for these tables. Once the data file grows we do not get the space back, period, unless we do a complete dump and load of every database. Log space needs to be accounted for, which makes our weekly swing of used space vary by 27%: https://stats.archlinux.org/munin/archlinux.org/gudrun/df.html
-Dan
On Tue, 20 Jul 2010 12:00:19 -0500, Dan McGee dpmcgee@gmail.com wrote:
On Tue, Jul 20, 2010 at 11:51 AM, Pierre Schmitz pierre@archlinux.de wrote:
On Tue, 20 Jul 2010 10:59:35 -0500, Dan McGee dpmcgee@gmail.com
We also do not have enough room to dump and reload a second copy of the DB. Any ideas on this, anyone? We didn't think very straight when we were sizing /var on a DB machine...
Good catch. But it should just fit. There is 3.2GB free disc space and the bbs database takes 1.2GB. Once everything is running the phpbb database can be removed. (I can keep a local dump file in my home dir for some time though...just in case)
No, that is not how InnoDB works which we recently switched to for these tables. Once the data file grows we do not get the space back, period, unless we do a complete dump and load of every database. Log space needs to be accounted for, which makes our weekly swing of used space vary by 27%: https://stats.archlinux.org/munin/archlinux.org/gudrun/df.html
Ah innodb ok. That saves all dbs in one file. (I was using myisam here) Hmm, we might run into a problem with our var partition some day...soon.
Well, let's change the plan to this: * make a backup of the db * rename phpbb to fluxbb (a rename should be painless, right) * .. * run the update script on the live db
For the time frame: I got time now till late tonight (let's say now + 5h) or tomorrow. For Thursday I probably don't have enough time if anything goes wrong. Otherwise some day next week.
On Tue, Jul 20, 2010 at 12:19 PM, Pierre Schmitz pierre@archlinux.de wrote:
On Tue, 20 Jul 2010 12:00:19 -0500, Dan McGee dpmcgee@gmail.com wrote:
On Tue, Jul 20, 2010 at 11:51 AM, Pierre Schmitz pierre@archlinux.de wrote:
On Tue, 20 Jul 2010 10:59:35 -0500, Dan McGee dpmcgee@gmail.com
We also do not have enough room to dump and reload a second copy of the DB. Any ideas on this, anyone? We didn't think very straight when we were sizing /var on a DB machine...
Good catch. But it should just fit. There is 3.2GB free disc space and the bbs database takes 1.2GB. Once everything is running the phpbb database can be removed. (I can keep a local dump file in my home dir for some time though...just in case)
No, that is not how InnoDB works which we recently switched to for these tables. Once the data file grows we do not get the space back, period, unless we do a complete dump and load of every database. Log space needs to be accounted for, which makes our weekly swing of used space vary by 27%: https://stats.archlinux.org/munin/archlinux.org/gudrun/df.html
Ah innodb ok. That saves all dbs in one file. (I was using myisam here) Hmm, we might run into a problem with our var partition some day...soon.
Well, let's change the plan to this:
- make a backup of the db
- rename phpbb to fluxbb (a rename should be painless, right)
- ..
- run the update script on the live db
For the time frame: I got time now till late tonight (let's say now + 5h) or tomorrow. For Thursday I probably don't have enough time if anything goes wrong. Otherwise some day next week.
I probably can't do it tomorrow unless it is way late my time as that is when I fly to Toronto. I also am booked out the rest of today through 5 hours from now so that is out too. :/ Thursday might work if we do it afternoon over here/evening for you, but you say you might not have a ton of time. If things do go wrong I will probably be around enough people that can help out at least.
Maybe we can shoot for some day next week.
-Dan
On Tue, 20 Jul 2010 12:29:43 -0500, Dan McGee dpmcgee@gmail.com wrote:
Maybe we can shoot for some day next week.
Yes, there is no real hurry. This also gives some time to prepare some stuff like resizing the partition as Thomas suggested.
Am 20.07.2010 17:59, schrieb Dan McGee:
We also do not have enough room to dump and reload a second copy of the DB. Any ideas on this, anyone? We didn't think very straight when we were sizing /var on a DB machine...
If you need more space, just say the word.
On Tue, Jul 20, 2010 at 3:48 PM, Thomas Bächler thomas@archlinux.org wrote:
Am 20.07.2010 17:59, schrieb Dan McGee:
We also do not have enough room to dump and reload a second copy of the DB. Any ideas on this, anyone? We didn't think very straight when we were sizing /var on a DB machine...
If you need more space, just say the word.
Is it a resizable volume, and can you do it when it is online? I don't know completely how you guys set up the root partitions on the hypervisor.
If we could get another 5-10 GB for /var on gudrun that would be great. We could definitely scale back what we have for / on both boxes by 2-3 GB, and /srv on gerolde has the biggest allotment of course.
-Dan
On Tue, 2010-07-20 at 16:03 -0500, Dan McGee wrote:
On Tue, Jul 20, 2010 at 3:48 PM, Thomas Bächler thomas@archlinux.org wrote:
Am 20.07.2010 17:59, schrieb Dan McGee:
We also do not have enough room to dump and reload a second copy of the DB. Any ideas on this, anyone? We didn't think very straight when we were sizing /var on a DB machine...
If you need more space, just say the word.
Is it a resizable volume, and can you do it when it is online? I don't know completely how you guys set up the root partitions on the hypervisor.
If we could get another 5-10 GB for /var on gudrun that would be great. We could definitely scale back what we have for / on both boxes by 2-3 GB, and /srv on gerolde has the biggest allotment of course.
You have to take the whole VM down to resize LVM volumes from outside the VM. Another thing is that shrinking volumes is not the easiest thing.
On Tue, Jul 20, 2010 at 4:10 PM, Jan de Groot jan@jgc.homeip.net wrote:
On Tue, 2010-07-20 at 16:03 -0500, Dan McGee wrote:
On Tue, Jul 20, 2010 at 3:48 PM, Thomas Bächler thomas@archlinux.org wrote:
Am 20.07.2010 17:59, schrieb Dan McGee:
We also do not have enough room to dump and reload a second copy of the DB. Any ideas on this, anyone? We didn't think very straight when we were sizing /var on a DB machine...
If you need more space, just say the word.
Is it a resizable volume, and can you do it when it is online? I don't know completely how you guys set up the root partitions on the hypervisor.
If we could get another 5-10 GB for /var on gudrun that would be great. We could definitely scale back what we have for / on both boxes by 2-3 GB, and /srv on gerolde has the biggest allotment of course.
You have to take the whole VM down to resize LVM volumes from outside the VM. Another thing is that shrinking volumes is not the easiest thing.
I thought that as well. That wouldn't be the end of the world; we've done downtime before, but unless we have free space that isn't already allocated that does make this tough, right?
-Dan
Hi all,
I have finished my preparation for the FluxBB update. The whole concept can be found at my profile page: http://wiki.archlinux.org/index.php/User:Pierre#FluxBB_1.4_Upgrade This will take about 3 hours. The update script itself run for more than two hours (assuming the server is similar in speed as my desktop). I also need to try to repair some previous failed conversions.
If you like to join me or see the current status visit #archlinux-dev
Greetings,
Pierre
On Thu, 29 Jul 2010 15:42:24 +0200, Pierre Schmitz pierre@archlinux.de wrote:
Hi all,
I have finished my preparation for the FluxBB update. The whole concept can be found at my profile page: http://wiki.archlinux.org/index.php/User:Pierre#FluxBB_1.4_Upgrade This will take about 3 hours. The update script itself run for more than two hours (assuming the server is similar in speed as my desktop). I also need to try to repair some previous failed conversions.
If you like to join me or see the current status visit #archlinux-dev
Greetings,
Pierre
The import and convert of the db might take some more time. Somehow disc i/o on gudrun is extreme slow. E.g. the import of the db dump is still running and I guess in the end it will finish in about an hour or more. On my local desktop this took one or two minutes... (or maybe innodb is 100 times slower than myisam)
So be patient; forums are hopefully back online by tomorrow. :-)
On Thursday, July 29, 2010, Pierre Schmitz pierre@archlinux.de wrote:
On Thu, 29 Jul 2010 15:42:24 +0200, Pierre Schmitz pierre@archlinux.de wrote:
Hi all,
I have finished my preparation for the FluxBB update. The whole concept can be found at my profile page: http://wiki.archlinux.org/index.php/User:Pierre#FluxBB_1.4_Upgrade This will take about 3 hours. The update script itself run for more than two hours (assuming the server is similar in speed as my desktop). I also need to try to repair some previous failed conversions.
If you like to join me or see the current status visit #archlinux-dev
Greetings,
Pierre
The import and convert of the db might take some more time. Somehow disc i/o on gudrun is extreme slow. E.g. the import of the db dump is still running and I guess in the end it will finish in about an hour or more. On my local desktop this took one or two minutes... (or maybe innodb is 100 times slower than myisam)
So be patient; forums are hopefully back online by tomorrow. :-)
Well MyISAM is shit if you care about data integrity or foreign keys or constraints, so yes it is slower on InnoDB but it shouldn't be 100x slower.
Without knowing how you're doing this I can't be of much help and I can't get to a real machine until later tonight.
-Dan
On Thu, 29 Jul 2010 12:46:12 -0500, Dan McGee dpmcgee@gmail.com wrote:
Well MyISAM is shit if you care about data integrity or foreign keys or constraints, so yes it is slower on InnoDB but it shouldn't be 100x slower.
It's now about 120 times slower... ;-)
But I guess it's not innodb's fault anyway. I don't see any high cpu load and we have enough ram. So i/o seems to slow. Maybe due to xen?
Without knowing how you're doing this I can't be of much help and I can't get to a real machine until later tonight.
I just did a mysqldump and I am now running "mysql fluxbb < fluxbb.sql". So nothing special. The dump is about 850 MByte.
So, next update:
The actual update script is finally running now. I had to increase the FcgidIOTimeout as apache kept killing the update script. Once all this is done (in about 200 hours ;-)) we should look for the reason of this slowness.
Just for the record: the fluxbb update script is a mess. Especially because it runs within a browser and not on cli.
On Thu, 29 Jul 2010 20:54:54 +0200, Pierre Schmitz pierre@archlinux.de wrote:
So, next update:
The actual update script is finally running now. I had to increase the FcgidIOTimeout as apache kept killing the update script. Once all this is done (in about 200 hours ;-)) we should look for the reason of this slowness.
Just for the record: the fluxbb update script is a mess. Especially because it runs within a browser and not on cli.
So, the script is still running. It just startet the last step which is rebuilding the search index. This will take a very long time at the current "speed". I don't know if this can be skipped safely.
I also wasn't able to set a maintenance page at bbs.archlinux.org explaining the situation. (apache didn't accept any allow from rule I tried and I need web access to run the script via browser)
Sorry for the extended downtime,
Pierre
On Fri, 30 Jul 2010 00:05:48 +0200, Pierre Schmitz pierre@archlinux.de wrote:
On Thu, 29 Jul 2010 20:54:54 +0200, Pierre Schmitz pierre@archlinux.de wrote:
So, next update:
The actual update script is finally running now. I had to increase the FcgidIOTimeout as apache kept killing the update script. Once all this is done (in about 200 hours ;-)) we should look for the reason of this slowness.
Just for the record: the fluxbb update script is a mess. Especially because it runs within a browser and not on cli.
So, the script is still running. It just startet the last step which is rebuilding the search index. This will take a very long time at the current "speed". I don't know if this can be skipped safely.
I also wasn't able to set a maintenance page at bbs.archlinux.org explaining the situation. (apache didn't accept any allow from rule I tried and I need web access to run the script via browser)
Sorry for the extended downtime,
Pierre
...working on the update again. Will hopefully back online soon.
On Fri, 30 Jul 2010 08:18:10 +0200, Pierre Schmitz pierre@archlinux.de wrote:
...working on the update again. Will hopefully back online soon.
The update is complete now. But it seems the server is running at its limit. We should definitely review our db config...there is something quite wrong.
On Fri, 30 Jul 2010 10:09:34 +0200, Pierre Schmitz pierre@archlinux.de wrote:
On Fri, 30 Jul 2010 08:18:10 +0200, Pierre Schmitz pierre@archlinux.de wrote:
...working on the update again. Will hopefully back online soon.
The update is complete now. But it seems the server is running at its limit. We should definitely review our db config...there is something quite wrong.
The server crashed on opening some topics..I had to restart mysql which seems to fix the problem. Is it possible that our innodb db is broken inconsistent or just damaged?
arch-dev-public@lists.archlinux.org