Update 30-07-2016 1000
CLOSED: We are still working on locating the real underlying issue – and the right solution for it. We’ve seen a 100% increase in admin traffic this week compared to last week – this may explain why but not the what-part.
It’s not due to inadequate server capacity – they use 50% CPU, and the suddenly the admin site hangs.
This evening we have split the Admin process into yet another separate process. So we’ll go to a total of 3 processes per server – we had one per server yesterday. Half of today we had two.
Tech support will have phone and computer on them at all times throughout the weekend – so if the alarm sounds, they will be over it as fast as they can get the phone out of the pocket.
Update 29-07-2016 1645
We have experienced some issues slow response times during the afternoon – but right now it’s running.
Here’s a list of the changes I’ve made.
I’ve managed to make a workaround for the search cache so that it no longer interferes with GolfBox Tournament Admin. ( 13:15 today )
I’ve also split the GolfBox Tournament Admin into more processes to make better use of the webservers new extra capacity. ( 15:30 today )
I’ll continue to monitor all systems very very closely..
Update 29-07-2016 1320
This is the 12:00 o’clock planned update.
The servers are currently running ok’ish – we see some blocking on one of them, which I’ll smack with my hammer as soon as I get the alarm for it. This is related to the search cache.
Regarding this cache – we have not been able to create a fix and deploy this yet. Planning and development is still in progress.
Next planned update around 15:00.
Update 29-07-2016 0745
The server upgrade has been completed according to plan and all webservers are running again – now with extra power available.
Besides that, we’ve identified the main issue we’ve seen – which is regarding the previously mentioned internal search cache. This search cache can in some circumstances interfere with the operations of the GolfBox Tournament Admin site. We’ll work on deploying a fix that will isolate this feature from the rest.
Next planned update @ 2016-07-29 12:00
Update 28-07-2016 1513
All systems are still running as expected – but as precaution we will add more power to our webservers.
This will happen tonight at 23:00 to 23:20 CET.
One server will be handled at a time and each server will take a couple of minutes to upgrade. The server has to be rebooted and therefore you might experience downtime while this is happening.
Update 28-07-2016 1300
All systems have been running smoothly all day so far.
I’ll keep montoring all system very closely and continue to logs from yesterday.
Next planned update @ 15:00 CET
Update 28-07-2016 1230
Scorecards are working again.
Update 28-07-2016 1220
Servers are still stable, but we are working on the PDF print issue.
Update 28-07-2016 0915
Right now all systems are running fine and I’m monitoring them very closely.
An internal competition search cache is currently the main suspect – and further investigations is going into this part of the system.
Next planned update at 12:00 CET
Update 27-07-2016 0006
Investigations are still in progress and leads are being looked into.
Next update @ 09:00 CET
Update 27-07-2016 2023
There are still occasional issues, but we are working on the system and fixing them as fast as we can.
Update 27-07-2016 2005
The systems have been running decent for a couple of hours due to an expected decrease of user load.
The rest of the day will be spent on locating bottle necks and attempts to fix these.
Next update @ 23:00 CET
Update 27-07-2016 1745
We are currently struggling to keep GolfBox Tournament running.
A very high number of tournaments are causing performance issues all across our system, but mainly GolfBox Tournament.
Right now it’s a matter of keeping the systems up – later I’ll dive into what I can do to prevent it from happening again.
Next update @ 20:00 CET
Update 27-07-2016 1645
CUP load has stabilised and the alarms are going down - the system is stabil at the moment, but we will keep monitoring the system.
Update 27-07-2016 1645
We still have some users affected of the high load, and will continue to work on lowering the load on the servers.
Next update at 1700
Update 27-07-2016 1630
There seem to be stabilising servers at the moment, but we are still monitoring them.
Next update at 1645
Update 27-07-2016 1615
There is access to the tournament module at the moment, it is however very slow updating.
Next update at 1630
Update 27-07-2016 1600
We are still having problems with high load on our servers, affecting all parts of GolfBox now.
Update 27-07-2016 1530
There are still server issues - we are still working the issue.
Update 27-07-2016 1505
We are still experiencing a higher load on our servers, but not as high as earlier.
Update 27-07-2016 1411
The load is stabilising at the moment, but we will keep monitoring the systems.
Opened 27-07-2016 1357
We have a higher than normal server load on our tournament servers - this can be affecting all systems. We are working on a solution.