Update 05-11-2021 - 14:00
On the 1st of November at 16:55 GMT+1 GolfBox started receiving reports that our websites were no longer reachable via the internet. We found the problem was not localized to one specific part of our software but instead every website on our servers. We contacted our hosting service provider (Sentia) who were aware of the situation and were doing everything they could to re-establish our connectivity.
All websites remained down until 19:20 GMT+1 which is when Sentia informed us that they had re-routed traffic through a different data center, but the problem was not fixed yet. Now our customers could login to GolfBox sites again, but traffic could not leave our servers, so no external services were available.
All connectivity was eventually restored at 05:00 GMT+1 and our servers reported completed end to end connections again. The result was a total of 9 hours downtime.
The root cause of the incident was an accident that resulted in the fiber optic line being cut in between 2 of the data centers. Sentia are still trying to find the reason the redundancies did not kick in as expected but that could take some time between the various parties to discover what did not happen.
We have included the of Sentia’s root cause analysis of the incident in the attached file at the very bottom of this page. You can also download it here.
Update 02-11-2021 - 06:00
Update 01-11-2021 - 21:00
The GolfBox system is still currently experiencing outages caused by a major fiber break at our hosting partner, although you can currently log in to GolfBox, out going traffic is not currently working from the system.
Update from our hosting partner - https://status.sentia.dk/
Update 11-10-2021 - 07:42
The issue should have been fixed during the night. All operations are back to normal.
Update 10-10-2021 - 23:05
Our hosting reports that the issue should be fixed, but we are waiting for an all-clear from the affected customers. The next update is tomorrow morning.
Update 10-10-2021 - 22:13
We are currently experiencing small outages in network connectivity with our hosting provider, they are looking into the cause of these issues and will return to us with a resolution as soon as possible, not all customers are affected but if you are affected please continue to check this page for updates. Next update @2300
Update 07-10-2021 - 10:46
The issue with printing of scorecards should have been solved.
Update 07-10-2021 - 10:42
At the moment we are experiencing issues with printing of scorecards. We are looking into it and will follow up as soon as possible.
Update 30-09-2021 - 10:35
Printer issues
It looks like the printing issue is resolved now. We are working on the post mortem to find out what when wrong after the firewall switch this morning.
Update 30-09-2021 - 08:50
Printer issues
Reboot did not help. We might have to roll back this morning's Service window.
Next update 10:15
Update 30-09-2021 - 08:38
Printer issues
Server are rebooting right now.
Update 30-09-2021 - 08:00
Printer issues
We have some printer issues right now.
Planned Service window Thursday 30-09-2021 06:00-07:00
Thursday morning at 06:00 Sentia will be making the final firewall switch from Odense to Kolding.
We expect a period of downtime of around 15 minutes but Sentia our hosting partner, has asked us to put in an hour service window just in case.
Update 23-09-2021 - 22:48
GOOD NEWS:
The solution for solving the communication issue that has affected some users has now been successfully resolved and all systems are available and reporting no errors.
Update 23-09-2021 - 16:08
Update 23-09-2021 - 14:08
Message from our hosting partner: The cause of the communication issue has most likely been identified. A solution is under review and is expected to be implemented later today/evening.
Update 23-09-2021 - 12:18
Sentia - the hosting partner of GolfBox has located the communication error to be outside Sentia setup, and they are working with their backbone providers (TDC, Telia, and Global Connect) to correct the issue.
Please note that this issue only affects users with specific internet providers. Most users are not affected by this issue.
Update 23-09-2021 - 08:56
Our hosting partner was during yesterday's efforts unsuccessful in solving the network communication issues that affect all of their clients in the entire hosting environment.
The issue is escalated to the highest level at our hosting partner and at GolfBox and has our full attention.
Update 22-09-2021 - 13:49
We have been informed by our hosting partner that the remaining issues we still experience will be solved as an emergency update to our hosting partner network setup later this afternoon.
Update 22-09-2021 - 11:44
Printing of scorecards which caused the majority of errors has been fixed in GolfBox application via a programming workaround and is working again.
We are still awaiting our hosting partner on the general issue.
Update 22-09-2021 - 10:42
The issue seems to relate to a browser issue and a recent browser update. If you experience issues using a specific browser, please try a different browser.
Update 22-09-2021 - 10:17
Periodic errors can currently be experienced in different places in the systems. It is not GolfBox itself that is the cause of the problems but our hosting partner.
We are working with our hosting partner to resolve the issue as soon as possible.
Update 28-08-2021 - 19:49
Right now we are seeing some improvements and we are hoping to have fixed the problem for today - we will continue to look and fix the root cause of the problem.
Update 28-08-2021 - 19:35
We are working with our hosting partner to find the issue, we may have to restart everything in a few minutes. Next update 20:00.
Update 28-08-2021 - 18:37
Unfortunately nothing new in this update - the system is working very slowly, so click once, do not hit F5. Next update 19:40 or before if there is any news.
Update 28-08-2021 - 18:21
Unfortunately nothing new in this update - the system is working very slowly, so click once, do not hit F5. Next update 18:40
Update 28-08-2021 - 18:01
We are still trying to shut down and restart different things, trying to find what is dragging the system down. Livescoring site is up again. It does look like the system is working, just very slowly - do not click more than once. Just wait for the system to react.
Update 28-08-2021 - 17:41
We have shot down the livescoring pages to see if this will make any improvements in the admin sites. Next update 18:00.
Update 28-08-2021 - 17:20
Nothing new in this update - next update 1740
Update 28-08-2021 - 17:00
We are trying to deactivate what we think is the culprit. Next update will be 1720.
Update 28-08-2021 - 16:40
A few users are seeing improvements but others are still experiencing a very slow system. We may have found the root problem and are working on solving the issues. Next update will be at 1700.
Update 28-08-2021 - 16:11
It looks like we are seeing some small improvements - we are still working on the servers to find the culprit.
Update 28-08-2021 - 15:50
We are working on different solutions to fix the speed issue and will update this page every 30 minutes or if we fix the problem.
Update 28-08-2021 - 15:15
We are currently seeing some issues with speed on the tournament servers - we are working on it and will update this page.
Update 07-08-2021 - 21:35
We have started seeing improvements on the servers, but are still working to identify the root cause. We will continue the work to stabilize the servers.
Update 07-08-2021 - 16:41
We are currently experiencing some slowness on the tournament servers - we are working on a fix right now.
Update 19-07-2021 - 09:45
There will be a service window starting Monday 19-07-2021 at 23:00 danish time where we will be restarting the servers.
We are starting with the web servers and then the file server after that. We are restarting the servers one at a time, so we should not experience too much downtime.
Update 14-07-2021 - 08:55
There will be a service window starting Wednesday 14-07-2021 at 23:00 danish time where we will be restarting the servers.
We are starting with the web servers and then the file server after that. We are restarting the servers one at a time, so we should not experience too much downtime.
Update 22-06-2021 - 09:15
The movement of server have went according to the plan and everything seems to work as expected. We will continue to monitor this to make sure all integrations are working.
Update 21-06-2021 - 07:45
On Monday (June 21-06-2021) evening, GolfBox in conjunction with their hosting partner Sentia will begin moving our environment to new hardware in a new data center.
The move will begin at 18:00 and we expect to be done by 06:00 Tuesday morning, during this time we do not expect any big interruptions in our service although we have been told that small interruptions can occur if the data that is currently being is accessed is being moved at the same time.
If you experience any issues, wait 5 minutes and if it still happens, contact us at support@golfbox.dk.
Update 10-06-2021 - 13:30
There will be a short service window tonight at 23:30, to restart the servers. Disruption should be minimal if any at all.
Operations should now be back to normal. We are making improvements to our Load Balancer/Firewall. Unfortunately in this process, we had to restart our servers this evening to force the traffic to flow correctly again. The final changes to the new LB/FW will be happening this week which should not require any more restarts.
We are experiencing issues with one of our servers because traffic is not being spread on other servers. We are looking into it with our hosting company.
GolfBox will be working on their servers internal firewall today and tomorrow, although no interruptions are expected, some may occur due to traffic being directed differently to the new firewall.
There are still some issues with score reporting that we are working on. We also have some reports of scores being reported more than once, we will also look into this and delete any duplicate scores.
Report scores load has been minimized and the page is responding as expected again.
The Report Scores page has longer answer times at the moment, due to the huge amount of scorecards coming into the system at the moment. The rest of the system does not seem to be affected at the moment. We are working on getting the answer times down to a normal level again.
We apologize for the inconvenience.
The Member Statistic report for the Centrale Foreningsregister needs to be re-calculated. That means that there will be a short service window this afternoon and until midnight. This will only affect the Member Statistic report for the Centrale Foreningsregister, which will be unavailable in that period of time.
There will be a short service window tonight at 2300, to restart the servers. Disruption should be minimal if any at all.
There is a scheduled maintenance window Tuesday, December 1st. 2020 from 22:30 danish time and approx. 6 hours for Vmware updates. This may cause a few disruptions as each server is being updated.
Update 18-11-2020 - 09:10
The support mail is working again - sorry for the inconvenience.
We are experiencing issues with our support mail (support@golfbox.dk), please use the contact/chat button in the right lower corner to contact us at the moment.
All servers have been restarted and all is running normally.
Update 16-11-2020 - 09:15
We are just restarting each server one at a time.
Update 16-11-2020 - 08:50
Web01 is up and running again but we are still working on a few issues.
Update 16-11-2020 - 08:34
Web01 is not responding and we are working with our hosting partners to reboot the server - users should automatically be moved to other servers and should not be affected by this.
Tonight we will restart our servers around midnight danish time, to ensure stable servers. This should have minimal impact on users as one server at a time will be restarted.
Tonight we will have an unscheduled service window to patch our webservers. This will start at 2230 danish time and could last up to 4 hours. This should have minimal impact on users as one server at a time will be patched.
The restart of all servers went according to plan and the system was offline for a total of 2 minutes.
Update 02-09-2020 - 14:00
Tonight we will restart our IIS servers at midnight danish time, to ensure stable servers. This will happens once a month (or as needed) until the server upgrades in November.
Update 31-08-2020 - 14:49
We have experienced issues on some platforms with log in to GolfBox due to upload of a new contact formula on golfbox.net. The upload has been reversed for now, and login should be back to normal.
Update 08-08-2020 - 19:35
All systems should now be back to normal.
Update 08-08-2020 - 17:06
One of the servers in our cluster is not currently responding, we are looking into and will update asap.
Update 27-07-2020 - 13:17
Database integrity checks are being run from this afternoon and a few days on due to a failure in I/O yesterday during a full backup. This should not affect the performance, but please contact GolfBox if you experience any fallouts in performance.
Update 10-07-2020 - 09:15
Patching of all webservers completed this week. All went according to the plan.
Update 06-07-2020 - 15:30
Service window - more nights this week
Patching of servers (software update) are planned to run over 2-3 nights this week, each time after midnight and for expected 15-90 minutes for each server. No users should be affected as remaining servers are still in operation to service the traffic.
Opdatering 02-07-2020 - 16:38
Opdatering 02-07-2020 - 15:30
Opdatering 02-07-2020 - 13:22
Opdatering 02-07-2020 - 08:55
Der kan i øjeblikket forekomme lange svartider. Der arbejdes på at normalisere servernes load igen. Det er ikke alle brugere der er berørt af dette.
Opdatering 01-07-2020 - 15:46
Der kan i øjeblikket forekomme lange svartider. Der arbejdes på at normalisere servernes load igen. Det er ikke alle brugere der er berørt af dette.
Opdatering 01-07-2020 - 09:13
Problemerne på webserverne blev løst først på natten. Vi beklager server status først bliver opdateret nu. Webserverne kører igen normalt.
Opdatering 30-06-2020 - 21:12
Der kan i øjeblikket forekomme lange svartider. Der arbejdes på at normalisere servernes load igen. Det er ikke alle brugere der er berørt af dette.
Opdatering 30-06-2020 - 17:41
Webserverne kører igen normalt.
Opdatering 30-06-2020 - 16:32
Der kan i øjeblikket forekomme lange svartider. Der arbejdes på at normalisere servernes load igen. Det er ikke alle brugere der er berørt af dette.
Opdatering 30-06-2020 - 08:15
Gårsdagens service vindue på web02 gik rigtig godt, vi fik repareret nogle dårlige sectorer på disken og performance er klart forbedret på web02.
Opdatering 29-06-2020 - 09:20
Vi har et service vindue på web02 d. 30/6/2020 kl. 0002 i ca. 1 time. Dette vil medføre at de personer som normalt er på web02, ikke vil have adgang til systemet i ca. en time.
Opdatering 23-06-2020 - 20:51
Webserverne kører igen normalt.
Opdatering 23-06-2020 - 20:26
Der kan i øjeblikket forekomme lange svartider. Der arbejdes på at normalisere servernes load igen. Det er ikke alle brugere der er berørt af dette.
Opdatering 23-06-2020 - 18:27
Webserverne kører igen normalt.
Opdatering 23-06-2020 - 17:57
Webserverne kører i øjeblikket med meget høj CPU og der kan forekomme lange svartider. Der arbejdes på at normalisere servernes load igen. Det er ikke alle brugere der er berørt af dette.
Opdatering 23-06-2020 - 11:55
Det er i løbet af natten lykkes vores server eksperter at forbedre hastigheden på en række af de mest belastende operationer som afvikles på systemet.
Gennem morgenen hvor flere og flere brugere har begyndt at forbinde sig til GolfBox ser det ud til at serverne fint kan klare belastningen, så vi har en tro på at vores tiltag har haft den nødvendige effekt.
Vi fortsætter dog vores normale arbejde med kontinuerligt at overvåge og at forbedre systemet.
Opdatering 23-06-2020 - 08:30
Der er i øjeblikket gode forbedringer, men vi fortsætter med at monitorere det hele. Der vil også blive installeret endnu en webserver i dag.
Opdatering 22-06-2020 - 12:59
Der arbejdes fortsat på problemerne på vores servere. Nogle systemer slås fra og til i processen, hvilket kan påvirke forskellige områder af GolfBox.
Opdatering 22-06-2020 - 07:45
Vi fortsætter arbejdet med at finde ud af hvad der gør serverne langsomme, der er tilkaldt externe experter til at hjælpe med dette.
Opdatering 21-06-2020 - 10:00
Webserverne kører i øjeblikket med meget høj CPU og der kan forekomme lange svartider. Der arbejdes på at normalisere servernes load igen. Det er ikke alle brugere der er berørt af dette.
Opdatering 18-06-2020 - 14:40
SERVICE WINDOW I AFTEN KL. 18-06-2330.
SQL serveren skal have en opdatering og der kan forekomme 5-15 minutters nedetid.
Opdatering 15-06-2020 - 08:30
Det ser ud til at vi har fundet en del af problemet, og forventer at have en løsning klar i løbet af en dag eller 2. Der kan fortsætte være langsomme servere for nogle brugere indtil vi har fået implementeret et fix.
Opdatering 11-06-2020 - 12:44
Over de sidste par timer har der været en forbedring af svartider - det er dog ikke helt oppe i fart for alle brugerne endnu.
Opdatering 11-06-2020 - 10:10
En af serverne spiker i CPU og dette får det til at gå langsomt for nogle brugere
Opdatering 11-06-2020 - 08:14
Der er i øjeblikket kø for scores sendt fra turneringsmodulet til handicap regulering, de er dog på vej ind i systemet.
Opdatering 10-06-2020 - 22:30
Serverne kører igen normalt
Opdatering 10-06-2020 - 20:20
Så ser det bedre ud med CPU og svartiderne ser ud til at blive normaliseret igen.
Opdatering 10-06-2020 - 19:45
Der er et CPU spike på en af serverne som får det til at køre langsomt for nogle brugere, der arbejdes på sagen.
Opdatering 10-06-2020 - 10:20
Systemerne er nu tilbage til normal drift. Vi beklager ulejligheden.
Opdatering 10-06-2020 - 09:38
3 af webserverne kører normalt igen, vi arbejder stadig med at få CPU loaded ned på den sidste.
Opdatering 10-06-2020 - 08:00
Webserverne kører i øjeblikket med meget høj CPU og der kan forekomme lange svartider. Der arbejdes på at normalisere servernes load igen. Det er ikke alle brugere der er berørt af dette.
Opdatering 09-06-2020 - 22:45
CPU loadet er nu faldet og serverne svarer igen normalt.
Opdatering 09-06-2020 - 21:20
2 af webserverne kører i øjeblikket med meget høj CPU og der kan forekomme lange svartider. Der arbejdes på at normalisere servernes load igen.
Opdatering 07-06-2020 - 13:20
Opdatering 07-06-2020 - 13:00
Vores serverproblemer skulle nu være løst på de fleste platforme. Vi er i gang med at lave noget oprydning som kan påvirke enkelte brugere, men de fleste brugere skulle opleve et normalt system på dette tidspunkt.
Opdatering 07-06-2020- 12:36
Vi nærmer os en løsning på problemet på vores servere. Systemet skulle være oppe og køre som normalt meget snart.
Opdatering 07-06-2020 - 11:56
Vi arbejder fortsat på at løse problemet på vores servere. Problemet har ikke nogen påvirkning på vores turneringsmodul, som kører normalt.
Opdatering 07-06-2020 - 10:28
Vi arbejder fortsat på at løse problemet på vores servere.
Opdatering 07-06-2020 - 09:27
Vi oplever fortsat problemer på vores servere.
Opdatering 07-06-2020 - 08:23
Vi oplever problemer med den ene af vores servere i øjeblikket.
Vores udviklere arbejder på at løse problemet.
Update 05-05-2020 - 17:05
Servers should be back to normal performance.
Update 05-05-2020 - 16:34
We are experiencing very long response times on one of our servers for all GolfBox services.
We are working on a solution.
Update 03-05-2020 - 14:45
Servers are stable because a variety of performance tuning operations was carried out on our servers.
Also, we have started deployment of new APP versions for all platforms with enhancments on performance.
Hosting specifications will be upgraded tomorrow.
Update 03-05-2020 - 09:00
We are experiencing very long response times on our servers for all GolfBox services.
The cause is due to the fact that our APP load is > 300% of normal traffic.We are working on multiple solutions including allocating more server power to the APP area.
Update 02-05-2020 - 12:15
The problems with the servers have now been resolved.
Update 02-05-2020 - 10:00
We have server issues. Our staff is working hard to resolve the problems
Update 28-04-2020 22:59
The problems with the servers have now been resolved.
Update 28-04-2020 - 20:00
We have server issues. Our hosting is working hard to resolve the problems
Update 04-07-2018 10:19
The problems with the servers have now been resolved.
Update 04-07-2018 09:14
We are experiencing problems with our servers on GB Tour.
Tech support are working hard to fix the problem.
Update 30-07-2016 1000
CLOSED: We are still working on locating the real underlying issue – and the right solution for it. We’ve seen a 100% increase in admin traffic this week compared to last week – this may explain why but not the what-part.
It’s not due to inadequate server capacity – they use 50% CPU, and the suddenly the admin site hangs.
This evening we have split the Admin process into yet another separate process. So we’ll go to a total of 3 processes per server – we had one per server yesterday. Half of today we had two.
Tech support will have phone and computer on them at all times throughout the weekend – so if the alarm sounds, they will be over it as fast as they can get the phone out of the pocket.
Update 29-07-2016 1645
We have experienced some issues slow response times during the afternoon – but right now it’s running.
Here’s a list of the changes I’ve made.
I’ve managed to make a workaround for the search cache so that it no longer interferes with GolfBox Tournament Admin. ( 13:15 today )
I’ve also split the GolfBox Tournament Admin into more processes to make better use of the webservers new extra capacity. ( 15:30 today )
I’ll continue to monitor all systems very very closely..
Update 29-07-2016 1320
This is the 12:00 o’clock planned update.
The servers are currently running ok’ish – we see some blocking on one of them, which I’ll smack with my hammer as soon as I get the alarm for it. This is related to the search cache.
Regarding this cache – we have not been able to create a fix and deploy this yet. Planning and development is still in progress.
Next planned update around 15:00.
Update 29-07-2016 0745
The server upgrade has been completed according to plan and all webservers are running again – now with extra power available.
Besides that, we’ve identified the main issue we’ve seen – which is regarding the previously mentioned internal search cache. This search cache can in some circumstances interfere with the operations of the GolfBox Tournament Admin site. We’ll work on deploying a fix that will isolate this feature from the rest.
Next planned update @ 2016-07-29 12:00
Update 28-07-2016 1513
All systems are still running as expected – but as precaution we will add more power to our webservers.
This will happen tonight at 23:00 to 23:20 CET.
One server will be handled at a time and each server will take a couple of minutes to upgrade. The server has to be rebooted and therefore you might experience downtime while this is happening.
Update 28-07-2016 1300
All systems have been running smoothly all day so far.
I’ll keep montoring all system very closely and continue to logs from yesterday.
Next planned update @ 15:00 CET
Update 28-07-2016 1230
Scorecards are working again.
Update 28-07-2016 1220
Servers are still stable, but we are working on the PDF print issue.
Update 28-07-2016 0915
Right now all systems are running fine and I’m monitoring them very closely.
An internal competition search cache is currently the main suspect – and further investigations is going into this part of the system.
Next planned update at 12:00 CET
Update 27-07-2016 0006
Investigations are still in progress and leads are being looked into.
Next update @ 09:00 CET
Update 27-07-2016 2023
There are still occasional issues, but we are working on the system and fixing them as fast as we can.
Update 27-07-2016 2005
The systems have been running decent for a couple of hours due to an expected decrease of user load.
The rest of the day will be spent on locating bottle necks and attempts to fix these.
Next update @ 23:00 CET
Update 27-07-2016 1745
We are currently struggling to keep GolfBox Tournament running.
A very high number of tournaments are causing performance issues all across our system, but mainly GolfBox Tournament.
Right now it’s a matter of keeping the systems up – later I’ll dive into what I can do to prevent it from happening again.
Next update @ 20:00 CET
Update 27-07-2016 1645
CUP load has stabilised and the alarms are going down - the system is stabil at the moment, but we will keep monitoring the system.
Update 27-07-2016 1645
We still have some users affected of the high load, and will continue to work on lowering the load on the servers.
Next update at 1700
Update 27-07-2016 1630
There seem to be stabilising servers at the moment, but we are still monitoring them.
Next update at 1645
Update 27-07-2016 1615
There is access to the tournament module at the moment, it is however very slow updating.
Next update at 1630
Update 27-07-2016 1600
We are still having problems with high load on our servers, affecting all parts of GolfBox now.
Update 27-07-2016 1530
There are still server issues - we are still working the issue.
Update 27-07-2016 1505
We are still experiencing a higher load on our servers, but not as high as earlier.
Update 27-07-2016 1411
The load is stabilising at the moment, but we will keep monitoring the systems.
Opened 27-07-2016 1357
We have a higher than normal server load on our tournament servers - this can be affecting all systems. We are working on a solution.
Kommentarer
0 kommentarer
Artiklen er lukket for kommentarer.