Tuesday, 24 December 2013

MOLE Maintenance Fri 27 Dec from 9am

On Friday 27th December we will be performing maintenance on MOLE as part of our ongoing programme of work to improve its performance.

MOLE will be unavailable from around 9am until around mid-day and should be considered at risk for the whole day.

We hope this maintenance work during the holiday period does not inconvenience you and we would like to reaffirm our continuing commitment to restore MOLE to full performance.

The purpose of this maintenance exercise is to repair the 'Test' tool which has recently developed faults. 

Thursday, 12 December 2013

MOLE downtime Friday 13th December

As part of our continuing improvement work on MOLE, we will be doing a bit of work on it first thing tomorrow. 

You won't be able to use MOLE from 6am and although we hope to be completed sooner, it should be back by 9am at the latest. We'll let you know when it's up and running again.

Monday, 9 December 2013

MOLE back, but at risk...

MOLE is now back up and running, but for the time being is at risk. We are keeping a close eye on it and will let you know if there are any developments.

MOLE update 9th Dec, 12:10...

Due to MOLE's continued instability we have taken it offline in order to keep working on it. At the moment we have no ETA on when it will be fixed but will keep you posted as we go. Thank you for your patience as we work through the problem.

MOLE problems this morning...

At around 10:15 this morning MOLE became unstable. We have been trying to get it back up and running but so far have been unable to regain stability. You may see MOLE come and go whilst we are working on it, and if you do find that it's working for you, please be aware that it may become unavailable without notice.

We'll keep you updated as the morning progresses.

Thursday, 5 December 2013

MOLE problems today

We are very disappointed to report that MOLE had problems again today. The service failed at around failed around 3:30pm and was returned to service around 4:15.

We do have an ongoing programme of work to restore MOLE to appropriate levels of service. Thank you for your patience in the meantime.

MOLE unavailable 6-8am Fri 6 Dec

With the exception of the unplanned outage on Tuesday, MOLE appears to be running at acceptable performance levels since our recent work to move it to new hardware and upgrade the database. Please continue to let us know how its performance levels are for you.

We will continue to work to improve its performance and reliability and the next maintenance event will be tomorrow, Friday 6 December, when MOLE will be unavailable for up to two hours from 6am.

We will also be applying a software patch sometime next week. We will update you with more information nearer the time.

Thursday, 28 November 2013

Today's MOLE Problems Summarised

From around 19:00 on Wednesday 27th November the MOLE front ends started to exhibit software failures which are still continuing.

The failures are the same on all four front ends. The failed component is detected & automatically restarted - taking about 10 mins to return to service.

The impact on staff and students is difficult to determine. When a front end node is about to fail responses get slower. When it does fail user sessions are redirected to one of the remaining servers. However, if more than two servers fail at the same time then users may experience poor performance - or in an extreme situation - no performance.

The failures have been raised with the suppliers, Blackboard. We're continuing to try help to diagnose the root cause and/or detour the problem.

MOLE back

MOLE has been returned to service again. We'll investigate what happened.

MOLE down

MOLE is currently unavailable. This isn't part of scheduled maintenance it's a fault. It may or may not be related to the work we are undertaking. We'll investigate, return MOLE to service and keep you updated.

Wednesday, 27 November 2013

MOLE hardware upgrade

This morning the work on MOLE that required the system to be taken offline was completed successfully.

We moved MOLE onto new hardware with updated operating systems and new load balancer (the bit of the system that spreads the users round the servers). A number of small changes were also made to the database server. This should provide a more resilient/expandable environment for the servers - with a more rapid start up time - which enables subsequent changes/upgrades to MOLE be made more swiftly. It also gives us easier monitoring and diagnostic information to help in resolving issues.

On Friday morning, the second period of downtime we announced on Monday will allow us to upgrade the database, and other upgrades are being planned at the moment that will will announce shortly. 

Tuesday, 26 November 2013

MOLE maintenance 6-9am, Wed 27 and Fri 29 Nov

As part of our programme of maintenance on MOLE we will be working on MOLE from 6am-9am Wednesday 27 November and 6am-9am Friday 29 November.

MOLE will be unavailable during this time but will be restored to service before 9am each day.  There will be additional periods of downtime in the coming weeks as part of our ongoing programme to restore MOLE to full performance.

If you are interested, the downtime on Wednesday will involve moving the MOLE service onto new hardware and the downtime on Friday will involve us upgrading the database behind MOLE to a new version.

MOLE outage this morning

This morning we had a network problem that affected a number of services, including MOLE, and you may have experienced issues if you were using MOLE during the time the problem persisted. 

The problem lay outside of MOLE and has now been rectified, and MOLE is now accessible as normal.

We apologise for the inconvenience this has caused.

Monday, 25 November 2013

Programme of MOLE maintenance announced

Email to University learning technologists and MOLE contacts in departments sent today Mon 25 Nov

Dear colleagues

As you will be aware, MOLE continues to suffer performance problems and we are putting together a comprehensive ongoing plan to address this. As part of this process we will be taking MOLE down on several occasions to perform maintenance. The first two periods of downtime are Wednesday 27 and Friday 29 November, this week.

Maintenance will take place 5am-9am this week but it should be considered at risk until midday, and as such no critical activities should be performed during that time. There will be subsequent periods of downtime, which will be communicated nearer the time, but we will try to restrict these to the regular MOLE ‘at-risk’ period of 6-9am on Friday mornings.

The maintenance work will involve moving MOLE onto new hardware with additional capacity. We will also be performing an iterative sequence of software updates and modifications, taking time to monitor the impact of each individual change we make.

We cannot say for sure when MOLE will be returned to reliable full performance service but we will continue to investigate, implement changes and then monitor in a measured, ongoing programme of work until our VLE has been restore to full service.

We are keen to provide regular updates on our progress but do not wish to bombard you with information, so we have have set up an opt-in MOLE-users email list which we will post regular updates about MOLE, including the current work. Subscribe to this list using the link below:

We have also set up a dedicated MOLE maintenance blog which you can revisit at your convenience to be updated on progress:

We apologise for the ongoing poor-performance of MOLE and for the inconvenience caused by the imminent periods of downtime. We are dedicated to resolving this issue and we thank you for your patience.