Network Status News & Information

A power surge at our primary Calgary site has caused one of our UPS's to kick in and the other to start reporting errors. Our primary internet modem is connected to the failed UPS but our backup modem is currently online and functioning in its place. Service remains up, but in a degraded state. DNS entries have been pointed at the backup static ip but if any services are caching our primary ip they may need to be refreshed. Once the UPS has been replaced we can update the dns entries again. Thanks for your patience. - MichaelW 12-NOV-2024 11:47 MST

The UPS is end-of-life and will require replacement. We are currently investigating options but the networking gear has been migrated off of it. All services remain up and functional. Thanks for your support. - MichaelW 12-NOV-2024 16:43 MST

We are currently investigating several alerts of the entire WM2 site being unreachable. Teams are dispatched and investigating now. We appreciate your patience during this unscheduled outage. - MichaelW 06-NOV-2024 12:29 MST

The error has been resolved and an investigation will be conducted that led to the necessity to reboot the networking equipment. Downtime was minimized and impact was minimal as this was only at the WM2 site. Customer websites remained up throughout. This only affected the WritheM Storage (ownCloud), Media, and Project servers including Intellimerge for Spotify. Thank you for your continued support. - MichaelW 06-NOV-2024 12:44 MST

This notice is to inform you of maintenance being performed to our network which will affect your service(s) for the impact duration noted below within the planned maintenance time. Site access is not required unless specified. Planned upgrade and maintenance to our Network helps prevent unexpected outages and provides the best in class service for our customers at no additional cost. Every effort will be taken to minimize service interruptions and we apologize for any inconvenience.

Ticket Number: CHG0613666 - Planned

Event Description: Network capacity expansion

Duration: outage for up to 15 minutes. - Rogers 13-JUN-2024 12:36 MDT

This will affect only the services hosted at our primary Calgary location. This includes services such as Intellimerge for Spotify and WritheM ownCloud services as well as streaming services. All client facing websites will remain unaffected as these are hosted in Texas. Thank you for your patience and understanding while our service provider upgrades their network. - MichaelW 13-JUN-2024 12:46 MDT

This notice is to inform you of maintenance being performed to our network which will affect your service(s) for the impact duration noted below within the planned maintenance time. Site access is not required unless specified. Planned upgrade and maintenance to our Network helps prevent unexpected outages and provides the best in class service for our customers at no additional cost. Every effort will be taken to minimize service interruptions and we apologize for any inconvenience.

Ticket Number: CHG0613666 - Planned

Event Description: Network capacity expansion

Duration: outage for up to 60 minutes. - Rogers 15-APR-2024 10:55 MDT

This will affect only the services hosted at our primary Calgary location. This includes services such as Intellimerge for Spotify and WritheM ownCloud services as well as streaming services. All client facing websites will remain unaffected as these are hosted in Texas. Thank you for your patience and understanding while our service provider upgrades their network. - MichaelW 19-APR-2024 14:04 MDT

Event Cancelled - Rogers 22-APR-2024 08:33 MDT

Event Rescheduled

Ticket Number: CHG0621449 - Planned

Event Description: Network Upgrade

Duration: outage for up to 15 minutes. - Rogers 30-APR-2024 15:04 MDT

We are currently investigating a reported raid array failure on our primary storage host in our Calgary site. We have taken the server offline temporarily while we investigate the issue. All data is stored in triplicate so no data loss will be experienced, but we need to investigate the impacts of why the host-spare did not take over immediately. Thank you for your patience while we look into the issue. Services including ownCloud, and any service on the project server will be temporarily offline with expected resolution before the end of the day. All customer owned websites remain up and unaffected. - MichaelW 2024-MAR-01 12:01 MST

A dead drive that contained the raid configuration was found to be the culprit. We are rebuilding the array now and services should be coming back up now. Although during the rebuild services will be available, they might be a bit slower than usual until the rebuild is complete. Thank you for your patience and support. A new drive has been ordered and no further downtime should be expected to install.  - MichaelW 2024-MAR-01 13:36 MST

We are just rebooting the main storage server at WM2 for installation of a replacement hard drive that died yesterday. Services at our primary Calgary site should be down for the next ~15 minutes but impact to customers should be minimal. All customer websites will remain up and accessible during this time. This outage will only affect services hosted at the WM2 site which include but are not limited to Intellimerge for Spotify, and WritheM ownCloud filestore. Thank you for your patience and continued support. - MichaelW 2023-OCT-26 13:01 MST

This notice is to inform you of maintenance being performed to our network which will affect your service(s) for the impact duration noted below within the planned maintenance time. Site access is not required unless specified. Planned upgrade and maintenance to our Network helps prevent unexpected outages and provides the best in class service for our customers at no additional cost. Every effort will be taken to minimize service interruptions and we apologize for any inconvenience.

Ticket Number: CHG0574447 - Planned

Event Description: Network Upgrade

Duration: outage for up to 90 minutes. - Shaw 19-OCT-2023 15:24 MST

We have received another service disruption notification from our Internet Service Provider. This will affect only the services hosted at our primary Calgary location. This includes services such as Intellimerge for Spotify and WritheM ownCloud services as well as streaming services. All client facing websites will remain unaffected as these are hosted in Texas. Thank you for your patience and understanding while our service provider upgrades their network. - MichaelW 23-OCT-2023 15:59 MST

We are currently working through some sudden DNS maintenance with our Texas host this evening. Client hosted websites and entries are unaffected but subdomains at writhem.com might be un-resolvable for the duration of the outage. You may also experience a problem with DMARC and DKIM authenticated emails over the next 48 hours. We forsee this impact as minimal but appreciate you patience during this 'hiccup'. - MichaelW 09-OCT-2023 10:37 MST

This notice is to inform you of maintenance being performed to our network which will affect your service(s) for the impact duration noted below within the planned maintenance time. Site access is not required unless specified. Planned upgrade and maintenance to our Network helps prevent unexpected outages and provides the best in class service for our customers at no additional cost. Every effort will be taken to minimize service interruptions and we apologize for any inconvenience.

Ticket Number: CHG0572899 - Planned

Event Description: Network Upgrade

Duration: outage for up to 30 minutes. - Shaw 28-SEP-2023 10:42 MST

This will affect only the services hosted at our primary Calgary location. This includes services such as Intellimerge for Spotify and WritheM ownCloud services as well as streaming services. All client facing websites will remain unaffected as these are hosted in Texas. Thank you for your patience and understanding while our service provider upgrades their network. - MichaelW 29-SEP-2023 14:29 MST

This notice is to inform you of maintenance being performed to our network which will affect your service(s) for the impact duration noted below within the planned maintenance time. Site access is not required unless specified. Planned upgrade and maintenance to our Network helps prevent unexpected outages and provides the best in class service for our customers at no additional cost. Every effort will be taken to minimize service interruptions and we apologize for any inconvenience.

Ticket Number: CHG0571260 - Planned

Event Description: Network capacity expansion

Duration: outage for up to 15 minutes. - Shaw 12-SEP-2023 09:49 MST

This will affect only the services hosted at our primary Calgary location. This includes services such as Intellimerge for Spotify and WritheM ownCloud services as well as streaming services. All client facing websites will remain unaffected as these are hosted in Texas. Thank you for your patience and understanding while our service provider upgrades their network. - MichaelW 12-SEP-2023 15:37 MST

It appears the interruptions have begun and work has started. - MichaelW 19-SEP-2023 03:19 MST

All services have been restored. Thank you for your patience. We will continue to keep an eye on services if they go out again as the outage window is not complete for another couple of hours. - MichaelW 19-SEP-2023 03:54 MST

We are currently investigating a system hang on the remus server in Calgary. Until resolved it appears that services hosted at the WM2 site will be intermittently unavailable. Thank you for your patience while we investigate the cause and steps needed to take to resolve. - MichaelW 11:45 12-AUG-2023 MDT

Although we run the server with a raid-0 like zfs partition this appears to be caused by a bad disk. We have swapped out the bad drive and are rebooting now. The array will rebuild and things should carry on in the next short while. The only services to be affected were the WritheM Project server including Intellimerge for Spotify and the ownCloud storage server. - MichaelW 12:13 12-AUG-2023 MDT

This notice is to inform you of maintenance being performed to our network which will affect your service(s) for the impact duration noted below within the planned maintenance time. Site access is not required unless specified. Planned upgrade and maintenance to our Network helps prevent unexpected outages and provides the best in class service for our customers at no additional cost. Every effort will be taken to minimize service interruptions and we apologize for any inconvenience.

Ticket Number: CHG0546000 - Planned

Event Description: Network capacity expansion

Duration: outage for up to 15 minutes. - Shaw 30-MAY-2023 09:03 MST

We have received the above outage notice from our ISP of an upcoming maintenance that will impact all services at the WM2 site. The services affected here do not include any primary customer websites but will impact the project server as well as services such as the WritheM ownCloud instance and Intellimerge for Spotify. Thank you for you patience while our ISP upgrades our network resiliency. -MichaelW 30-MAY-2023 12:08 MST

This notice is to inform you of maintenance being performed to our network which will affect your service(s) for the impact duration noted below within the planned maintenance time. Site access is not required unless specified. Planned upgrade and maintenance to our Network helps prevent unexpected outages and provides the best in class service for our customers at no additional cost. Every effort will be taken to minimize service interruptions and we apologize for any inconvenience.

Ticket Number: CHG0524016 - Planned

Event Description: Improve reliability with network maintenance

Duration: Outage for up to 90 minutes for all services. - Shaw 14-MAR-2023 12:28 MST

We have received the above outage notice from our ISP of an upcoming maintenance that will impact all services at the WM2 site. The services affected here do not include any primary customer websites but will impact the project server as well as services such as the WritheM ownCloud instance and Intellimerge for Spotify. Thank you for you patience while our ISP upgrades our network resiliency. -MichaelW 14-MAR-2023 12:52 MST

This notice is to inform you of maintenance being performed to our network which will affect your service(s) for the impact duration noted below within the planned maintenance time. Site access is not required unless specified. Planned upgrade and maintenance to our Network helps prevent unexpected outages and provides the best in class service for our customers at no additional cost. Every effort will be taken to minimize service interruptions and we apologize for any inconvenience.

Ticket: CHG0523110 - Planned

Description: Forced Fibre Relocation

Impact Duration: Outage for up to 6 hours. -Shaw 10-FEB-2023 14:11 MST

The maintenance has been completed. Thank you for your understanding. - MichaelW 07-MAR-2023 07:05 MST

We have just suffered a power supply failure on remus. This machine hosts the control, worker, and bastian servers. The machine has a redundant power supply but the controller what failed so no outage should be felt beyond a quick reboot which the machine is going through now. Services should be back up shortly and a new power supply module has been ordered with a date of 4-MAR scheduled to install. During installation the server will need to be brought down. Installation should take no longer than 30 minutes. Thank you for your understanding and patience. Updates will follow here next weekend when we start the installation of the new hardware. - MichaelW 11:05 26-FEB-2023 MST

Work on replacing the power supply will now begin. We will update as progress is made. -MichaelW 08:23 04-MAR-2023 MST

The new hardware is installed and services are just booting up. We will monitor the status as services come back. Thank you for your patience - MichaelW 10:15 04-MAR-2023 MST

We are currently experience a complete internet service provider outage at our primary site. We are engaging the appropriate teams now and will post any updates here as the situation develops. Thank you for your patience. -MichaelW 17:16 20-FEB-2023 MST

Reference Number: INC1276772

Our teams are looking into it. Please stay tuned for updates. A further update will be provided at 20:23 MT.-shaw-duncan 17:23 20-FEB-2023 MST

Technical crews are dispatched to arrive on site to investigate this matter. A further update will be provided at 20:46 MT. Thank you for your patience.  -shaw 18:46 20-FEB-2023 MST

This issue has been resolved. Thank you for your patience. -Shaw 21:00 20-FEB-2023 MST

Extreme cold weather combined with heavy snow fall is currently affecting internet services to WM2. Intellimerge for Spotify and all other project servers are experiencing very slow response times to the internet backbones we are connected to. We will update here as information is provided by our Internet Service Provider. Thank you for your patience. - MichaelW 23-DEC-2022 11:37 MST

A complete internet outage is now being experienced at our primary Calgary site. Our ISP has been engaged and teams have been dispatched to resolve as quickly as possible. Thank you for your understanding as we rush to get services restored during this holiday season. - MichaelW 23-DEC-2022 13:00 MST

Service techs have installed the new hardware this evening and services have been stable for the last few hours. We are now closing this ticket and marking it as resolved. Thank youf or your patience during the last couple of days. Have a great holiday season. - MichaelW 25-DEC-2022 1:52 MST

We will be installing some new power equipment at the WM2 site on Saturday morning. It should take a few hours and services hosted at WM2 will be offline for the majority of the time. The details will be posted here as progress is made and services are restored. This should ensure that our batteries are full and our downtime is minimized. The largest impact felt will be the outage of the WritheM Project service which includes Intellimerge for Spotify, WritheM Storage server (romulus), and the WritheM Media streaming server (marcellus). We will also apply some firmware updates to the bios if we have time. Thank you for your understanding and ongoing support. -MichaelW 13-NOV-2022 13:20 MST

Work has begun. We will update here as services are brought back. - MichaelW 19-NOV-2022 10:02 MST

Work has concluded. Servers are just booting up now and services are coming online. Thank you for patience. - MichaelW 19-NOV-2022 14:22 MST

We experienced a massive DNS outage this evening that required our techs to perform emergency maintenance on our primary Texas server. We await a root cause analysis by the datacenter and will be investigating required measures to avoid further outages of this nature in the future. Thank you for your continued support. Updates will follow here if suitable. - MichaelW 21-OCT-2022 12:48 MDT

We are currently investigating an outage reported that affects our primary 10GbE network switch. Updates will follow. Thank you for your patience while we work to fix this. -MichaelW 2022-OCT-13 20:14 MST

We have isolated the issue and replaced the faulty DAC. Services should be back up momentarily. Sorry for the inconvenience. - MichaelW 13-OCT-2022 20:19 MST

We have noticed a large number of errors coming from the praetor control server. We are performing a reboot in an effort to bring the services back online. Services should be offline for no more than 30 minutes. This will only affect services hosted at WM2 and no client web sites are affected. Major services affected will be the ownCloud, Intellimerge for Spotify, and the game servers. Thank you for your understanding. - MichaelW 2022-MAY-02 07:25 MDT

Maintenance Advisory Details

Ticket Number: CHG0451300 - Planned

Description: Network upgrade affecting internet services

Duration: 25 minutes - Shaw 19-APR-2022 12:39 MST

Contractors will be on-site between 10:00 and 18:00 on 4th of March. For approximately 1 hour starting at 11am all services hosted at WM2 will be down while the electrical panel is switched out for a new panel. Additional outages may occur throughout the day but will be minimal. All client websites will remain online and unaffected as they are hosted at WM4. The largest affected services will include WritheM Media, game servers, and the project server including Intellimerge for Spotify. Thank you for your patience and understanding. - MichaelW 24-FEB-2022 18:35 MST

Due to a supply shortage we will be rescheduling this maintenance to an undetermined date. We will update this ticket as soon as we hear from the electricians of a new maintenance window. Sorry for any unforseen issues that this may cause. All services will remain up and unaffected during the original maintenance window. Thank you for you understanding. - MichaelW 02-MAR-2022 14:00 MST

We have rescheduled this change to this coming Monday. Power will be disconnected at approximately 11am and should be off for no more than 2 hours. Thank you for your understanding. -MichaelW 17-MAR-2022 17:27 MDT

Work has begun and services are being shutdown now. -MichaelW 21-MAR-2022 12:03 MDT

The new panel has been installed and services are being restarted now. Outages should only last for a few more minutes, but we'll continue to monitor for the remainder of the day as they come back. -MichaelW 21-MAR-2022 12:48 MDT

We are currently investigating a high iodelay on one of our servers at WM2-AB-CA. This is causing sporadic outages of services hosted at our primary Calgary site. All customer websites remain unaffected and up. Services such as Intellimerge for Spotify and many of the WritheM Media services are currently offline as a result. Thank you for your patience. - MichaelW 24-JAN-2022 8:30 MST

A restart of the storage server has resolved the issue. We will continue to monitor for any further performance degradation. Thank you for your support  - MichaelW 24-JAN-2022 13:50 MST

Good morning, we plan to restart the remus vm host this morning to apply some overdue changes to the root certificate authorities. As a result of the recent LetsEncrypt root certificate expiration we have been delaying the much needed updates as long as possible but the time has come. A quick reboot should be all that's needed now. Downtime should be no longer than an hour for all services to start after the reboot. This will only affect services hosted at WM2-AB-CA and no client websites will be affected. Thank you for your understanding - MichaelW 05-NOV-2021 08:30 MST

We are currently investigating some increased error rates on the plebian server. This affects services like Intellimerge for Spotify and any other services hosted on the plebian and praetor servers. We will update here as we progress. Thank you for your patience. - MichaelW 2021-OCT-28 14:11 MST

Services have been restored. A reboot of the server seems to have resolved the issue. Thank you for your support and understanding - MichaelW 2021-OCT-28 16:55 MST

Affected Area: Calgary
Affected Services: Internet
Reference Number: INC0997695
Summary: Some customers in Calgary are experiencing an interruption to Internet services. We are working to restore service as quickly as possible and apologize for any inconvenience this may cause. - Tom 19-AUG-2021 11:46 MT

Our teams are looking into it. Please stay tuned for updates. - Tom 19-AUG-21 12:46 MT

Technical crews are dispatched to arrive on site to investigate this matter. Thank you for your patience. - Tom 19-AUG-21 12:52 MT

Tech crews are on site working towards a resolution. - Tom 19-AUG-21 13:28 MT

Services in the area have been restored. Thank you for your patience. - Tom 19-AUG-21 13:42 MT

We are planning on installing a new redundant power supply backplane on the romulus storage server at WM2 this Sunday morning. Planned outage time will be about 3 hours. No client facing websites will be affected but all services at WM2 will be shutdown as a precaution while we perform the hardware installation. Services affected will include the WritheM Project server including Intellimerge for Spotify. We apologize for the outage as we strive to reduce any problems in the future. Thank you for your understanding and support. - MichaelW 13-AUG-2021 20:37 MTN

Maintenance is complete and services are up. Thank you for your patience! - MichaelW 15-AUG-2021 11:29 MTN

We have been tracking an increased number of errors over the last several hours and a few crashes from our controller. Investigations continue and results will be posted as soon as we are complete with a fix. The impact to public facing clients should be minimal but we thank you for your patience anyway.  - MichaelW 13-JUL-21 22:51 MDT

A fix has been implemented and seems to have worked. We will continue to monitor the situation into tomorrow. - MichaelW 14-JUL-21 23:04 MDT

We have received a new UPS which should help mitigate all of the power related issues we have been experiencing at our primary Calgary site recently. We plan to install this new equipment on the morning of Saturday 27 May. The installation should be minimal but all systems will be brought down and a UPS run-time calibration will be performed which will extend outage to upwards of 3 hours beginning at 10am MTN. All client hosted websites will remain unaffected and available during this outage. We do expect outages on the WritheM Project server which includes Intellimerge for Spotify though. We appreciate your understanding during this outage as this should help make things more resilient in the future.  - MichaelW 27-MAY-2021 13:22 MST

All done! Thanks for your patience. -MichaelW 29-MAY-2021 15:08 MST

Our primary Calgary site finds itself without power or internet currently. We are investigating and will post updates as they become available. Services hosted at WM2 have been shut down. Thank you for your understanding. -MichaelW 2021-APR-25 05:49 MST

Crews have been dispatched and are investigating. Incident ID #0579. Estimated Restoration: Apr 25 2021, 7:30 a.m.  -Enmax 2021-APR-25 05:57 MST

Power has come back but we continue without internet. Servers with be started anyway shortly. Thank you for your patience. -MichaelW 2021-Apr-25 09:05 MST

All services are starting and the internet has been restored. Things should be returning to normality very shortly. Thank you for your support -Michael 2021-Apr-25 10:16 MST

We will be performing our annual disaster recovery testing at 11:00 on 11-APR-2021 MST. This is the time we test our recovery and backup systems if the worst was to ever happen. This includes a complete reconstruction of our primary Calgary site. Services should automatically switch over to the new site if an error is detected and data restored from backups automatically. Production services at WM4-US-TX will be unaffected therefor all customer websites will remain unaffected. This is only a test of the WM2-AB-CA site. We do expect a short interruption to our database services hosted here which could affect the WritheM Projects including ArtStart and Intellimerge for Spotify. Thank you for your understanding. - MichaelW 04-APR-2021 10:58 MST

We are currently without power at WM2-AB-CA, likely due to the large wind gusts experienced in the area. 

More details will follow as updates are provided to us. 

Thank you for your understanding and support. -MichaelW 01-Apr-2021 19:38 MTN

Power has been restored and services are coming online now. Outage cause was reported as a Pole fire (Ref.#9991). - MichaelW 01-Apr-2021 20:25 MTN

Greetings,

We will need to take offline cerato at the scheduled time for a maintenance. All sites and services will be offline while this work is in progress.

Please follow for updates. - RohitM 01-MAR-2021 02:50 CST

This will affect all customer websites hosted on our primary WM4-US-TX server during the maintenance. DNS should be unaffected. We will strive to minimize this downtime and appreciate your understanding and support. - MichaelW 01-MAR-2021 11:13 MST

Total downtime was 12m 42s. All maintenance is complete. Thank you for your understanding and patience. - MichaelW 02-MAR-2021 13:44 MST

Reference Number: CHG0310282, CHG0310281 & CHG0310280
Summary: Some customers may experience an interruption of the affected services between midnight and 6:00 am MT. This interruption is expected to last approximately six (6) hours. Services will be restored automatically when the maintenance is complete. - shaw-rutu 2021-Feb-13 10:51 MTN

We are just going to do a quick reboot of the romulus storage server. A new hard drive is not showing the correct size and a reboot will likely correct the issue. Services will be suspended but should be back fairly quickly. Thank you for your patience. - MichaelW 2021-JAN-21 13:41 MTN

We are currently without power at our primary Calgary site. All services at this site have been shut down as our provider investigates and attempts to resolve the issue. Thank you for your patience as crews work to resolve this issue. -MichaelW 2020-Nov-25 20:41 MST

Power has been restored and we are now bringing the services back up. ETA is 30 minutes. Thank you for your support. -MichaelW 2020-Nov-25 21:53 MST

Services have been restored. Thank you for your understanding during this outage. The root cause for this outage was reportedly a car accident hitting a power pole thus knocking out both internet and power for the area. - MichaelW 2020-Nov-25 22:23 MST

We will be rebooting the Remus server to apply some new firmware to the motherboard in an attempt to help deal with some high iowait times that could be related to memory. Downtime should be minimal. Virtual machines hosted on remus including praetor, plebian, vivet, and cetus will also be offline for the duration of the reboot. No customer websites will be affected by the downtime. Thank you for your understanding - MichaelW 2020-Nov-04 10:45 MST

The new firmware did not resolve the issue so we will be taking this opportunity to migrate the vm's to a different host to allow for further diagnostics. Services will remain during the migration. - MichaelW 2020-Nov-04 11:30 MST

We will be taking the remus server at our primary calgary server offline for upwards of an hour on the 22nd of October at 11am MST to install a new redundant power supply into the machine. Services hosted on this machine will be offline for the duration of the maintenance which include Intellimerge for Spotify, praetor, plebian, vivet, and ludus. Thank you for your understanding. Updates will be posted as we begin work. - MichaelW 10-OCT-2020 10:59 MST

Work will begin now and should be offline for upwards of an hour. Thank you for your patience. - MichaelW 22-OCT-2020 11:00 MST

Maintenance is now complete. Thank you for your support! - MichaelW 22-Oct-2020 12:45 MST

A lightning storm caused a power surge at our WM2-AB-CA site this evening that apparently was strong enough to bypass our power conditioner and seems to have negatively affected one of our servers. Remus is currently unable to boot and techs are diagnosing the issue and working to resolve. This outage affects all virtual machines as well as most public services hosted at WM2-AB-CA including the WritheM Project server. We apologize for any inconveniences this may cause you and thank you for your support. -MichaelW 2020-08-01 23:14 MDT

It looks like the motherboard was damaged with the power surge. Replacement parts have been ordered on rush delivery but we'll be down on this server until they arrive. We apologize for the inconvenience this may cause you. Thank you for your understanding and patience. - MichaelW 2020-08-05 16:25 MDT

The new motherboard has finally cleared customs and is on it's final decent... please make sure your tray tables are in their--- wait. uh hem. We do expect to have the new hardware installed and systems to be restored before the end of next week. Thank you very much for your ongoing patience and understanding. - MichaelW 2020-08-28 16:11 MDT

The motherboard has arrived and been installed. Services should be coming back online now. It seems the hardware was held up at customs due to COVID-19 but all is well now. - MichaelW 2020-08-31 20:48 MDT

Beginning July 4th, 2020 we will be taking all of the servers at our primary Calgary Alberta site down for extended maintenance. As part of this, we will be moving all equipment to a new physical site. Downtime is expected to be upwards of 2 weeks to allow proper shutdown, disassembly, transport, reassembly, and testing of all of the equipment required to run the services hosted at WM2-AB-CA. Servers included in the migration include: cetus, ludus, patronus, praetor, plebian, remus, romulus, and vivet. Primary client websites hosted on cerato at WM4-TX-US, and our backup services on hercules hosted at WM1-AB-CA will continue to operate during this outage. With the exception of the WritheM Projects site, including Intellimerge for Spotify, and cetus+ludus dedicated client servers, no impact is expected by clients during the outage. The largest impact to clients we expect, will be to the Streaming services, and the Intellimerge for Spotify platform. We thank you for your patience and will update the end-time of the outage as we know more. - MichaelW 2020-06-12 14:43 MDT 

Happy Independence Day to our american friends. Work has now begun and the servers are now offline. We have submitted a dns change that should propagate shortly that will advise incoming people of the outage that may not have been aware. Thank you for your patience and understanding. Next update 11 July 2020 - MichaelW 2020-07-04 10:10 MDT

The servers have successfully received at their new location and electrical work is scheduled for Wed of this week. We expect we should be getting the servers back up this coming Thursday 16 July 2020. If anything changes, this is the place to see any updates. Thank you for your patience while we get things back to normal. - MichaelW 2020-07-11 09:40 MDT

During the migration we lost several databases and had to restore from backups. Considering the size of the arrays lost, the rebuild took several days. The final databases are being reloaded now and systems are up. We expect one final, and very brief, outage to close up the server case and rack it. Good thing we had backups of backups of backups. Thank you for your patience and support during this extended outage. - MichaelW 2020-07-23 17:22 MDT

Our ISP has reported an outage caused by the significant storm cell rolling through the area at the moment. Teams are aware and working to correct the problem as soon as possible. Thank you for your support. - MichaelW 2020-Jun-13 19:45 MTN

We have identified a networking outage on our primary data server at WM2-AB-CA. Teams have been notified and are working to resolve the issue. This outage affects most services at WM2-AB-CA however client files remain intact and sites hosted at WM5-TX-US are fully operational. Thank you for your patience. - MichaelW 2020-May-09 09:35 MTN

A fix has been applied and services have been restored. Thank you for your support and understanding. - MichaelW 2020-May-09 14:07 MTN

We will be upgrading the IntelliMerge system to 2.0 which will require a full halt of the current version in preparation for migration. Merges will not be performed and account changes will not be possible during this time. We will post updates here when we begin work and complete it. This only affects the IntelliMerge for Spotify platform. All other services/servers will be unaffected. Thank you for your understanding. - MichaelW 2020-Apr-28 12:11 MTN

We have begun the migration. updates to follow... - MichaelW 2020-May-01 11:01 MTN

Maintenance is complete. Migration as successful and the new version is now live. Thank you for your patience. - MichaelW 2020-May-01 12:25 MTN

An extended power outage has been reported at our primary Calgary site. Although we are currently on backup power, our energy supplier is reporting that the outage will last longer than our backup will provide. In about 30 minutes we will be shutting down our servers and will expect a 7am restart. This outage will be upwards of 5 hours.  We apologize for any inconvenience this may cause. Next update at 02:30 - MichaelW 2019-Dec-10 02:19 MTN

The servers have been shutdown as crews are onsite working on the power outage. Servers have been set to auto-start should the power come back before 7am.  Networking equipment remains online and should last on backups until fixed. Next update 08:00 - MichaelW 2019-Dec-10 02:47 MTN

Power has been restored and servers are up with the exception of our bastian server, patronus. It looks as though a corrupt boot partition is preventing it from booting and may have failed a while ago but gone undetected until we rebooted the server. patronus remains offline while all other services have been restored. - MichaelW 2019-Dec-10 09:35 MTN

All systems are go! Thanks for your patience. - MichaelW 2019-Dec-20  11:40 MTN

We will be installing the new heatsinks on the processors of remus on Monday at 1PM MTN. Downtime should be upwards of an hour but will not affect any client websites. Updates will follow. Thanks for your patience. - MichaelW 2019-Dec-06 16:01 MST

Maintenance has been completed and temperatures are registering within range. Thank you for your continued support - MichaelW 2019-Dec-09 16:01 MST

We will be taking the remus server offline this morning for about an hour. This should allow us time to reseat one of the processors that is reporting abnormally high thermal readings. The hope is that a reapplication of thermal paste will resolve the issue. This will also affect most services at WM2-AB-CA as remus is the virtual host for servers like plebian, praetor, ludus, and vivet. romulus and patronus will remain online. Thank you for your patience. - MichaelW 26-Nov-2019 11:00 MST

Tonight a new Static IP will be activated for WM2-AB-CA/writhem.net. All services may report offline while the DNS entries are updated and propogate to the new ip. Downtime is expected to be minimal. Potentially all services at WM2 will be impacted by the DNS change. Please update any manual entries to our new IP: 184.67.75.110. Thank you for your patience and support. - MichaelW - 24-SEP-2019 10:53 MDT

We will be postponing the outage until Friday night. Thanks for your understanding and continued support - MichaelW 25-Sep-2019 08:18 MDT

Maintenance is now complete. Both the new and the old IP will actually work for another day. We'll be turning off the old IP some time tomorrow. Thanks very much! - MichaelW 28-Sep-2019 01:10 MDT

In an effort to improve uptime and stability, we will be upgrading our primary web hosting server's hardware! Here are some of the benefits of the upgrade: 

• All solid-state drive(SSD) backed storage. This means much faster access and write times for data.
• Moving to Intel(R) Xeon(R) E5 processors vs existing AMD.

The scheduled maintenance will begin on Mon, July 22nd for the cerato server, intermittent downtime is expected starting at 9 PM CDT. All customer websites may be impacted by the downtime. We apologize for any inconvenience this may cause. - MichaelW 19-Jul-2019 11:18 MTN

We have rescheduled this maintenance. The updated scheduled maintenance will begin on Fri, July 26th for cerato, during the outage websites may experience intermittent downtime starting at 9 PM CDT. We apologize for any inconvenience this may cause.  - MichaelW 23-Jul-2019 11:44 MTN

Some customers may experience an interruption of services between midnight and 6:00 am MT. This interruption is expected to last approximately six hours. Services will be restored automatically when the maintenance is complete. Reference Number: CHG0188994 - shaw-overnights-mike 9-Jul-2019 00:00 MST

We will be conducting some power equipment testing and maintenance between the hours of 11 and 13 today. Services hosted at WM2 including plebian, praetor, remus, romulus, patronus and vivet will be intermittently offline during this time. Primary websites will remain online and all client data will be accessible during this maintenance window. Thank you for your understanding. - MichaelW 21-MAY-2019 @ 09:28 MTN

Maintenance has concluded. Thank you for your continued support and understanding. - MichaelW 21-May-2019 13:48 MST

We are currently aware of a reported outage in Calgary and techs have been dispatched to investigate. No ETA is set but could be up to two hours. Sorry for the inconvenience. -LeoW5D7 26-APR-2019 11:38

We are currently investigating an outage at our primary Calgary site. It appears the resource scheudler is stuck and requires some investigation. Teams have been dispatched and we will resolve as soon as possible. Thank you for your understanding. Streaming services and containers hosted at the site are currently offline - MichaelW 17-APR-2019 15:26 MST

An outage has been detected of the Container Management (pollux) service. Teams are being dispatched to investigate and updates will be posted here. Thank you for your patience. -echelon 29-JAN-2019 14:20

It looks as though the container management interface that we use at WM2-AB-CA has gotten stuck in a loop with a memory leak of some kind. A quick reboot of castor + pollux will be performed. Services should take about 30 minutes to be detected as resumed. A root cause analysis will be performed in the coming days. Thanks very much for your understanding. -MichaelW 29-JAN-2019 21:26

Beginning on 10th of November 2018 we will be upgrading the repono/esx servers at WM2-AB-CA. We will be taking these two servers offline, decommissioning them and then installing the new hardware at this time. We expect the outage to last for up to a week for some services, including media and vpn services. We will be migrating some of the hardware in our existing servers to two yet-to-be-named servers that we would love some help naming. If you head over to our reddit thread there is already some great suggestions as well as the guidelines for server name submissions. Thanks very much for your understanding. We will be updating this post as work begins in November. - MichaelW 06-OCT-2018 10:27 MTN

Services are restored. We will need to take things offline for a quick reboot in about a month or so though to finalize some hardware configs. Keep your eyes peeled for that upgrade in the future. - MichaelW 02-DEC-2018 23:55 MTN

Starting on Nov 27 at 23:00 and continuing until Nov 28 04:00 we will start performing reboots on the following servers. During the maintenance, services will be briefly unavailable:
cerato
hercules - AaronG 26-Nov-2018 12:00 CST

We will be scheduling a 15 minute outage window starting at 3pm MTN on 25th of September. This will be minimal but used to install some new networking hardware acquired for the WM2 site. vivet, repono, echelon, adamo, patronus and memoria will be offline for the duration of the outage. Thank you for your understanding. - MichaelW 23-SEP-2018 14:45 MST

Thank you. The maintenance has been completed and services are restored. - MichaelW 25-SEP-2018 15:24 MST

Our backup Calgary site has experienced a power surge that apparently disrupted some networking equipment. Replacement hardware has been ordered and is enroute. Installation is expected this weekend and customer impact is expected to be minimal. As this is a backup site, no primary customer data or services was being hosted when the node went down. The node is expected to experience upwards of 72 hours of downtime. Thank you for your understanding. -MichaelW 2018-APR-12 20:35 MDT

Beginning at 14:30 MTN we will be performing a quick reboot on our primary storage server at the WM2-AB-CA Calgary site. This will cause outages to media services and calgary content but will be offline for no longer than 15 minutes. Thank you for your understanding and ongoing support. - MichaelW 2018-MAR-18 14:29 MDT

We are currently investigating an ISP network outage. Appropriate teams have been notified and are working to resolve the issue as soon as possible. Thank you for your patience. - MichaelW 2017-10-31 00:29 MTN

We are currently investigating a network outage at our primary Calgary location. It appears to be an outage at our ISP and proper teams have been engaged. Thank you for your patience while services are being corrected. - MichaelW 2017-10-12 00:09 MST

NASBOX and is associated Thor data array will be taken offline for approximately 1 hour in order to replace and upgrade an existing controller card. The outage will also affect all esx hosted machines as nasbox will be offline for the duration of the maintenance. All client data will remain online and accessible via http on the cerato server, but ssl traffic will be inaccessible. Thank you for your patience and understanding as we make upgrades to serve you better. - MichaelW 2016-12-27 01:21 MT

Unfortunetly we will need to reschedule this outage as the upgrade was unsuccessful. It appears that the parts we have received from our supplier are defective and need to be reaquired. More information pertaining to this upgrade will be posted here. As part of this upgrade we will take advantage of the downtime and install the existing array into a new server when we receive all of the parts. The new server will be codenamed repono. Latin for storage. Some simple specs on this upgrade are as follows:

• 24GB DDR3 ECC RAM
• Dual Xeon 5640 @ 2.67GHz
• 2x IBM m5014 HBA Controller Cards
• QLogic Fiberchan 24xx Card

Roughly two hours ago we experience sevre lag on our ESX host at our primary calgary site. We attempted to reboot the server but now it is not responding. All data remains intact but we are currently working to restore the esx host and all associated services hosted by this server. All SSL Websites, SSO functions, Logging and Gaming services will be offline until we can restore the ESX Server. We appologize for any inconveniences this may cause you and appreciate your understanding at this time. We will post updates here as they are available. - MichaelW 2016-12-05 01:42 MST

We are currently investigating a total network loss at our Calgary WM2 location. SSL websites are down, but all client HTTP websites remain up. We appologize for the outage and are currently investigating. It looks as though a vendor outage may be affecting us. The vendor has been notified and we remain at their mercy. - MichaelW 2016-09-20 13:41 MST

As there was a brief Orderbox Database Outage recently, we will be performing a failover of the current primary database server to a standby to rule out any hardware issues. We will be undertaking the following actions during the maintenance:

1. Investigate the offending queries & tables and identify how latency increased suddenly and take relevant preventive measures.

2. Review redundancy measures for accessibility to our Data Centre’s in case of corporate network issues and implement the necessary measures.

3. Failover to a new database server to rule out hardware issues.

The maintenance details for this are as follows:

Post failover we will also be running some stress tests / burn-ins on the current primary to identify any hardware issues.

Affected Services: Domain Registraitons, Domain Renewals, Domain Transfer Requests, Other Domain inquiries.

We apologize for the inconvenience, please feel free to contact our support team in case of any queries. - DomAdmin 2016-05-20 15:45

Originating with a stop error on our network attached storage server we are currently performing an unscheduled upgrade to our nas. This should hopefully take less than an hour. While we are upgrading we have paused the esx machines that rely on the datastore, so we are currently looking at a complete outage from our WM2 site. This does not affect our http client server, or our VPN services. Thank you for your patience. - MichaelW 2016-04-09 21:56

We are currently experiencing a network outage  at the internet service provider level.  We have alerted the proper department and are awaiting for a technician to repair the current weather related outage. All servers in our Calgary WM2 site will be offline for the duration of the outage.  Thank you for your patience. - MichaelW 29/02/16 01:46am

We will be scheduling a large network upgrade at our WM2-AB-CA site this wednesday afternoon. The upgrade itself should take no longer than 30 minutes, but possible IP changes could backlog the propogation of our hostnames. We will post updates here as they become available, once the upgrades have begun. - MichaelW 25/01/16 3:33pm

We are currently experiencing a network outage courtesy of our Calgary based ISP. They have been notified of the outage and we will update here as we get more information. Thank you for your support and understanding. - MichaelW 17/11/15 12:33am

We are currently investigating a loss of network at our primary Calgary site. We will do everything we can to resolve services as quickly as we can, however our primary team to handle such calls are currently on vacation. We are working hard to resolve this unexpected and most inopportune outage as quickly as we can. Thank you for your support and understanding. - MichaelW 09/08/15 12:33am

We are currently beginning a network upgrade at WM1-AB-CA that should take the majority of the day. No client services will be affected as mirrors are running without problem at our primary Calgary Datasite. Thanks for your patience and continued support of WritheM. - MichaelW 15-07-29 9:46am

We are currently experiencing network loss at our primary Calgary site. Technicians are currently in contact with the internet service provider and will update here as we get more information. Thank you for your patience. - - MichaelW 25/02/15 12:55 PM

We will be upgrading the network connection at WM2-AB-CA and need to replace a modem in order to provide the fastest service possible. Connections problems to the site will be experienced in the outage window but should not last the entire outage window. Thank you for your understanding. - MichaelW 23/02/15 10:30 AM

We are currently investigating a stop error attributed to imminent hardware failure on our primary esx server hosted at WM2-AB-CA. We appologize for the inconvenience while we sort things out and get the services back up and running. - MichaelW- 21-02-15 09:20

The cerato server will be rebooted at 12:01am on Nov 18th. This will help us provide better stability on this server. We don't anticipate more than a few minutes of downtime, however we will be closely watching this server to identify any issues as fast as possible. We will update this thread as the maintenance gets under way.  - JLong - 11-10-14 11:20

This is a reminder of the upcoming maintenance schedule to begin shortly. During this time the server will be offline and we expect about 15 to 20 minutes of downtime. I will update this thread upon completion of the maintenance.- tbell - 11-17-14 11:55

Thank you for your patience during this time. cerato is back online at this time. - tbell - 11-18-14 03:14

We will be performing MySQL maintenance on October 23rd from 0300 to 0600 CDT. We do expect some brief MySQL service interruptions for the affected servers.  - AKempski - 10-22-14 11:36

Cerato will be coming down on 10/22/14 at 00:30 CDT for a FSCK of /usr.
Unfortunately there is no ETA on this process.

We will update this thread accordingly once we have more information. - tbell - 10-21-14 05:57

We will be performing maintenance on the MySQL service on the cerato server between the hours of 02:00 AM CDT and lasting until 06:00 AM CDT. We expect only 5-10 minutes of downtime for the MySQL service. - JWhite - 09-10-14 11:47

We are currently without power at our primary Calgary datacentre. We have the city investigating the power loss and have been running critical systems on battery, but all non-essential services have just been shut down to prolong the battery power. We apologize for this unexpected outage. - 19-08-2014,  10:47AM by MichaelW

We are planning on installing a new RPC Unit and Network Switch at our primary Canadian datacentre on Tuesday night. All major servers will be offline for the outage. We thank you for your understanding. - 07-07-2014,  14:40PM by MichaelW

The cerato server will be undergoing scheduled MySQL maintenance on 06/20/2014 between 2am and 6am CDT. We thank you for your patience.. - 19-06-2014,  3:24AM by ZErskine

The cupcake server will be undergoing scheduled MySQL maintenance on 06/17/2014 between 2am and 5am CDT. We thank you for your patience.. - 16-06-2014,  7:13PM by MarkV

Hello, at this time one of our datacenters, is experiencing an issue. We have notified them of the issue, and their technicians are working on it currently. I can assure you it will be up as soon as possible. - 08-04-2014,  10:18PM by AnthonyK

We are currently applying emergency patches to resolve the reported Heartbleed Bug, which would allow attackers to annonymously and invisibly steal confidential client data. More information on this bug can be found at http://heartbleed.com/. - 08-04-2014,  04:15PM by MichaelW

We apologize but we will have to bring down all internet access at WM2-AB-CA for emergency network maintenance. We will update this post with more information as it becomes available. Thank you for your understanding. - 27-03-2014,  05:53PM by MichaelW

We are currently investigating a network outage at our Canadian datacentre. - 03-03-2014, 8:33AM by MichaelW

We apologize but we have had to bring down the cerato server for emergency maintenance. We will update this psot with more information as it becomes available. Thank you for your understanding. - 23-02-2014, 11:53AM by JLong

We are currently investigating the reboot/stall of our major esx host at our WM2-AB-CA location. Staff are currently bringing the servers back up that are hosted on the server and services are coming back shortly. Sorry for an inconvenience this may cause. - 23-02-2014, 10:03PM by MichaelW

Beginning on Friday November 15th at ~8:00pm we will be installing new hardware at our WM4-TX-US site. This is in accordance with our tri-annual hardware upgrade plan. We have already directed traffic to our new dns servers and expect the migration to start copying data at this time. Account access will be locked in order for data will be transferred. We do not expect any downtime to your content, but the content may be unable to update for upwards of 2 hours while we migrate to the new, faster hardware. We appreciate your patience during this migration, feel free to follow any changes at status.writhem.com - 1-11-2013, 2:01PM by DatacenterM

We are scheduling a major outage that will affect all Servers/services hosted at WM2-AB-CA for this coming weekend. Although we have migrated critical client applications to WM4-TX-US and WM5-TX-US for the duration of the outage, nearly all non-customer servers will be unavailable for the outage. We plan on upgrading the Server rack and ESX Server. This requires that all servers in the current rack be shut down, the rack removed, then installed in the new rack along with some new ESX Hardware. This will have a massive affect on the ability for us to host and develop content for our clients north of the border. Thank you for your patience and understanding through this outage. - 14-10-2013, 2:53PM by MichaelW

This scheduled maintenance will affect all services hosted on the adamo server which will include WritheM News. We are planning a quick hardware reconfigure of the adamo server which will require the machine to be turned off for roughly 10 minutes. We thank you for your understanding during this outage. - 29-7-2013, 1:19PM by MichaelW

Due to the massive flooding that has affected the downtown core of Calgary, AB Canada. We have been notified that rolling power outages may be experienced. Our backup systems are currently operational and we do not expect to experience any down time, but felt that customers should be notified of possible outages prior to them happening. If you are within the Calgary area, please stay safe. Thanks for your understanding. - 21-6-2013, 2:58PM by MichaelW

The cupcake server is being upgraded onto brand new, more powerful hardware which will include the latest versions of cPanel and CentOS. 
We plan to facilitate this upgrade as quickly and seamlessly as possible. Ensuring total satisfaction with this maintenance is our primary objective. We will keep you updated here throughout. The upgrade process itself will result in an exact copy of all accounts being moved to new hardware and will ensure that the freshest possible up to the minute data is retained. 

Once the data switchover is complete we will begin diverting all traffic to the new server so that customers do not miss any traffic or experience connectivity problems. Please be aware that there may be minimal amounts of downtime, however we will do everything within our power to ensure a smooth transition. - 7-6-2013, 11:12PM by DatacenterM

We will be installing a new power supply in our data store server starting at noon on Saturday may 11th. The outage should not last more than an hour... Ludus and all non raid services will be unaffected.- 10-5-2013, 9:52PM by MichaelW

Starting at 3pm MTN on May 5th 2013 we are scheduled to perform some minor maintenance on the nasbox server which will affect most services at WM2-AB-CA. The outage will consist of a shutdown of the server, install of some new hardware, and reboot. It shouldn't take more than an hour, but we appreciate your understanding. Ludus should remain untouched during this outage.-  5-4-2013, 10:02AM by MichaelW

The drive containing partition /dev/sdc1 in RAID array /dev/md/1 has failed.

One of our database servers (memoria) just filled its hard drive while repairing a crashed table and is now causing sporadic connectivity issues. We are working on expanding the hard drive and should have thing back online shortly.. -  3-21-2013, 11:00PM by MichaelW

We are currently investigating a nasbox outage...  - 3-22-2013, 11:58AM by MichaelW

We will be taking the servers listed below down for a scheduled reboot on Thursday, February 14th starting at 1AM CST. We do not have an ETA but expect to have minimal downtime for each box assuming all goes well. We apologize for any inconvenience that this may cause but the maintenance is necessary as the previous nights maintenance was not successful for these boxes.

We will be conducting scheduled maintenance on this machine that will force this machine to be offline for the duration of the outage. This will affect all gaming services including Mumble, CalgaryCompany Forums, Terraria and Minecraft servers.

Our Billing and members area are receiving an upgrade today. The area may be intermittently unavailable during the upgrade. Thank you for your patience.

This server will be coming down briefly for a reboot on Friday, October 19, starting at 02:00 MST. We apologize for the inconvenience, but the reboots are needed to load the best available version of the kernel for these servers.

We estimate about 5 minutes of downtime for each server. We will keep this report up to date with the status of maintenance. - 10-17-2012, 10:25AM by JMagalich

We thank you for your patience while we performed the necessary reboot. The operation is now complete and the server is back online. - 10-19-2012 2:35AM by Kevin S

We will be performing network maintenance for our servers housed in the dallas area datacenter starting at midnight, 11:00PM MST on May 4th. Unfortunately one of the steps requires replacing a major router, so we will be seeing a loss of connectivity to our server for a short period of time. We appreciate your understanding and thank you for your patience during this network outage. - 04-26-2012, 11:13 AM by ZEdgerton 

The maintenance has been completed and everything is back to normal. Thank you for your patience during this outage. - 05-04-2012, 01:38AM by oliver

UAT has been taken offline for the christmas season. As production has been rolled out successfully, the site will be used as a development platform only.

Release Canadidate for production enhancments requires the use of the UAT Environment. Reactivated.

Our main VMWare host will be receiving a service pack update tonight at midnight. The server will reboot after the patch is applied and outages may last for up to 20 minutes. Thank you for your understanding...

Disc 4 of the Oden Array will be replaced with a new hard drive tonight starting at 9pm MTN. Downtimes could exceed 1 hour on all echelon, compello, mediabox, and sandbox services, to allow replacement and rebuild of the array.
All bluebox services will remain functional during the outage.
Thank you for your understanding during the outage. We hope that this will increase the stability of all data captured in the Oden array and allow us to submit the disc 4 for warranty replacement.