It’s Friday the 13th today. I’m not really superstitious, but i should have known not to schedule a cutover to Office 365 on this day…
I’ve been getting ready for a customer’s Exchange Hybrid migration from on premise Exchange 2010 to Exchange Online which was due to take place tonight. All was going well, my batches of users have been nicely in sync and awaiting completion at 95% for a couple of weeks. Since i’m all about the detail, i was having a good scout through the customer tenant, having a look at some of the user migration logs to ensure that all mailboxes were truly completing their 24 hour delta sync. No issues detected, until i happened to notice that a bunch of users that i did a new migration for a few days ago had turned up in the tenant as contacts. It looked as though the attributes hadn’t been stamping correctly on some of these objects. First thing i noticed right out of the gate was that when clicking on any user details in the tenant, it threw a .NET Framework error:
Odd, pressing OK on above error then takes me onto the user page where clicking on any configuration options gave the following error:
This issue was repeating itself across multiple users. Next step was to open a Support call via the Tenant scoped as SEV A (I always call out my estimation of the Severity A/B/C as i’ve been used to doing this for years with on premise Exchange Premier support). I was pleasantly surprised to get a callback from one of the Exchange Online migration engineers within 30 minutes. I have seen a huge improvement in response times within the last 3 months. I had a couple of instances in late 2015 where a support call lingered for 8 hours and was passed around the system which led to alot of frustration and which i ended up escalating internally through Microsoft to the appropriate Manager. His feedback was that they were onboarding alot of support staff to cope with the massive volumes of users being pushed through the funnel, but that there was a lead time involved, which of course is only natural. Microsoft is not immune to the cause and effect of market conditions and to me it looked as though they had underestimated the onboarding volume to Exchange Online last year.
Anyway back to my issue. After 90 minutes of log gathering via PowerShell from the on premise Hybrid CAS server, the engineer came back and announced I was affected by a known incident; specifically the dreaded Incident EX45856. It was already reported on the Office 365 service Health Twitter handle:
(when it’s on Twitter, you know it’s serious, right?)
…and reported on the Office 365 Service Dashboard within the tenant as a service degradation:
The official support response was:
“…As discussed, your issue “Remote moved mailboxes are showing under Contacts” is identified as known issue and our backend team is working to get this resolved. We have updated your domain name with the backend team. You can check the status of the issue on Office 365 Dashboard under Incident number: EX45856…”
At time of writing the detailed service statement status is shown as at 91% resolution whatever that means:
Long story short, it’s Friday the 13th and my customer was pretty understanding which is not always true. Hopefully it won’t be long before things are resolved and we can reschedule the cutover date. It’s good to see the Service Health dashboard is actually reflective of real time issues and that the front end Microsoft support staff are keeping the health status updated accordingly.