News

Updates about the MXroute service

Spammers want to use your domain

- Posted in News by

This is a topic that should interest all MXroute customers. I want to stress that this is not about a security issue with MXroute, and no MXroute services have been compromised. The only compromises I am referring to are customers using the same passwords across the internet, where other services may not end up being safe places.

Over time it has become increasingly apparent that spammers want to use the MXroute infrastructure to send spam. In fact, I'm convinced that they specifically target our customers to try to find ways to get inside and use our infrastructure to send that spam. For this reason, I want to stress that you should be using unique passwords for every single account on the internet. I won't force you to, but I might not be able to do business with you for long if you continually use passwords that are easily stolen somewhere else on the internet and then used to abuse our service.

I've noticed that when spammers compromise customer account passwords at MXroute, they do not brute force (attempt passwords until they find one that works), and they do not use any secret path (like a compromised server on our side). Instead, they directly authenticate as our users over SMTP, using our customer's email passwords. These events tend to happen in groups by server. This leads me to believe that the spammers are first looking for groups of customers (perhaps by MX record), searching to see who they can find on compromised password lists (passwords found from compromises of other services, where email login might correlate), and then they use one or two email accounts per day until they exhaust their findings. This typically only occurs in bursts of 1-3 days and then they rest for a bit.

If you've dealt with IT security, none of this is news or interesting (except perhaps the pattern that we see). If you haven't, this may raise some questions. I'm going to take a shot at guessing these and answering them.

Are you sure this isn't a problem with MXroute security, and that your servers haven't been compromised?

As much as anyone can ever be. I may not know about some unpublished vulnerability used by state actors to undermine governments, but I do keep up with known vulnerabilities for both the open and closed source software that we use and apply patches as soon as they are made. I do audit the security of the servers regularly, and there have never been any signs of intrusion. Given how common it is for people to reuse passwords, and how many services have been compromised over the last decade, it is far more reasonable to assume that I am accurate in my understanding of the cause of these events.

Why don't you enforce stronger passwords?

A couple of reasons, actually. The first being that if you reuse a password across the internet that matches security requirements, this does nothing to prevent the situation. The second is that there is no increase in security by customers leaving in annoyance and using another service that does not have these requirements. In the past when we had enforced this, customers did complain about stronger password requirements and they did take their business elsewhere to not be subject to it. Sometimes meaning well doesn't result in a positive net impact on the result.

Where do people find these passwords?

Over time there have been so many compromised services across the internet, and in recent years people have begun compiling databases from these compromises into what are called "combolists." These are gigantic databases of known login information from compromises that go at least as far back as the first major Yahoo compromise. I can show you three places right now where you can go and probably find some of your old passwords on a number of combolists:

https://cracked.to/Forum-Combolists
https://weleakdata.com/forumdisplay.php?f=17
https://raidforums.com/Forum-Databases

Scary, right? I've found a number of my old passwords in there. I've found incredibly important corporate accounts for major businesses in them as well. This should highlight why it's important that a password compromise ONLY compromise what was compromised, and not all of the rest of your internet accounts.

What do you do to combat this?

I suspend outbound mail for the compromised accounts and email the customer to let them know. These days it doesn't happen for long before it's caught, and plenty of filters prevent a number of these events, but it still happens and should be highlighted.

Why not use two-factor authentication?

We wanted to provide basic SMTP/POP/IMAP service, and not something that requires direct integration by the developers of your email applications. The protocols themselves do not support two-factor authentication. We can talk about things like app passwords (unique passwords generated by us for your applications), but that really defeats the type of service that we set out to be and isn't actually two-factor authentication in the first place.

Aren't you afraid that this makes MXroute look bad?

Absolutely not. They want your email accounts because they want to use the service we've worked so hard to build out for you. We should be flattered, and you should know that it is an indication of the quality of work that is being done to provide you with a high quality service. The higher the quality of email delivery, the higher the value the service will have to spammers.

If you'd like to discuss this or ask questions about it, we've created a space for that:
https://community.mxroute.com/t/discussing-password-security

 

What we learned this week

- Posted in News by

This past weekend and the days after it, we learned some valuable things that we'd like to share with you.

How did we get here?

Originally I said that we had no intention of migration cPanel servers to DirectAdmin, though we were provisioning new customers on DirectAdmin servers. As the weight of cPanel's dramatic price increase took hold, we found that we were not able to secure the licensing deals that would allow us to barely float above water. Those licensing deals were calculated into our overhead increase of a bit over $10,000 per year, which meant that this figure was actually a fair bit lower than what we ended up paying.

What happened?

Because of the weight of the cPanel pricing increase, I decided to start experimenting with a migration from cPanel to DirectAdmin. I didn't know if this was going to work well at all, I needed to experience the process and see what worked and what failed. I started this process with a server that desperately needed a hardware upgrade, to take care of two problems at once (server migration was happening regardless). This was our London server (london.mxroute.com).

It took one day to perform the backups necessary to start the process, another day to transfer them to the new server, another to perform the restorations, and almost another to sync the changes that occurred on the old server during those previous days.

After the restores were complete, I noticed that the number of things which transferred perfectly turned out to be impressive. By my account, nearly everything had transferred flawlessly. This is where I faced a difficult decision, and it can be disputed whether or not I made the right one.

What did we learn?

It can be very difficult to gauge the level of failure in a particular change when only viewed through my own perspective. What seems, to me, like a flawless change can be viewed differently by customers who may use features that I've not thought to test, or who had use cases that I hadn't imagined. This was true with the migration of the London server, and I'd like to share with you what I've learned didn't go over so well.

  • Notification, or lack thereof.

I spent days testing to see if this could even work, I didn't want to confuse customers by emailing them all and saying "Hey, sometime in the next week this might change to that, or it might not, we'll have to see." Yet, at the end of it, I didn't want to waste the days that I had spent working toward something that I had deemed a success.

  • Catch-alls didn't migrate.

Catch-all accounts might be expendable to some, but might also be vital to others. That they didn't migrate is an oversight and one that varies in degree by customer.

  • Domain forwarders didn't migrate.

Honestly I didn't even realize customers were using them, forwarders for a whole domain where user@domaina would go to user@domainb, relative to the value of "user." This was another oversight.

  • Filters didn't migrate.

This I was aware of, and they couldn't have been migrated as the two filtering systems are incompatible. However, a more clear plan around how to handle this would have been ideal.

  • SSO in WHMCS broken

You couldn't just log in to DirectAdmin from portal.mxroute.com in the same way that you could with cPanel before. More customers relied on this than I had realized, and this led to the realization of the next issue. This has since been fixed.

  • Users don't know their cPanel passwords

Many users came to us not knowing their cPanel password, therefore not knowing their DirectAdmin password. This was an oversight that can be avoided.

  • CentOS 7 default limits were off

There were several system limits that were way off from defaults. Most noteworthy was the max_user_watches value. I would have never anticipated this, and can't imagine what led to such an odd scenario that I've never faced. Yet, I'll know to look for it next time.

  • Dovecot needed to be tweaked

Dovecot performance quickly tanked after the migration went live. It turns out this was mostly due to the previous issue, but we still did need to tweak some for the level of traffic going through the server.

  • Roundcube contacts did not port over

This has been an issue in most migrations. It was an easy fix when cPanel defaulted to using MySQL for the Roundcube database, but now they use SQLite and it's not reasonable to port them over. It impacted only the few customers that use them, but could have been better planned for.

  • Autoresponders ported poorly

Autoresponders ported in such an odd way that they returned errors rather than messages, and though that didn't actually stop the flow of email it was poorly handled and could have been better prepared for.

  • No redirects for services like webmail, cPanel

We could have redirected ports 2083 and 2096 for cPanel and webmail, and would likely do that if we were to repeat these events.

So what now?

Now we re-evaluate where we stand. We dropped some overhead by getting rid of a cPanel server. That may be sufficient, but there are at least two more servers that could possibly receive the same treatment (Acadia and Aus). If we decide to continue down this path, we've identified what went wrong after interacting with our customers, and we should be able to mitigate those issues for future actions. If we cannot mitigate them reasonably, we would not be able to proceed with any more migrations of this type.

I don't know what we'll do next as I need to evaluate the methods for mitigating these individual issues that occurred. I apologize sincerely and hope that I've made it up to everyone by offering a free 100GB upgrade (even to $5/year and lifetime 10GB customers) to everyone on the London server, as well as credits and/or refunds to those who suffered the worst issues. Whatever happens next, I know that we'll be better equipped for it, and if it involves changes then customers will be better informed of it.

 

Learning as we go

- Posted in News by

Learning how to scale out a support system in a small business is one of the most challenging things that I have ever done. MXroute support wait times are high, and the workload grows at a rate which exceeds my personal ability to answer the questions. I can look at this as a failure and retreat, or I can see it as an opportunity to learn and improve.

When scaling out a support infrastructure, it's easy to learn the ways not to do it. You experiment, observe the result, and quite often your theory doesn't hold up to reality. As discouraging as failure can be, no one articulated the opportunity it presents better than Thomas Edison:

I have not failed. I've just found 10,000 ways that won't work.

There in lies the opportunity. Each failure is simply a lesson in what not to do. Here's what we've learned doesn't work:

  1. A staff-created Knowledgebase.
  2. Exposing common questions prior to accessing the contact form.
  3. Using Slack to siphon off quick questions and prevent tickets.

So once again we're trying a new strategy. We're going to change the support workflow in pieces. I'll go over the rough plans for that now.

This is to allow you to ask the questions in your own words. Perhaps even duplicate questions because one of the existing ones may not be worded in a way that it looks like it could answer your question. This creates a community-assisted database of material that can be used to expose better answers to questions prior to opening support tickets. That brings us the next step.

  • Second, we're going to launch a chat bot.

This is a heavily trained bot that will be intended to help you identify the right answers to your questions without ever needing to speak to a human.

Now, we value human support just as much as you do. We also recognize that having a human answer every question is simply not a viable strategy at our scale. The more questions we can answer without a human, the more available we can be when you actually do need to talk to us. It's about maximizing the quality of human interactions by automating the repetitive ones.

Rest assured that MXroute support is not dead, we're just struggling to find the right answer. We've found a bunch of ways that don't work, so we're that much closer to the ones that do.

 

Affiliate Payout Update

- Posted in News by

Hey everyone!

I updated you a while back on the closure of our affiliate system. There are still many of you who have pending payouts, and I wanted to post an update to let you know that I have not forgotten about you. I'm trying to balance a few things here, and for transparency I'll share them:

  1. Growing our outbound infrastructure to sustain our current customers and some growth beyond it.
  2. Paying for additional help on support.
  3. Costs associated with migrating infrastructure (with overlap).
  4. Affiliate payouts.

Now, in all honesty affiliate payouts are last on that list because they are last in priority. The reason being that those payouts rely on the previous 3 items going over well, and being well paid for. The first and third on the list are temporary, and they ensure the continued funding of MXroute. That continued funding is what ensures the payouts.

We're by no means in any financial bind or at risk of failure, simply prioritizing the items that ensure our continued success. At the latest, we expect to have these affiliate payouts completely resolved within a few days after Black Friday.

Thank you so much for your patience. We're not going anywhere, we haven't forgotten about you, and we're so incredibly grateful for you.

Jarland

 

Affiliate System Shutdown

- Posted in News by

Greetings friends!

It is with great regret that I announce today that we have had to shut down the affiliate system on MXroute. All amounts owed are pending payout and will be paid out. We will always make good on our promises, even if it takes some time for this to happen.

We greatly appreciate every one of our affiliates who recommended customers to us, and we have been thrilled to share the wealth throughout this process. However, we are noticing that the amount to pay out is increasing while our profits are not. You have likely also noticed the high support wait times as we continue to struggle on that front, offering support mostly through our Discord chat. It has been decided that this is in the best interest of both MXroute and it's customers, that we move toward paying for additional help with our support.

Thank you all for your wonderfully kind words about us, we do hope that you will continue to recommend us, and we hope to grow into the kind of company that you will want to recommend for years to come.

Kind Regards, Jarland

 

Admitting failure and moving forward

- Posted in News by

Admitting failure is an extremely vital part of running a business. Often we tend to double down on our failures in hopes that more effort will convert them into success. That, however, is a great way to take a business toward it's end. That is especially true when the business is small and run by few people. Failure will happen often and repeatedly, if you're doing anything that is even worth doing. You must continually recognize it and choose how to move forward after it.

What I'd like to talk about today is where I've failed, and what I'm doing next. You most likely read my previous blog entry about our issues dealing with support workload and meeting my own expectations. While I stand by the concept that was outlined for repair, but I failed to recognize that I lack the resources to turn that concept into reality. As a result, these are my current failures: Handling support requests, resolving product issues.

To address these failures, three things will be happening. The first has already occurred, but will be made more clear with the others.

Support

I have split up support ticket categories into specific ticket types with specific input fields relevant to that topic. These are things like product bug reports, email delivery issues, feature requests, etc. This allows better prioritization of tickets based on the actual issue type and severity. This also allows us to set more clear expectations. For example, I can say that a feature request is the lowest priority up front. I can also say that a bug report clearly identified to be a client side issue can be lower priority than a platform wide issue. The remainder of issues will be handled as community support, via our Discord chat. Community support does not mean that I will not be present there myself, merely that the support will be a community and as we grow customers will be able to offer assistance to each other as well.

Product Simplication

We are going to make a sweeping change to our product offering that will impact all new orders. By lowering the complexity of our product offering in general, we will eliminate a variety of reasons to need support. This is not live at the time of this post, and you will know it is live by the third change.

New Expectations

A new website will be launched with the simplified product, outlining more clearly what is to be expected of us, what items I will support, and what product functions exist. This reduces the need for new customers to require support with things I consider basic like configuring their DNS, their email clients, etc.

With these efforts I hope to take what is growing failure and instead grow with new expectations of myself and my customers, clearly communicated and outlined, so that it instead becomes success. This is a top priority, and more will be visible over the next few weeks.

 

The State of MXroute Customer Support

- Posted in News by

Hey friends,

I wanted to take some time today to more formally acknowledge the state of MXroute customer support. Additionally, I wanted to tell you what led to the current reality, as well as what is being done to ensure resolution of the least favorable details.

1. The current state of MXroute customer support.

If you have opened a support ticket at MXroute in the last few months, the odds are high you have noticed that we have a high wait time for a response. Ticket response time has been anywhere between an hour and a month. For someone committed to providing a service as intricate as email, this is not something of which to be proud.

2. The catalyst.

I have identified three items that, collectively, created the current reality.

First, we blew up in popularity. You recommended us to your friends, family, and complete strangers. Thank you! I'm sorry to blame such an incredibly positive thing for a negative situation. The truth is, great things can have negative consequences. You're not at fault though; it merely tore open a wound that I created in the first place. You'll see that next.

Second, I created this service for system administrators like myself, the nerdiest of nerds. It wasn't intended to be well documented or easy for a new user to understand. I know now that it doesn't matter who I created it for, I made a mistake when I failed to build it in such a way that less technically inclined users could feel right about and the experience and understand how to use the product.

Third, I built the business model around the idea of customers being system administrators who just wanted to get out of the email game and spend that time doing more important things. The business model came with a projection of average expected support volume per customer, which was quite low. For a long time, this projection proved accurate, and it reinforced the business model. When our customer base began to shift, the support volume per customer changed in such a way that the business model was unsustainable. I could not manage it alone, and I could not hire anyone to help because of the low prices I chose to charge.

3. The wrong answer.

It is essential to speak about the reason why the most obvious answer is not the correct one for us. The obvious answer is the first one that anyone logically arrives at:

"Spend more time answering support tickets, get faster at solving them. "

It seems logical, almost too much so. If it were that easy, why are companies continually losing their quality of support as they grow? Do we honestly think that no one at Comcast had the idea of "If we answer customers quickly and give them what they need when they reach out, our support will be great." They run a vast infrastructure and, for the most part, it does what it's supposed to. They are not that unintelligent.

If growth is not in line with the business plan, and you continually throw all of your resources at support, how long will it take before you have depleted both your time and financial resources, and you are no longer able to grow your efforts at a rate that is equivalent to the customer growth? It isn't a question of if it happens, it's a question of when.

It is at this stage that we should consider creative solutions. I have landed on a solution, and I believe it is the right one for both MXroute, and it's customers:

4. The answer.

Available resources to answer support tickets must be split. We should use half of it for responding to tickets, and the other half of it should be used preventing tickets through product improvement, documentation, and policy. In this way, we will slow the rate of growth of the ticket volume, which increases the effectiveness of the time spent answering tickets by allowing that time to make a more substantial dent in the overall volume.

With this plan, we intend to improve your experience with our service, improve the clarity around our features, and only lose potential temporary gains on ticket response time.

5. Conclusion.

Business is hard. I don't think there is anyone among us who only makes the right decisions. Increasing our pricing was not on the table as a solution to this problem, I will provide what I set out to provide, at the prices I have charged along the way. I've got your back, and I'm incredibly grateful that you have mine as well. You have all shown an incredible amount of understanding concerning this event. That is why I felt you deserved to know every detail.

 

Hosted DNS no longer supported

- Posted in News by

A while back, we suggested that customers use our hosted DNS to make their lives easier. As it turns out, it doesn't really do that at all. Rather, it adds unnecessary complication. It has been consistently simpler for customers to host their DNS elsewhere than with us, which in turn takes time away from improving the actual base product (email). As a result, we announced that it would no longer be supported a while back. That announcement was simply done via our billing panel, not via proactive notice. Today, I'm letting you know directly that this feature is no longer supported and I highly recommend moving away from it if you are using it. This is only important if you are using nameservers of ours, like ns1-3.mxroute.com, ocean/ocean-ns2.mxroute.com, etc. What is currently online will not soon be removed, but future efforts may continue to work against this feature and it is possible that it will be broken in many ways.

If you are not using our hosted DNS, there is no concern here, you may ignore this notice. If you are, your DNS is not being taken offline intentionally at this time. It could be months or years before that might happen (if even then).

Kind Regards,
Jarland Donnell
MXroute Administrator