Spammers want to use your domain

- Posted in News by

This is a topic that should interest all MXroute customers. I want to stress that this is not about a security issue with MXroute, and no MXroute services have been compromised. The only compromises I am referring to are customers using the same passwords across the internet, where other services may not end up being safe places.

Over time it has become increasingly apparent that spammers want to use the MXroute infrastructure to send spam. In fact, I'm convinced that they specifically target our customers to try to find ways to get inside and use our infrastructure to send that spam. For this reason, I want to stress that you should be using unique passwords for every single account on the internet. I won't force you to, but I might not be able to do business with you for long if you continually use passwords that are easily stolen somewhere else on the internet and then used to abuse our service.

I've noticed that when spammers compromise customer account passwords at MXroute, they do not brute force (attempt passwords until they find one that works), and they do not use any secret path (like a compromised server on our side). Instead, they directly authenticate as our users over SMTP, using our customer's email passwords. These events tend to happen in groups by server. This leads me to believe that the spammers are first looking for groups of customers (perhaps by MX record), searching to see who they can find on compromised password lists (passwords found from compromises of other services, where email login might correlate), and then they use one or two email accounts per day until they exhaust their findings. This typically only occurs in bursts of 1-3 days and then they rest for a bit.

If you've dealt with IT security, none of this is news or interesting (except perhaps the pattern that we see). If you haven't, this may raise some questions. I'm going to take a shot at guessing these and answering them.

Are you sure this isn't a problem with MXroute security, and that your servers haven't been compromised?

As much as anyone can ever be. I may not know about some unpublished vulnerability used by state actors to undermine governments, but I do keep up with known vulnerabilities for both the open and closed source software that we use and apply patches as soon as they are made. I do audit the security of the servers regularly, and there have never been any signs of intrusion. Given how common it is for people to reuse passwords, and how many services have been compromised over the last decade, it is far more reasonable to assume that I am accurate in my understanding of the cause of these events.

Why don't you enforce stronger passwords?

A couple of reasons, actually. The first being that if you reuse a password across the internet that matches security requirements, this does nothing to prevent the situation. The second is that there is no increase in security by customers leaving in annoyance and using another service that does not have these requirements. In the past when we had enforced this, customers did complain about stronger password requirements and they did take their business elsewhere to not be subject to it. Sometimes meaning well doesn't result in a positive net impact on the result.

Where do people find these passwords?

Over time there have been so many compromised services across the internet, and in recent years people have begun compiling databases from these compromises into what are called "combolists." These are gigantic databases of known login information from compromises that go at least as far back as the first major Yahoo compromise. I can show you three places right now where you can go and probably find some of your old passwords on a number of combolists:

https://cracked.to/Forum-Combolists
https://weleakdata.com/forumdisplay.php?f=17
https://raidforums.com/Forum-Databases

Scary, right? I've found a number of my old passwords in there. I've found incredibly important corporate accounts for major businesses in them as well. This should highlight why it's important that a password compromise ONLY compromise what was compromised, and not all of the rest of your internet accounts.

What do you do to combat this?

I suspend outbound mail for the compromised accounts and email the customer to let them know. These days it doesn't happen for long before it's caught, and plenty of filters prevent a number of these events, but it still happens and should be highlighted.

Why not use two-factor authentication?

We wanted to provide basic SMTP/POP/IMAP service, and not something that requires direct integration by the developers of your email applications. The protocols themselves do not support two-factor authentication. We can talk about things like app passwords (unique passwords generated by us for your applications), but that really defeats the type of service that we set out to be and isn't actually two-factor authentication in the first place.

Aren't you afraid that this makes MXroute look bad?

Absolutely not. They want your email accounts because they want to use the service we've worked so hard to build out for you. We should be flattered, and you should know that it is an indication of the quality of work that is being done to provide you with a high quality service. The higher the quality of email delivery, the higher the value the service will have to spammers.

If you'd like to discuss this or ask questions about it, we've created a space for that:
https://community.mxroute.com/t/discussing-password-security

 

What we learned this week

- Posted in News by

This past weekend and the days after it, we learned some valuable things that we'd like to share with you.

How did we get here?

Originally I said that we had no intention of migration cPanel servers to DirectAdmin, though we were provisioning new customers on DirectAdmin servers. As the weight of cPanel's dramatic price increase took hold, we found that we were not able to secure the licensing deals that would allow us to barely float above water. Those licensing deals were calculated into our overhead increase of a bit over $10,000 per year, which meant that this figure was actually a fair bit lower than what we ended up paying.

What happened?

Because of the weight of the cPanel pricing increase, I decided to start experimenting with a migration from cPanel to DirectAdmin. I didn't know if this was going to work well at all, I needed to experience the process and see what worked and what failed. I started this process with a server that desperately needed a hardware upgrade, to take care of two problems at once (server migration was happening regardless). This was our London server (london.mxroute.com).

It took one day to perform the backups necessary to start the process, another day to transfer them to the new server, another to perform the restorations, and almost another to sync the changes that occurred on the old server during those previous days.

After the restores were complete, I noticed that the number of things which transferred perfectly turned out to be impressive. By my account, nearly everything had transferred flawlessly. This is where I faced a difficult decision, and it can be disputed whether or not I made the right one.

What did we learn?

It can be very difficult to gauge the level of failure in a particular change when only viewed through my own perspective. What seems, to me, like a flawless change can be viewed differently by customers who may use features that I've not thought to test, or who had use cases that I hadn't imagined. This was true with the migration of the London server, and I'd like to share with you what I've learned didn't go over so well.

  • Notification, or lack thereof.

I spent days testing to see if this could even work, I didn't want to confuse customers by emailing them all and saying "Hey, sometime in the next week this might change to that, or it might not, we'll have to see." Yet, at the end of it, I didn't want to waste the days that I had spent working toward something that I had deemed a success.

  • Catch-alls didn't migrate.

Catch-all accounts might be expendable to some, but might also be vital to others. That they didn't migrate is an oversight and one that varies in degree by customer.

  • Domain forwarders didn't migrate.

Honestly I didn't even realize customers were using them, forwarders for a whole domain where user@domaina would go to user@domainb, relative to the value of "user." This was another oversight.

  • Filters didn't migrate.

This I was aware of, and they couldn't have been migrated as the two filtering systems are incompatible. However, a more clear plan around how to handle this would have been ideal.

  • SSO in WHMCS broken

You couldn't just log in to DirectAdmin from portal.mxroute.com in the same way that you could with cPanel before. More customers relied on this than I had realized, and this led to the realization of the next issue. This has since been fixed.

  • Users don't know their cPanel passwords

Many users came to us not knowing their cPanel password, therefore not knowing their DirectAdmin password. This was an oversight that can be avoided.

  • CentOS 7 default limits were off

There were several system limits that were way off from defaults. Most noteworthy was the max_user_watches value. I would have never anticipated this, and can't imagine what led to such an odd scenario that I've never faced. Yet, I'll know to look for it next time.

  • Dovecot needed to be tweaked

Dovecot performance quickly tanked after the migration went live. It turns out this was mostly due to the previous issue, but we still did need to tweak some for the level of traffic going through the server.

  • Roundcube contacts did not port over

This has been an issue in most migrations. It was an easy fix when cPanel defaulted to using MySQL for the Roundcube database, but now they use SQLite and it's not reasonable to port them over. It impacted only the few customers that use them, but could have been better planned for.

  • Autoresponders ported poorly

Autoresponders ported in such an odd way that they returned errors rather than messages, and though that didn't actually stop the flow of email it was poorly handled and could have been better prepared for.

  • No redirects for services like webmail, cPanel

We could have redirected ports 2083 and 2096 for cPanel and webmail, and would likely do that if we were to repeat these events.

So what now?

Now we re-evaluate where we stand. We dropped some overhead by getting rid of a cPanel server. That may be sufficient, but there are at least two more servers that could possibly receive the same treatment (Acadia and Aus). If we decide to continue down this path, we've identified what went wrong after interacting with our customers, and we should be able to mitigate those issues for future actions. If we cannot mitigate them reasonably, we would not be able to proceed with any more migrations of this type.

I don't know what we'll do next as I need to evaluate the methods for mitigating these individual issues that occurred. I apologize sincerely and hope that I've made it up to everyone by offering a free 100GB upgrade (even to $5/year and lifetime 10GB customers) to everyone on the London server, as well as credits and/or refunds to those who suffered the worst issues. Whatever happens next, I know that we'll be better equipped for it, and if it involves changes then customers will be better informed of it.

 

cPanel Pricing Update

- Posted in Uncategorized by

As many of you may already know, cPanel has recently decided to increase the price of its control panel. You can read more about that price increase here:

https://blog.cpanel.com/announcing-account-based-pricing/

Currently, our yearly cost for cPanel is $3,780. After this change, our annual cost would be $14,124. As you can imagine, this is not accounted for in our business strategy.

Hopefully, by now all of our customers are well aware that we don't like to pass on increased costs to our customers. This increased cost is our burden, not yours. After spending much time reviewing our options, we have landed on a two-tier strategy:

Step 1:

We are migrating cPanel servers into a network that can save us money on licensing. This change will bring our yearly cost to $9,492. While still a dramatic increase, it is quite a bit less of one.

Step 2:

Replace cPanel. It's time for MXroute to grow out of cPanel, but there are many considerations to make along the way. We cannot migrate out without accounting for all of the variables, so this is not something we can put a deadline on.

We've quite perfected the strategy for migrating servers, and several of our older servers are set for migration to new hardware. Migration is not a significant process at this stage and goes mostly unnoticed by customers. We are not yet sure what this means for region-specific servers (Aus and London). However, we do have some time to work on those plans and provide you with an update.

Thank you so much for your business. I promise that every measure taken will be careful and thorough, with the primary goal being not to interrupt or downgrade your service quality.