The Email Hole

Email is not something to get too upset about, until you lose one to downtime by your provider of choice. And then you realise that it is too important to be left to free services, or even a domain hoster.

I use a hoster called Hostway, and they went spectacularly down last week. (This despite the fact, or perhaps because of it, that Hostway launched a new service recently offering 150 GB of space for $10 a month.) It was only about a day, but several domains I based there lost email access when their storage failed. Now I have no idea who might have been trying to reach me and couldn’t because of bounced emails, what newsletters I’ve been removed from because of bounced emails, what email newsletters I may have missed

Now this kind of thing happens, but it made me realise that losing one email is the same as losing all of them if you don’t know which email it is, since it may be the important one you’ve been waiting for offering you money/marriage/a new nose. Email is different to hosting a website: a website can go down, and you’ll lose some traffic, but it will come back up again. Email is a stream of discrete bits of information, and there’s no way of telling whether there are any missing.

In short, a good hoster needs to guarantee that, should something go wrong, no email is left behind. Hostway have not, so far not been able to assure me of that. They say that emails lost during the outage have been recovered, but as far as I can work out that does not refer to those lost because of the outage — in other words, those emails that were stored on their servers and not recovered by users before the outage hit. (Emails to their technical staff about this were responded to with pasted notifications from their support team, which didn’t address this issue.

This surprises me, but shouldn’t. They are listed by Netcraft as the second most reliable hoster last month and I’ve not had many problems with them. But they are a domain hoster, which means that bullet-proof email is not top of their priorities. As Syd Low of AlienCamel puts it (declaration of interest: I’ve been using Syd’s email service the past few years, and it’s rock solid), there are three types of email service: bundling services (like Hostway), free services (like Gmail) and paid services (like AlienCamel) which provide Web access, lots of redundant backups to make sure no email goes missing, plus anti-spam, anti-virus and anti-phishing features.

My lesson from all this: email is too important to entrust to people who don’t take it seriously, or who aren’t getting money for your business. Of course, no one wants to pay for something they’re getting for free, or more cheaply, but sometimes free and cheap is not enough.

From the Ashes of Blue Frog

The Blue Frog may be no more,  but the vigilantes are. Seems that despite the death of Blue Security in the face of a spammer’s wrath, the service has built an appetite for fighting back. Eric B. Parizo of SearchSecurity.com reports on a new independent group called Okopipi who intend “to pick up where Blue Security left off by creating an open source, peer-to-peer software program that automatically sends “unsubscribe” messages to spammers and/or reports them to the proper authorities.”

Okopipi has already merged with a similar effort known as Black Frog and has recruited about 160 independent programmers, who are dissecting the open source code from Blue Security’s Blue Frog product. The idea seems to be the same: automatically sending opt-out requests to Web sites referenced in received spam messages, the idea is to over-burden the spammer’s servers (or those of the product he’s advertising) as a deterrence and incentive to register with Okopipi. By registering he can cleanse his spam list of Okopipi members.

Some tweaks seem to be under consideration: Processing will take place on users’ machines and then on a set of servers which will be hidden to try to prevent the kind of denial-of-service attack that brought down Blue Frog.

Possible problems: I noticed that some of the half million (quite a feat, when you think about it) Blue Frog users were quite, shall we say, passionate about the endeavour. These are the kind of folk now switching to Okopipi. This, then, could become an all-out war in which a lot of innocent bystanders get burned. The Internet is a holistic thing; if Denial of Service attacks proliferate, it may affect the speed and accessibility of a lot of other parts of it, as the Blue Frog experience revealed. (TypePad was inaccessible for several hours.)

Another worry: Richi Jennings, an analyst with San Francisco-based Ferris Research, points out on Eric’s piece that project organizers must ensure that spammers don’t infiltrate the effort and plant backdoor programs within the software. “If I’m going to download the Black Frog application,” Jennings said, “I want to be sure that the spammers aren’t inserting code into it to use my machine as a zombie.” I guess this would happen if spammers signed up for the service and then fiddled with the P2P distributed Black Frog program.

Another problem, pointed out by Martin McKeay, a security professional based in Santa Rosa, Calif., that spammers will quickly figure out that the weak link in all this is it rests on the idea of a legitimate link in the email for unsubscribing, and that spammers will just include a false link in there. Actually I thought the link Blue Frog used wasn’t unsubscribe (which is usually fake, since if it wasn’t would then pull the spammer back within the law) but the purchase link. How, otherwise, would folks be able to buy their Viagra?

One element I’d like to understand better is the other weakness in the Blue Frog system: That however the process is encrypted, spammers can easily see who are members of the antispam group by comparing their email lists before and after running it through the Blue Frog/Black Frog list. Any member who is on the spammer’s list will now be vulnerable to the kind of mass email attack that Blue Frog’s destroyer launched. How is Okopipi going to solve that one?

The Red-faced Blue Frog

What’s intriguing about this Blue Security/Blue Frog episode, where angry spammers attack the anti-spam company with a Distributed Denial of Service (DDoS) attack, which in turn directs traffic (unwittingly or wittingly, it’s not clear yet) and temporarily brings down blog hoster TypePad, is this: The guy behind Blue Security, Eran Reshef, is founder of Skybox, a company “focused on enabling the continuous enterprise-wide assessment of vulnerabilities and threats affecting corporate networks.”

This is at best somewhat embarrassing for Reshef, and for Blue Security, at worst it exposes him and the company to ridicule and lawsuits. Getting involved in battling spammers is not a task taken on lightly, and the one thing that Blue Security had going for it was that it seemed to know what it was doing. Users download software and register their email addresses in a central database. Spammers are encouraged to remove those email addresses; if they don’t, the software will respond to subsequent spam by visiting the website advertised and automatically filling the order form. If enough people have the software running this, in theory, creates an overwhelming amount of traffic for the spammer and brings their business to a halt. Blue Security now says it has tens of thousands of members.

But then came last week’s attack. Reshef initially said that that no such DDoS took place on the www.bluesecurity.com server, something contested by some analysts. He has since said that a DDoS did take place, but against operational, back-end servers  and not connected to his company’s front door. This, he said, he only spotted later. He says that when he redirected traffic to his blog at TypePad there was no DDoS on the bluesecurity.com website; that, he says, came later. This appears to be borne out by web logs provided to TechWeb journalist Gregg Keizer.

Blue Security’s handling of this raises more questions than it answers. Many are highly technical and not ones I understand. But there are some basic ones. Was the company not prepared for spammers to retaliate? Did it not have any procedures in place? Why did it redirect traffic to TypePad without informing them first? Why did it not coordinate closely with its ISP? And why, given Reshef’s expertise on DDoS attacks with Skybox, was he not able to spot the DDoS attack on his backend servers?

The Blue Frog vs PharmaMaster

I’ve been trying to make some sense of this recent drama involving Blue Security, an anti-spam registry that effectively tries to deter uncooperative spammers by overwhelming their servers, and recent outages at TypePad and LiveJournal apparently caused by a revenge attack by spammers on Blue Security. (Here’s some more information on Blue Security and the Blue Frog.) The outages were caused when Blue Security redirected the spammers’ attacks on its website to the company’s blogs which were hosted on TypePad and LiveJournal.

So what really happened?

  • Blue Security’s web site has been under attack for most of this past week, via a distributed denial-of-service (DoS) attack which basically tries to overwhelm a site with traffic sent from as many computers as possible (the site is now back up);
  • To try to deflect the attack, which effectively suspended its service, Blue Security changed its Internet address to its TypePad blog;
  • This overwhelmed SixApart’s servers, temporarily affecting all its blogging services, including TypePad and LiveJournal;
  • Meanwhile, spammers presumably linked to the DDoS attack sent threatening emails to, apparently, anyone on the list of the Blue Security do-not-intrude registry. Blue Security works by building a network of users who report spam. The source of the spam is then contacted and then asked to remove all email addresses of its members from their spam lists. If they fail to do so, software installed on users’ computers fills out forms on websites linked to in any subsequent spam, creating a wave of traffic to the spammer’s web site, that, in theory, brings the spammer’s activities to a stop.
  • The spammer, or another spammer, then contacted Blue Security via ICQ instant message, to taunt and threaten the company, apparently in a bid to stop its activities.
  • The spammer, or another spammer, has also been sending emails containing Blue Security contact and registration information. This might have been done in the hope of getting recipients to complain to those email addresses and phone numbers to further overwhelm the company’s resources.

This account is not uncontested. According to a Blue Security press release:

  • Blue Security claims that it was not the victim of a DDoS attack, but that the spammer — identified as PharmaMaster –– persuaded a staff member of a top-tier Internet Service Provider to block Blue Security’s IP address at the backbone. This would have blocked all traffic from outside Israel, where the Blue Security web site is located.
  • Blue Security then closed its web site and posted a note on its blog (hosted elsewhere.)
  • Shortly afterwards, Blue Security says, PharmaMaster launched a DDoS attack on any site associated with Blue Security, causing outages at five top hosting providers, a major DNS provider and a popular blog site.
  • Blue Security has denied reports, including one by the Associated Press, saying that its do-no-intrude lists have been compromised. Blue Security works by allowing compliant spammers to run its email list through a program which compares it with a special encrypted list of Blue Security members. While the spammer is not able to see or access the Blue Security list, Blue Security members’ email addresses will be removed from the spammer’s list. This is done, in part, so individual Blue Security members are not then known to a spammer, and so the spammer cannot gain access to the Blue Security registry for spamming purposes. The AP report suggests the spammer has figured out a way to work out which email addresses belong to Blue Security members by merely comparing its own list before and after running it through the Blue Security removal process. Those email addresses no longer on the spammer’s list must be Blue Security members, the report says.

This account is contested by some security analysts, who point out what they say are some inconsistencies in Blue Security’s account:

  • Elsewhere Blue Security’s Eran Reshef acknowledges that Blue Security didn’t just post a note on its blog, but it redirected traffic from its bluesecurity.com URL to the TypePad blog. He is quoted as saying he didn’t anticipate that the spammer would launch a DDoS attack on such a large player. “I didn’t think he was so crazy as to attack them,” said Reshef. This raises the question: Was this done before or after the DDoS began? Rashef says it was.
  • If Blue Security’s routing was changed internally, as Blue Security suggests, there should be a record. One analyst says he can find no record of anything “fishy.”

Blue Security clearly has its supporters. An article on one website has received, at the time of writing, more than 200 comments. The Blue Security blog’s single post received more than 100 before comments were closed.

Perhaps one of the most interesting aspects to all this is how clearly at least one spammer perceives Blue Security as a threat to its business. Not only is it trying to scare the company and members of its registry into abandoning their approach, but it is also adopting more open tactics: contacting the target directly via ICQ, perhaps in an effort to intimidate or negotiate, and to email and post comments to the above websites to try to scare members into removing their names from the registry and uninstalling the software that returns spam to the sender’s servers.

You don’t need to agree with Blue Security’s tactics to acknowledge they must be making some kind of impact for this to happen. What is perhaps a little bit scary is that Blue Security don’t seem to have been ready for this attack, and reveal some naivety and lack of understanding about how the Internet works by merely redirecting the assault to other servers. Not only would this not solve their problem, it also exposes them to legal action by the companies behind the redirected servers if it emerges that they were not informed beforehand. Still a lot of questions to be answered on this one.

The Anger of the Blogger Spammed

There’s something just so lame about comment spam dressed up as a legitimate comment that it gets me angrier than I do with ordinary spam, blog or otherwise, for some reason. (Comment spam/blogspam/linkspam is when individuals automate posting of comments on blogs to build traffic and Google rankings by having links to their sites on other sites. Some comment spam is just gibberish, but would still boost Google rankings because of the links contained somewhere in the comment, while others pretend to be legitimate comments.)

I think it’s because I’m as much a sucker as the next guy for anyone saying anything nice about me or my blog, and the anger of realising I’ve just been spammed by some dork who wants to promote their website on your real estate is of a deep, visceral kind.

This I just got on a posting about the weirdness of online auctions in Singapore:

Excellent Blog. Very informative. And very well organized.

Online Auctions are really looking up with more and more people interested in buying and selling product online.

Keep it up. We need more such blogs which provide quality information.

No sign in there the writer has actually read the blog. Clearly just a blast at all blogs mentioning the word ‘auction’. In the name and URL field of the comment the sender gives his name and his website. I would publish both here but it would just drive traffic, and I’m guessing if the guy is already stooping to comment spam he’s not going to be shamable. Still, if you were to block all comments from 202.65.144.5 you might be doing yourself a favor. And let’s just say the spammer in question is quite prominent in Indian circles as “an Internet Entrepreneur, Online Biz Consultant, Hypnosis & NLP “Guru” and a Prolific writer.” Prolific as in prolific spammer?

Bottom line: Please don’t comment spam me. All comments have to be approved first so you’re just wasting my time and yours, not the reader’s.  And shouldn’t we be treating comment spammers like ordinary spammers, and making all efforts to shame them and inform their ISPs?

technorati tags: , ,

The Blue Frog Claims Some Early Success

Blue Security, the anti-spam company I wrote about somewhat skeptically for WSJ.com (subscription only, I’m afraid) a few months back, are claiming initial success in their Do Not Intrude Registry. (Simply put, users sign up for the service and Blue Security threatens a kind of mass ‘visit’ to any spammer that continues to spam any user. The ‘visit’ would slow down the spammer’s server to the point he couldn’t operate.)

A press release from Blue Security says that nearly 30,000 users have joined the service, more than a quarter of whom have reported spam dropping by at least half. Those are impressive figures:

Blue Security has collected data showing that spammers who are receiving the opt-out requests have begun to comply. Community members will see the change gradually as more spammers comply with the Registry and remove member e-mail addresses from their mailing lists.

I was skeptical about the service because I wasn’t convinced it was either ethical or legal. The threat is basically to launch a Denial of Service attack on the spammer if it doesn’t comply, a move which is too law-of-the-jungle for most tastes. (Blue Security denied that what they’re doing is illegal.)

But perhaps there’s some merit to what they’re doing? If their figures are correct maybe Blue Security are onto something. As their press release says:

The Do Not Intrude Registry is based on the concept of changing the spam economy, a process that takes time.

How To Cripple A Customer In The Name Of Spam Catching

Why does a hosting service remove features in the name of improved service?

After a lot of toing and froing, I’ve realised why a lot of emails being sent to me were bouncing. It’s because my hosting company, Hostway, have upgraded all their domains to an Advanced Mail Service, which offers extra spam and anti-virus features. Unfortunately, they have in the process removed a feature that was my own anti-spam service, using something called Catch-All Aliases. And now I’m stuck in email hell.

An ‘alias’ is an email address that you set up that doesn’t really have its own account, it merely routes through to an existing account. So, say my email address is joe@bloggs.com, I could add an extra email address — an alias — for enquiries to sales@bloggs.com. In fact me and sales are the same people, and the emails end up in the same place, but it’s useful for me to offer a second address to a) impress people, b) let my email program sort them into separate folders to keep, say, business and pleasure separate.

That’s the alias bit. The ‘Catch-All’ bit occurs when you send an email to an address that doesn’t exist at bloggs.com. So say I sent an email to brian@bloggs.com. Now brian doesn’t exist, or has left the company. But if I had set my joe@bloggs.com address to be the default, Catch-All address, that email would end up in my inbox, just like the sales email. The advantage of this? Well, there are several. Brian’s emails don’t get lost, just because his email account has been deleted, or someone misspelled his name. But it also means you can give out any email address you like to websites, mailing lists or individuals and let your email filters do the work. So, for example, any Google email alert you sign up for could be given the email address google@bloggs.com, so that you could set up your email filter to move all those emails into one place. Any email alert from Factiva could go to factiva@bloggs.com. You don’t have to registered any of these email addresses. They just end up in your default email box. But by setting up a filter, your email program can look inside the header, see it was addressed to ‘google@bloggs.com’ and funnel it through to a folder called Google alarts. Neat.

Not just that. If one of these virtual email addresses falls into the hands of spammers, you have a pretty good idea of who gave it away and can give them a hard time. In short, Catch-All Aliases are a great way to keep control of your inbox. I used them all the time, and must have set up more than 150 of them on the fly. The beauty of them is you don’t have to do any setting up; you can make up a name on the spot, and then when an email comes in with that address in the header field, you can decide what kind of filter to apply to it.

Not any more. Hostway have this week decided to ditch Catch-All Aliases. The tech guy tells me it’s because they were creating too much spam. (This is when spammers just blast lots of emails at bloggs.com with any old combination of letters and digits before the @ sign. If there’s a catchall address all these emails get through.) Now spam is a problem for hosters, I can appreciate. But to make a move like this, without allowing the user any chance to opt out of such a change, is to me extraordinary. I had a good antispam system going. I didn’t need their service. But I didn’t have any choice.

Now in the short term I’m left with the chaos of trying to remember what aliases I’ve come up with over the past few years and physically adding those addresses to my list of registered aliases on Hostway’s configuration page. In the longer term, of course, I’m going to be looking for another hosting company. Any recommendations?

Bottom line here, is the cost of spam to people like Hostway must be making them buckle and pushing them into illconsidered moves like this. But I would beg other companies to ponder deeply before removing features that their customers may really, really need. If I were in business I would be losing customers every time an email to me bounced. As it is I am going to be registering aliases deep into the night and hunting for a new home for my domains. Not how I planned to spend my time.

InspectorBrown Responds

Here’s what Rick Brown said of his Inspector Brown anti-phishing toolbar in response to my questions about its failure to catch the cross scripting phish mentioned here:

Our software works to protect our community of users and allow each user the ability to fight back against spam, phishers and online fraud.

Yes, its true, not all smart people will care to report bad links or websites, but a percentage of users will gladly do so.

The idea is simple, when a member of our community gets an email from a known spammer or phisher, they report it, either by sending an email to reports@inspectorbrown.com or clicking on the “Report a Site” button from the Inspector Brown toolbar. Immediately, once the site is reported, our software goes to work analyzing the site for clues. How long has the site been active/registered online? Is it IP based, does it show certain patterns that make it stand out?

The toolbar was also designed as a marketing tool. Financial institutions and any large corporation wanting to protect and promote their image can benefit from a branded toolbar that shares a common database with other businesses. If certain smart employees or users report to our system every user using our software gets the same protection. The toolbar was designed to allow additions such as links to certain departments within a company, information tickers for stocks or weather, the options are endless.

Our software differs from spam blockers as they are what we call “band aid” approaches. Spam is still sent to the users and may end up in spam folders, however some emails such as your message to me, was sent inadvertently to my spam folder even though it was legitimate email. All this traffic affects the ISPs and corporations and users who rely so heavily on email.

What if you went to the grocery store and bought 100 dollars worth of food, brought it home only to find out that $70 of the food was bad? You would be pretty upset. However, ISPs constantly send all of us unwanted e-mail that makes up the majority of traffic sent via our Internet connections.

Our software intends to weed out the bad traffic. If users can’t access the websites of spammers and phishers, they can’t purchase their goods or fall victim to their crime. The criminals will have to resort to other methods. The more users who become part of our community increases the chance of a percentage of users who will be vigilantes and want to fight back, stopping the bad guys from invading our lives. The more users who join our community increases the speed at which the sits are reported. Each user is given a score to determine the trust level we have with each user. This prevents the bad guys from using our software to “punish” their competition.

There is no perfect method to stop spam and phishing scams, but our software adds one more layer of protection in a unique way.

Thanks, Rick.

Where Did That Email Come From?

An interesting new tool from the guys behind the controversial DidTheyReadIt?: LocationMail. (For some posts on DidTheyReadIt, check out here, here, here and here.)

LocationMail tells you where e-mail was sent from. It uses the most accurate data in the world to analyze your e-mail, trace it, and look up where the sender was when the message was sent. Find out where your friend was when she e-mailed you, or where a business contact is really writing from.

LocationMail integrates seamlessly into Outlook or Outlook Express; once installed, it shows you location information next to each message. LocationMail shows the City, State, Country, Company, ISP, and Connection Speed of the sender.

Installs painlessly into Outlook but crashed my Outlook Express. In Outlook a popup window appears with details of where the email was sent from, including the company, location, connection type, domain and IP address. LocationMail does this by using what it thinks is the IP address of the sender and running it through data from DigitalEnvoy and IP registrars. (A fuller explanation is here.) The makers hope to target a range of customers:

With phishing and other forms of Internet fraud becoming more and more problematic, LocationMail protects you from e-mail based frauds. The program can tell you if an email you seemingly received from your local bank was actually sent from a location half way around the globe. By instantly tracing the source of your emails, LocationMail helps keeps you safe from identify thieves. LocationMail lets you identify and eliminate fraudulent transactions from eBay and other Internet-based auction houses.

LocationMail protects companies who accept orders by email. Credit cards are regularly stolen from people in affluent countries, and used for placing online orders by criminals from other countries. By telling you an email’s origination location, the program helps you detect fraudulent inconsistencies.

Whether you’re a business person who wants to keep track of the demographics of prospects and customers, a manager who wants to ensure that incoming email addresses are legitimate and consistent, or a home computer user who is curious about where friends are e-mailing from, LocationMail has the tools that you need.

It costs $30. Another program that does something quite similar is eMailTrackerPro which will also identify the network provider of the sender, including contact information for abuse reporting, and uncovers the ‘misdirection’ tactic commonly used by spammers. Of course, LocationMail may not help that much, since legitimate emails might not, in Internet terms, originate from the place where they should. But it does a pretty good job and is useful if, say, you’re not sure about whether an email is spam or not (it does happen) the fact it originated in Seoul should provide a clue (unless you know lots of people in Seoul, of course).

And most importantly, this isn’t an invasive technology.

A Better Way To Measure The Spam Flood

Here’s an interesting take on spam which helps illustrate how big a problem it has become.

Florida-based email service ZeroSpam Net (0SpamNet) says (via email, afraid no URL available at time of writing) that current methods of measuring spam, as a percentage of total email traffic, has become meaningless.

Two years ago, seeing Spam grow from 60% to 70% in a month or two had some meaning. Over the last couple of months the impact of Spam growing from 85% to 90% has been lost by being reported as a percentage. That last 5% of growth as a percentage of total traffic represents a 50% growth in the total volume of Spam. Measurement of Spam volume as a percentage of total traffic is a poor indicator of the ever increasing size of the Spam problem.

Instead it proposes an index, which it calls the ZSN Spam Index, which accounts for spam and legitimate email growth against a constant reference value of 100 valid messages. This takes into account the increase in normal email traffic — roughly 12% per year. The index goes back to November 2002, with a value of 66.67 — i.e. about 67 spam messages for every 100 valid emails. Now the index is at 782.12. That’s 800 spam messages for every 100 valid ones. Gasp.

Here’s the chart (PDF).

Why do people never talk about CAN-SPAM anymore, I wonder?