tag:blogger.com,1999:blog-85230466727264552132024-03-07T13:27:47.210-08:00The Mailinator(tm) BlogEmail Workflow Testing for your QA Teampaulhttp://www.blogger.com/profile/11412172362500455307noreply@blogger.comBlogger16125tag:blogger.com,1999:blog-8523046672726455213.post-68787226769663228892020-07-14T04:21:00.000-07:002020-07-14T04:21:58.920-07:00This Blog has Moved !The Mailinator Blog has moved to:<div><br /></div><div>https://manybrain.github.io/m8r_blog/</div><div><br /></div><div>Check us out there ! </div>paulhttp://www.blogger.com/profile/11412172362500455307noreply@blogger.com0tag:blogger.com,1999:blog-8523046672726455213.post-31088148365378832802019-12-18T06:38:00.000-08:002019-12-18T06:44:11.795-08:00Mailinator Private DomainsThe public Mailinator system has always had a privacy policy somewhere along the lines of "There is No Privacy". (see our <a href="https://www.mailinator.com/v3/#/#privacy_pane">Privacy Polic</a>y page for the more comprehensive version). It's no wonder then we call that part of the system "Public Mailinator" - the design of the system itself doesn't allow privacy. Every inbox is available to anyone.<br />
<br />
People often think of that system as "disposable email" which isn't particularly wrong, but we more think of it as a website that accepts email. Forgot any notion you have of email "accounts" or "ownership". In the public system, every email and every inbox is public for every one.<br />
<br />
This is why a few years ago we created the concept of Mailinator Private Domains and the Mailinator Private system. Many companies use Mailinator to test their Email Workflow. Whether that's automated testing of their sign-up system or load testing deliverability (and speed) of their next marketing campaign. Mailinator can give them trillions of different email addresses on tap.<br />
<br />
No need to create accounts, anytime an email arrives at Mailinator (public or private), the inbox is created to house that incoming email. And that goes for Mailinator Private Domains too.<br />
<br />
Subscribers to the system get their own private domain. And that's private like - completely private to their team. The system will assign a new subscriber a private domain immediately (i.e. it will most definitely be something other than @mailinator.com). That being said, subscribers can change their private domain to any domain that they own - e.g. yourPrivateDomain.com.<br />
<br />
In either case you can use any inbox you want @yourPrivateDomain.com.<br />
<br />
Having your own private domain affords many more big advantages:<br />
<br />
<ul>
<li>Unlike the public Mailinator system - you don't just see one inbox at a time. You can see ALL inboxes in your domain in real-time - on one web page. Some subscribers send at pretty high speeds and it's amazing to watch what looks like a normal email inbox scrolling at high speed with incoming messages. </li>
</ul>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://www.mailinator.com/img/screenshot12.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="292" data-original-width="800" height="116" src="https://www.mailinator.com/img/screenshot12.png" width="320" /></a></div>
<div>
<br /></div>
<ul>
<li>An API. You can access all your private domain email via API (the API happens to work for the public system as well)</li>
<li>Message Rules. It's your domain - it's your rules. You can say "All email that arrives at joe@yourPrivateDomain.com" should be auto-forwarded to my gmail account". Or "all the links in the email should be automatically clicked". Or "the email should be converted to JSON and forwarded to this webhook". It's a pretty powerful system.</li>
</ul>
<div>
If you're using Mailinator for testing your Email Workflow, it'd be worth your time to try out the Private system. The advantages are real. Try out a <a href="https://www.mailinator.com/v3/?trialshow=true">Free Trial</a> today.</div>
Sam Liptonhttp://www.blogger.com/profile/00553272279024887872noreply@blogger.com0tag:blogger.com,1999:blog-8523046672726455213.post-47003826956568643712019-11-11T07:27:00.003-08:002019-11-11T07:27:53.316-08:00Mailinator Updates: new API, SMS messaging, and Routing Rules! <table border="0" cellpadding="0" cellspacing="0" class="module" data-end-index="5312" data-start-index="5095" data-type="spacer" role="module" style="table-layout: fixed; width: 100%px;"><tbody>
<tr data-end-index="5323" data-start-index="5319"><td bgcolor="" data-end-index="5427" data-start-index="5332" role="module-content" style="padding: 0px 0px 30px 0px;"></td>
</tr>
</tbody></table>
<table border="0" cellpadding="0" cellspacing="0" class="wrapper" data-end-index="5614" data-start-index="5474" data-type="image" role="module" style="table-layout: fixed; width: 100%px;">
<tbody>
<tr data-end-index="5625" data-start-index="5621">
<td align="center" data-end-index="5730" data-start-index="5634" style="font-size: 6px; line-height: 10px; padding: 0px 0px 0px 0px;" valign="top">
<img alt="" border="0" class="max-width" data-end-index="6167" data-start-index="5741" src="https://marketing-image-production.s3.amazonaws.com/uploads/ee7a1469bb9f652d049eb09bdf3d15aa978a4bc813f7705dad6a5ffc5fcabc8d3365c509851d3d4a8a829465c7deb841d12c20e062e2251743230bfa24261f52.png" style="display: block; font-family: Helvetica, arial, sans-serif; font-size: 16px; height: auto !important; max-width: 100% !important; width: 100%;" width="600" />
</td>
</tr>
</tbody></table>
<br />
<table border="0" cellpadding="0" cellspacing="0" class="module" data-end-index="6352" data-start-index="6214" data-type="text" role="module" style="table-layout: fixed; width: 100%px;">
<tbody>
<tr data-end-index="6363" data-start-index="6359">
<td bgcolor="" data-end-index="6487" data-start-index="6372" height="100%" style="padding: 30px 30px 50px 30px;" valign="top">
<h1 class="p1" data-end-index="6541" data-start-index="6500">
<span style="font-size: 24px;">Mailinator Updates!</span></h1>
<h4 class="p1" data-end-index="6664" data-start-index="6621" style="text-align: center;">
<span data-end-index="6694" data-start-index="6664" style="font-size: 18px;">SMS numbers now available</span></h4>
<div class="p1" data-end-index="6749" data-start-index="6733">
Need to
add testing SMS workflow to your QA process? Mailinator offers private
SMS inboxes for all your testing needs. SMS messages are retrievable via
Web, API, and Rules just like all incoming email.</div>
<div class="p1" data-end-index="6974" data-start-index="6958">
</div>
<h4 class="p1" data-end-index="7031" data-start-index="6988" style="text-align: center;">
<strong data-end-index="7039" data-start-index="7031"><span data-end-index="7069" data-start-index="7039" style="font-size: 18px;">Fully Active Rule System</span></strong></h4>
<div class="p1" data-end-index="7132" data-start-index="7116">
Setup
Rules to automatically react to emails arriving at your Private Domain.
Emails can be forwarded via SMTP or WebHook, have links within an email
automatically "clicked", or setup a "email blackhole" for high-speed
testing.</div>
<div class="p1" data-end-index="7383" data-start-index="7367">
</div>
<div class="p1" data-end-index="7413" data-start-index="7397">
The Mailinator API let's you pull emails, the Rule system tells Mailinator to push them back to you!</div>
<h4 class="p1" data-end-index="7564" data-start-index="7521" style="text-align: center;">
<span data-end-index="7594" data-start-index="7564" style="font-size: 18px;">New API</span></h4>
<div class="p1" data-end-index="7631" data-start-index="7615">
We've redesigned our API. Apart a more logical REST flow, the new API has some new capabilities:</div>
<ul data-end-index="7739" data-start-index="7735">
<li class="p1" data-end-index="7756" data-start-index="7741">List and retrieve attachments of any MIME type</li>
<li class="p1" data-end-index="7824" data-start-index="7809">You can
now Inject emails into the system using HTTP Post. This is very handy
for testing message flows when you want instant delivery.</li>
</ul>
<div class="p1" data-end-index="7988" data-start-index="7972">
New format:<span class="c-message__body" data-end-index="8063" data-qa="message-text" data-start-index="7999" dir="auto"> </span></div>
<div class="p1" data-end-index="8123" data-start-index="8079" style="text-align: center;">
<span data-end-index="8153" data-start-index="8123" style="font-size: 10px;"><span data-end-index="8210" data-start-index="8153" style="font-family: courier new,courier,monospace;"><span class="c-message__body" data-end-index="8274" data-qa="message-text" data-start-index="8210" dir="auto">GET /v2/domains/<strong data-end-index="8298" data-start-index="8290">:domain</strong>/inboxes/<strong data-end-index="8331" data-start-index="8323">:inbox</strong>/messages/<strong data-end-index="8364" data-start-index="8356">:message_id</strong></span></span></span></div>
<div data-end-index="8418" data-start-index="8413">
<div class="p1" data-end-index="8435" data-start-index="8419">
<div data-end-index="8441" data-start-index="8436">
</div>
<div data-end-index="8460" data-start-index="8455">
(<em data-end-index="8465" data-start-index="8461">Note: The old API will continue to remain active. You do not need to migrate to the new API at this time</em>). </div>
<div data-end-index="8460" data-start-index="8455">
<br /></div>
<div data-end-index="8460" data-start-index="8455">
View All the Mailinator System Documentation here:</div>
<div data-end-index="8460" data-start-index="8455">
<br /></div>
<div data-end-index="8460" data-start-index="8455">
https://manybrain.github.io/m8rdocs/#mailinator</div>
</div>
</div>
</td></tr>
</tbody></table>
Sam Liptonhttp://www.blogger.com/profile/00553272279024887872noreply@blogger.com0tag:blogger.com,1999:blog-8523046672726455213.post-75702228985314454212018-07-15T10:39:00.000-07:002018-07-15T10:42:49.635-07:00Mailinator.com : Anatomy of a Spammy Campaign<br />
<br />
Mailinator is a popular disposable email service. It's also become a great tool for QA Teams to test email receipt, acknowledgment, authentication loops, and formatting. But in the beginning (um, wow - 15 years ago? really?) it was a tool to help people avoid spam.<br />
<br />
Mailinator gets many millions of emails per day. At peak, it can be thousands of emails per second. Over time it has become ever more efficient at dealing with the deluge; processing, compressing, storing, and analyzing them. Once someone uses a Mailinator address, they tend to abandon it and use a different one next time. But the Internet never forgets. It seems like every Mailinator address ever used has found its way onto a marketing list somewhere.<br />
<br />
Here's a picture of the last 6 days of email entering the system:<br />
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhCSKIl7sV6B5udlqMl3LYBPHE-l0Hlb2eNTZMlmN-_Wn0cY7zUhbfpHaWMqFTvEQVkHdGarXLDAfP4kR3xDQvoTxCch9NW5M9TN9hztRCtzWP3K-1E4AGazfeCjaqFlUpjhd3qlyd4vibw/s1600/spam1.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="274" data-original-width="1024" height="106" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhCSKIl7sV6B5udlqMl3LYBPHE-l0Hlb2eNTZMlmN-_Wn0cY7zUhbfpHaWMqFTvEQVkHdGarXLDAfP4kR3xDQvoTxCch9NW5M9TN9hztRCtzWP3K-1E4AGazfeCjaqFlUpjhd3qlyd4vibw/s400/spam1.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Six days of frequent Subjects coming to Mailinator</td></tr>
</tbody></table>
<br />
This graph (click on the image for a bigger version) shows a sub-set of email entering the Mailinator system. Each vertical bar represents a 10 minute interval. For each 10 minute interval, we count emails that have the same subject line that arrived 10 or more times. In each bar, a colored segment represents emails with a single subject line. Blasts of email that come as one-offs aren't on this graph, so this isn't so much a representation of Mailinator's total volume but more a segment of our flow that feels spammy.<br />
<br />
Also, we're only selecting from email sent to the Public Mailinator system. Our subscribers get a separate, private (persistent) email system where they can send their test emails. None of those emails are represented here - and they wouldn't be anyway since they're not spam.<br />
<br />
Look at that spike of 190k emails on July 4th at 12:40pm in the center of the graph. You also might notice that most of that line is one color (green). That shows the many tens of thousands of emails with the same subject line that Mailinator received in that 10 minute interval.<br />
<br />
Consider also the corresponding graph:<br />
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEixIBR3F26C8x1h5p2NEygjCHovtbw2c5aGTmR3UC36aiXYExb7P9d-wFUjK_JltPbR3A3KKHPAN5tm7jq8dC1ktreDMO2SwiH35AiG2QIrVpnlo6AMoyy-1Jvf-oN2xdkDUWNNTczvLKFr/s1600/spam2.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="271" data-original-width="1024" height="105" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEixIBR3F26C8x1h5p2NEygjCHovtbw2c5aGTmR3UC36aiXYExb7P9d-wFUjK_JltPbR3A3KKHPAN5tm7jq8dC1ktreDMO2SwiH35AiG2QIrVpnlo6AMoyy-1Jvf-oN2xdkDUWNNTczvLKFr/s400/spam2.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Six days of sending IP addresses</td></tr>
</tbody></table>
<br />
This graph shows the IP addresses that sent emails in the same time period. Notice that the spike is still there, but the graphs don't match up. Similarly to the first graph, only IP addresses that sent 10 emails within 10 minutes are on the graph. Except for a few spots, you don't really see the same incidence of colored bands. The IP addresses are far more spread out. The implication here is that many, many IP addresses are responsible for sending the same email to Mailinator. It's not uncommon to see thousands of IP addresses involved in one spam blast.<br />
<br />
Even so, you can still see that spike at 7/4/2018 12:40pm. That was a company sending a 'marketing campaign' from one IP address. We won't mention who sent those (what they're doing isn't particularly interesting, and is far from unique), but it's likely that IP address was blacklisted in Gmail and other more discriminating email systems. Clearly, Mailinator isn't particularly discriminating.<br />
<br />
Here's a closer look at the top graph:<br />
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjsFHlLQrzDj7v9PGndiCVni5k3qRwSFcGmXM_S_NnlUW4NhQIlhZrHg1rsuAIlWB0C4JVcaKa-s_1R5QAIr85_CiQ81ozLkTWcATr3Ew61Tp6miDBSHEjQKEXHlJZUy4bhU-uKP5jiKXQn/s1600/spam3.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="636" data-original-width="1458" height="174" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjsFHlLQrzDj7v9PGndiCVni5k3qRwSFcGmXM_S_NnlUW4NhQIlhZrHg1rsuAIlWB0C4JVcaKa-s_1R5QAIr85_CiQ81ozLkTWcATr3Ew61Tp6miDBSHEjQKEXHlJZUy4bhU-uKP5jiKXQn/s400/spam3.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Spam Campaign at 5am</td></tr>
</tbody></table>
<br />
Here we've zoomed in on the "Top Subject Lines" graph for 7/5/2018 3:10am to 7/5/2018 8:47pm. The first thing we notice is that spammers are prompt! That campaign starts exactly at 5am.<br />
<br />
Look at the overlay. From 6:40 to 6:50 we got just over 60,000 emails that fit our spammy profile. There are three spam campaigns listed. The first one is for Michael Kors Bags. Classic Style! 90% off! What a steal! Well, I'm sure they're very nice and convey an exclusive feeling. They sent our system 15,400 exclusive offers in that 10 minute interval.<br />
<br />
Next in line looks to be (just guessing) a phishing campaign for people who might happen to have a Netgear router.<br />
<br />
The third one is a combination of several subject lines all coming in around 4.4k, 4.5k, and 4.6k (if we mouse-over other intervals the Netgear campaign is all over the place, but this one is consistently 4.5k).<br />
<br />
Notice we're highlighted on an email with the subject line "Welcome to our company:" Let's consider a graph that shows all the IP addresses that sent us an email with that subject line in that time interval - and we'll collect them all, not only the ones that sent 10 or more:<br />
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjdlKIb8BUTUC5V_JemQRJqnrUIOcAlHTo-jhx2udUfslO1EXMxLE9amWrX5joDv5ZB7tTlIcyoIlhG0x4blgmKqsIud5wKwFklJMhJKSr4bcL2V2eFFxDy1JTvPVZphrmMhwnX1fYFN8B-/s1600/spam4.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="546" data-original-width="1316" height="165" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjdlKIb8BUTUC5V_JemQRJqnrUIOcAlHTo-jhx2udUfslO1EXMxLE9amWrX5joDv5ZB7tTlIcyoIlhG0x4blgmKqsIud5wKwFklJMhJKSr4bcL2V2eFFxDy1JTvPVZphrmMhwnX1fYFN8B-/s400/spam4.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">One Email - thousands of Sending IPs</td></tr>
</tbody></table>
<br />
<br />
So many different colors! Which means that it's a lot of different IP addresses.<br />
<br />
Check out the mouseover for that 10 minute interval. For all those emails, the most we got from a single IP address was 3. That's a pretty well distributed spam network! Good job guys.<br />
<br />
That email and it's siblings ("New offer", "Good day!", and "Hello!") are all part of the same campaign sent from many thousands of computers. From what we can see, it's being going on for more than a week.<br />
<br />
For a while now, Mailinator has used this type of information to better store the emails that people are actually requesting. Remember, Mailinator doesn't have user accounts associated with inboxes.<br />
<br />
The public inboxes here don't belong to anyone. The very nature of Mailinator is that inboxes are completely public. A fair characterization of our privacy policy here is: "At Mailinator, there is none!", and this is a good time to remind people that they should never send private information to Mailinator's public system (our FAQ used to say that if you sent super-sensitive private stuff here then you were a stupidhead - but now we're more polite).<br />
<br />
[But still, please, don't do it]<br />
<br />
We think we can help provide pretty good anonymity in some cases. The public system doesn't really want to know who you are.<br />
<br />
Mailinator populates inboxes when email arrives. One way to think about our system is that all inboxes (trillions of them *) already exist, it's just that most aren't currently being used. Many users have found us to be a very handy tool for receiving a one-off email. More and more people are using us for email testing and that's very encouraging.<br />
<br />
The sort of analysis we talk about here has helped us make some some useful and interesting architectural choices, and it has informed the way we deal with the enormous volume of email that Mailinator has evolved to receive.<br />
<br />
It's also got us thinking - what sort of information lurks in data that flows our way every day? We have tools and a back-end infrastructure that allows a very fine-grained look at a really large volume of spam. Now that we have the ability, we're really excited to dig a little deeper into this pile of potted meat product.<br />
<br />
Thanks for using Mailinator and keep sending it spam. It makes our graphs look pretty cool.<br />
<br />
<br />
<br />
<br />
<span style="font-size: x-small;">* It's actually much more than trillions. We looked at the RFCs, and with 64 characters in the <local-part> of an address, and the weird mix of rules for allowable characters, and even weirder rules for when and where some characters are and aren't allowed, it's a bit convoluted. But with a crayon and the back of an old envelope, we figure that a decent approximation is something like 2.4 x 10^111 inboxes for a domain.</span><br />
<span style="font-size: x-small;"><span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;">If your crayon is sharper than ours, we'd love to hear your estimate.</span></span><br />
<span style="font-size: x-small;"><span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;">Also, Mailinator can handle 212 characters in the <local-part> - which makes the number considerably larger.</span></span><br />
<span style="font-size: x-small;"><span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;">You might need two crayons.</span></span><br />
<br />
<br />
<br />Jackhttp://www.blogger.com/profile/10254753015357068852noreply@blogger.com0tag:blogger.com,1999:blog-8523046672726455213.post-85624840648742263112017-07-28T05:28:00.001-07:002017-07-28T05:28:53.029-07:00Introducing the Mailinator Real-Time Inbox<div style="-webkit-text-stroke-color: rgb(34, 34, 34); -webkit-text-stroke-width: initial; color: #222222; font-family: Arial; line-height: normal;">
<span style="-webkit-font-kerning: none;"><a href="https://mailinator.com/v2/inbox.jsp">Mailinator</a> is changing the way email works.</span></div>
<div style="-webkit-text-stroke-color: rgb(34, 34, 34); -webkit-text-stroke-width: initial; color: #222222; font-family: Arial; line-height: normal; min-height: 14px;">
<span style="font-kerning: none;"></span><br /></div>
<div style="-webkit-text-stroke-color: rgb(34, 34, 34); -webkit-text-stroke-width: initial; color: #222222; font-family: Arial; line-height: normal;">
<span style="font-kerning: none;">For most users, an inbox is an email’s final destination on a journey through the internet. It’s an endpoint, your endpoint, that holds your emails as they come to you. Checking your email, or refreshing your browser, is when your inbox asks if there’s anything new for you.</span></div>
<div style="-webkit-text-stroke-color: rgb(34, 34, 34); -webkit-text-stroke-width: initial; color: #222222; font-family: Arial; line-height: normal; min-height: 14px;">
<span style="font-kerning: none;"></span><br /></div>
<div style="-webkit-text-stroke-color: rgb(34, 34, 34); -webkit-text-stroke-width: initial; color: #222222; font-family: Arial; line-height: normal;">
<span style="font-kerning: none;">But Mailinator doesn’t see it that way because inboxes @mailinator.com don’t really belong to anyone. The public <a href="https://mailinator.com/v2/inbox.jsp">Mailinator</a> system gets many (oh so many) millions of emails per day - and you can read any of them. So can anyone else. With <a href="https://mailinator.com/v2/inbox.jsp">Mailinator</a>, you don’t even have to create an inbox. You can just start sending email to it.</span></div>
<div style="-webkit-text-stroke-color: rgb(34, 34, 34); -webkit-text-stroke-width: initial; color: #222222; font-family: Arial; line-height: normal; min-height: 14px;">
<span style="font-kerning: none;"></span><br /></div>
<div style="-webkit-text-stroke-color: rgb(34, 34, 34); -webkit-text-stroke-width: initial; color: #222222; font-family: Arial; line-height: normal;">
<span style="font-kerning: none;">One way to think about it is that all inboxes already exist. Another way to think about it is that Mailinator creates inboxes on-the-fly as email arrives.</span></div>
<div style="-webkit-text-stroke-color: rgb(34, 34, 34); -webkit-text-stroke-width: initial; color: #222222; font-family: Arial; line-height: normal; min-height: 14px;">
<span style="font-kerning: none;"></span><br /></div>
<div style="-webkit-text-stroke-color: rgb(34, 34, 34); -webkit-text-stroke-width: initial; color: #222222; font-family: Arial; line-height: normal;">
<br /></div>
<div style="-webkit-text-stroke-color: rgb(34, 34, 34); -webkit-text-stroke-width: initial; color: #222222; font-family: Arial; line-height: normal; min-height: 14px;">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://mailinator.com/v2/inbox.jsp" style="margin-left: 1em; margin-right: 1em;"><img alt="" border="0" data-original-height="288" data-original-width="512" src="https://mailinator.com/m8rblog.gif" title="" /></a></div>
<br />
<br /></div>
<div style="-webkit-text-stroke-color: rgb(34, 34, 34); -webkit-text-stroke-width: initial; color: #222222; font-family: Arial; line-height: normal;">
<span style="font-kerning: none;">From Mailinator’s point of view the email flow is relentless and constant. We call it the Mailinator Stream. Previously, inboxes got new mail the way all inboxes do - periodically they asked the server if there was any new email to display.</span></div>
<div style="-webkit-text-stroke-color: rgb(34, 34, 34); -webkit-text-stroke-width: initial; color: #222222; font-family: Arial; line-height: normal; min-height: 14px;">
<span style="font-kerning: none;"></span><br /></div>
<div style="-webkit-text-stroke-color: rgb(34, 34, 34); -webkit-text-stroke-width: initial; color: #222222; font-family: Arial; line-height: normal;">
<span style="font-kerning: none;">But now we’ve hooked you up to the stream. Directly. To every inbox. In real time.</span></div>
<div style="-webkit-text-stroke-color: rgb(34, 34, 34); -webkit-text-stroke-width: initial; color: #222222; font-family: Arial; line-height: normal; min-height: 14px;">
<span style="font-kerning: none;"></span><br /></div>
<div style="-webkit-text-stroke-color: rgb(34, 34, 34); -webkit-text-stroke-width: initial; color: #222222; font-family: Arial; line-height: normal;">
<span style="font-kerning: none;">Welcome to the Mailinator Real-time Inbox. </span><br />
<span style="font-kerning: none;"><br /></span>
The moment someone sends an email to Mailinator, it’ll show up on your screen (just as fast as the internet allows). There’s no delay. You’ll never again have to hit ‘refresh’ on a Mailinator inbox. (<i>Go ahead and give it at try - pick an inbox @mailinator.com, send an email to it, and watch it arrive an instant later - be sure to turn off the "undo" feature of Gmail or it'll delay 30 seconds before sending</i>)</div>
<div style="-webkit-text-stroke-color: rgb(34, 34, 34); -webkit-text-stroke-width: initial; color: #222222; font-family: Arial; line-height: normal; min-height: 14px;">
<span style="font-kerning: none;"></span><br /></div>
<div style="-webkit-text-stroke-color: rgb(34, 34, 34); -webkit-text-stroke-width: initial; color: #222222; font-family: Arial; line-height: normal;">
<span style="font-kerning: none;">Now, with Mailinator’s inbox page tapped into the stream, an inbox is a filter. Everything that matches your filter will show up in real time.</span></div>
<div style="-webkit-text-stroke-color: rgb(34, 34, 34); -webkit-text-stroke-width: initial; color: #222222; font-family: Arial; line-height: normal; min-height: 14px;">
<span style="font-kerning: none;"></span><br /></div>
<div style="-webkit-text-stroke-color: rgb(34, 34, 34); -webkit-text-stroke-width: initial; color: #222222; font-family: Arial; line-height: normal;">
<span style="font-kerning: none;">For most users, that won’t make much of a difference. You’ll get your anonymous, no-sign-up emails like always, except now they'll arrive instantly. An inbox will still act like an inbox.</span></div>
<div style="-webkit-text-stroke-color: rgb(34, 34, 34); -webkit-text-stroke-width: initial; color: #222222; font-family: Arial; line-height: normal; min-height: 14px;">
<span style="font-kerning: none;"></span><br /></div>
<div style="-webkit-text-stroke-color: rgb(34, 34, 34); -webkit-text-stroke-width: initial; color: #222222; font-family: Arial; line-height: normal;">
<span style="font-kerning: none;">But paid subscribers with private domains can query the stream at a deeper level. (For those who didn't know, you can upgrade your Mailinator account to get your own private, not-auto-deleting version of Mailinator pointing at at your own domain that your whole team can access simultaneously).</span><br />
<span style="font-kerning: none;"><br /></span>
<span style="font-kerning: none;">Subscribers can create more complex filters and persistent queries that capture more than just a single <inbox> from the stream. You can tap into multiple inboxes at the same time. You can also use a trailing asterisk as a wildcard. Asking your inbox to capture <b>testing*</b> will fill your inbox with all the emails sent to testing1, and testing99, and testing-xyz, and anything else which matches.</span></div>
<div style="-webkit-text-stroke-color: rgb(34, 34, 34); -webkit-text-stroke-width: initial; color: #222222; font-family: Arial; line-height: normal; min-height: 14px;">
<br />
<span style="font-kerning: none;"></span></div>
<div style="-webkit-text-stroke-color: rgb(34, 34, 34); -webkit-text-stroke-width: initial; color: #222222; font-family: Arial; line-height: normal;">
<span style="font-kerning: none;">Mailinator has always been about providing public email addresses for anyone to use (no signup required), and now it does it in real-time with some great new features. Interested in your own private Mailinator where every inbox you can think up is yours (instantly and wildcard searchable)? Request a <a href="https://mailinator.com/pricingmatrix.jsp">Trial</a> today.</span></div>
<div style="-webkit-text-stroke-color: rgb(0, 0, 0); -webkit-text-stroke-width: initial; font-family: Arial; line-height: normal; min-height: 14px;">
<span style="font-kerning: none;"></span><br /></div>
<br />
<div style="-webkit-text-stroke-color: rgb(0, 0, 0); -webkit-text-stroke-width: initial; font-family: Arial; line-height: normal;">
<span style="font-kerning: none;">Thanks for using <a href="https://mailinator.com/v2/inbox.jsp">Mailinator!</a></span></div>
<div>
<span style="font-kerning: none;"><br /></span></div>
Jackhttp://www.blogger.com/profile/10254753015357068852noreply@blogger.com0tag:blogger.com,1999:blog-8523046672726455213.post-13430494878865373402017-05-04T18:55:00.000-07:002017-05-05T06:01:20.092-07:00Mailinator and the Recent Google Docs Phishing Attack<div class="gmail_default" style="color: #222222; font-family: arial, sans-serif; font-size: 13px;">
Yesterday (Wednesday, May 3rd), someone launched a clever phishing attack against Google users.
<br />
<br />
They wrote a little application and falsely named it "Google Docs." Given the chance, it compromises your Gmail account and reads your contacts list. Then, using Google’s own email servers, it sends itself to all your contacts.
<br />
<br />
This kind of attack lives or dies by how successful it is at getting people to believe it and click through. If no one clicks, or if no one grants it the permissions it asks for, then it doesn't spread and nothing happens. But every time someone clicks through and grants the app the access it wants, the attack multiplies by the number of times it propagates.
<br />
<br />
The attack was very successful at getting people to click through and grant it permissions, and it spread very quickly.
<br />
<br />
But it did so in a quirky way, one that made Mailinator one of its victims. It emailed itself from one victim's account to the next set of targets by BCC (blind carbon-copy). If you received the email, it came to you that way. Your email address wasn't in the TO field, it was in the BCC field.
<br />
<br />
That's a little bit clever. Since the BCC: is blind, the victim doesn't see themselves included in a long list of recipients (many of whom they don't recognize), which is often a big clue that the email is nefarious (or some dumb joke that a clueless relative has forwarded to everyone they know). Any one attack email might have been BCC'd to dozens of new victims.
<br />
<br />
<blockquote class="twitter-tweet" data-lang="en">
<div dir="ltr" lang="en">
If you get a Google Docs invitation email that looks like this picture, do NOT click Open. There is a phishing scheme/virus circulating. <a href="https://t.co/oHudUpSL7j">pic.twitter.com/oHudUpSL7j</a></div>
— Massac Technology (@massactech) <a href="https://twitter.com/massactech/status/859857846749581317">May 3, 2017</a></blockquote>
<script async="" charset="utf-8" src="//platform.twitter.com/widgets.js"></script>
<br />
When you send an email, you can't just BCC people. To make it legit, there has to be something in the TO field. Since all the victim's contacts were in the BCC field, the address the emails are TO isn't part of the attack. The attacker just needs a dumping ground. Something that looks just-so to the target; not a name that stands out for being unfamiliar, but also a real address that works and won’t bounce. That will improve deliverability and believability. So who did the attacker choose to send these emails to?
<br />
<br />
hhhhhhhhhhhhhhhh@mailinator.com
<br />
<br />
What is the significance of that inbox? As far as we can tell, there isn't one. Since all addresses already exist at Mailinator, using your finger to hold down a key for a bit generates a completely innocuous - but usable - inbox name. The significance of using mailinator.com is clear: our well-deserved reputation for successfully handling high volumes of incoming email.
<br />
<br />
In effect, they were relying on Mailinator’s proven ability to receive lots of email.
<br />
<br />
From the attacker's perspective, it doesn't matter much what is in the TO field. They just need to make sure that the emails get out the door. Mailinator is very good at receiving emails, the attack spread very quickly, and very soon the hhhhhhhhhhhhhhhh inbox at Mailinator was getting thousands of emails.
<br />
<br />
We noticed the activity early on, and shut down the inbound stream of emails to the hhhhhhhhhhhhhhhh inbox. Unfortunately, this did nothing to stop the attack. That's because nothing about the attack was happening via Mailinator. Mailinator was simply another recipient of the email. The attack could not propagate itself via Mailinator but it sure could send us email when other people propagated it. By the time we shut down the inbound stream, hundreds of thousands of emails to hhhhhhhhhhhhhhhh had shoved their way through our system. Any one inbox at Mailinator only shows 50 emails, and that inbox was (at peak) receiving a few thousand emails per second. Because the volume was so large, it wasn’t possible to read any of these emails at Mailinator. Before anyone could have clicked on one to read it, Mailinator had expired it to make room for thousands more that were pouring in.
<br />
<br />
The emails to hhhhhhhhhhhhhhhh were of no consequence to the attack. Even the name was irrelevant.
<br />
<br />
Strangely, we've seen a few articles mentioning that that inbox was the sender. Most even showed a screenshot on the same page contradicting that idea with <b>the Mailinator address clearly in the TO field</b> and one of the victim's contacts as the sender.
<br />
<br />
Just to be clear: the Gmail phishing attack sent email to a victim’s contacts using Google’s email servers and all emails were FROM another Google user. Each time the attack propagated, it also emailed TO Mailinator as a receiver, nothing more. None of those emails came from Mailinator, they came from Gmail.
<br />
<br />
It's true that all of the attack emails were TO: hhhhhhhhhhhhhhhh@mailinator.com, but it seems pretty clear that if you're looking for where the emails came from, you'd want to look at the FROM: field.
<br />
<br />
Additionally, the emails can't be from Mailinator because Mailinator can't send email. It’s a receive-only service. The Mailinator system was not part of the attack - it was just a recipient like all the other victims. The only difference between us and the other recipients was that we received hundreds of thousands of these emails in a very short time.
<br />
<br />
If you visit that inbox now (here's a link to <a href="https://www.mailinator.com/inbox2.jsp?public_to=hhhhhhhhhhhhhhhh">hhhhhhhhhhhhhhhh</a> inbox - don't worry, it's safe to click, it will take you right to Mailinator), you’ll notice no such phishing emails. We turned the inbound stream of attack emails off very soon after the phishing scam started.
<br />
<br />
Service to our legitimate users was uninterrupted.
<br />
<br />
Mailinator remains, as always, the best place to get a free, disposable email. We can’t prevent people from sending email to us (receiving email is the whole point of the service!) and we still love our regular users: thousands of QA Teams that send us millions of test emails, and you - whenever you want to protect your real email address from spam.
</div>
Jackhttp://www.blogger.com/profile/10254753015357068852noreply@blogger.com0tag:blogger.com,1999:blog-8523046672726455213.post-46938121752340890652015-08-27T12:01:00.000-07:002015-10-03T10:21:21.944-07:00Mailinator's API : Closing the Loop for QAMailinator's API provides programmatic access to all email within the Mailinator system. If your QA department has a need for delivery testing, this is an unprecedented way to have immediate access to thousands of email inboxes.<br />
<br />
<a href="http://mailinator.com/apidocs.jsp">Mailinator API docs</a>
<br />
<br />
Since we launched the API a year ago, we've been constantly surprised about the uses people have found for the API. Testing signup systems (with a confirmation email) seems to be a popular choice, but other customers test marketing campaigns and test automated alert systems sending us thousands of emails each day.<br />
<br />
The API works on both the public Mailinator system and the private domain system. For the public system you have instant access to any of Mailinator's millions of emails, obviously including the emails you send there.<br />
<br />
If you're looking for more privacy, you can setup a private domain and only you (and your API token) can see emails that arrive for that domain.<br />
<br />
<br />
If you're interested in accessing emails via API - <a href="http://mailinator.com/">Sign up at Mailinator</a> today and try out the API!paulhttp://www.blogger.com/profile/11412172362500455307noreply@blogger.com0tag:blogger.com,1999:blog-8523046672726455213.post-77381177220703689562014-09-22T14:08:00.001-07:002014-09-22T14:13:33.483-07:00Playing Wasteland 2? Find the Mailinator gun !<a href='http://www.mailinator.com'>Mailinator</a> has always been about vanquishing spam - and now in Wasteland 2, you can vanquish some Wasteland Wolves using "The Mailinator" gun !
<br>
<br>
<a href='http://www.mailinator.com'>Mailinator</a> is proud to be a backer of the newly released <a href='http://wasteland.inxile-entertainment.com/'>Wasteland 2</a>. If you're playing the game (and you should be!), be sure to lookout for the gun called "The Mailinator".
<br>
<br>
<b>(hint: you might consider having one of your rangers specialize in Heavy Weapons !)</b>
<a href="http://wasteland.inxile-entertainment.com/"><img border="0" width='300' height='200' src="http://i.imgur.com/VtLU8f5.jpg" border='0'/></a>paulhttp://www.blogger.com/profile/11412172362500455307noreply@blogger.com0tag:blogger.com,1999:blog-8523046672726455213.post-18552939628828736702013-11-09T16:12:00.000-08:002015-09-27T10:28:11.302-07:00Increasing Brand-Value of Mailinator.com with a Web Comic<div class="MsoNormal" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<span style="font-family: Arial; font-size: 10pt;"></span></div>
<div class="MsoNormal" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<span style="font-family: Arial; font-size: 10pt;">[This is a guest post by Marketing/Growth-Hacking consultant Martha Andrews - contact her directly at martha@manybrain.com. Follow Martha on Twitter <complete id="goog_1242636096"><a href='https://twitter.com/another_martha'>@another_martha</a></complete>] </span></div>
<div class="MsoNormal" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<br /></div>
<div class="MsoNormal" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<span style="font-family: Arial; font-size: 10pt;">We have been doing some investigation in increasing brand awareness for Mailinator.com. </span></div>
<div class="MsoNormal" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<br /></div>
<div class="MsoNormal" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<span style="font-family: Arial; font-size: 10pt;">Mailinator has always been a useful site providing free, disposable email - but analysis of the usage patterns of Mailinator indicated that users tended to use it on a monthly or bi-weekly basis rather than weekly or daily. In some ways this makes sense – the site provides a utility, but the use case for the general public isn't necessarily needed on a daily basis. When it is needed however - our goal is to make sure you think of Mailinator.com!</span></div>
<div class="MsoNormal" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<br /></div>
<div class="MsoNormal" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<span style="font-family: Arial; font-size: 10pt;">In order to raise all usage levels of the site, we focused on public awareness of the site, expecting that an increased DAU (and MAU) would/should be an expected side effect. </span><span style="font-family: Arial; font-size: 10pt;">I’ve frequented many trainings in the last year, and I cannot tell you how many times I’ve heard, “Create interesting and useful content and people will flock to your site…</span><i style="font-family: Arial; font-size: 10pt;">Its all about providing content</i><span style="font-family: Arial; font-size: 10pt;">.” </span></div>
<div class="MsoNormal" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<br /></div>
<div class="MsoNormal" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<span style="font-family: Arial; font-size: 10pt;">But what content could we give to our users? We already give away the service – we have a useful tool, and want more folks to know about it and use it.</span></div>
<div class="MsoNormal" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<br /></div>
<div class="MsoNormal" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<span style="font-family: Arial; font-size: 10pt;">Before creating content, something to consider is who is in our user base? Web analytics told us the primary user base is male, ages 18-40 – think of the average Reddit user. </span><span style="font-family: Arial; font-size: 10pt;">So how do we engage 18-40 year old men,</span> <i style="font-family: Arial; font-size: 10pt;">and potentially other demographic groups</i><span style="font-family: Arial; font-size: 10pt;">? How do we get them to think of Mailinator.com more often? What could we provide that would be of use of them? Getting folks to think of the site more often, and giving them content and reasons to visit the site would likely expand their current usage pattern.</span></div>
<div class="MsoNormal" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<br /></div>
<div class="MsoNormal" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<span style="font-family: Arial; font-size: 10pt;">We tried targeted ads through traditional web and social channels. Specifically - we experimented with Google Adwords, Reddit.com, Facebook, and Twitter. The ads provided momentary increases in traffic to varying degrees but nothing long lasting. We can get more users to Mailinator any day of the week by spending some money, but that cannot be sustained given that mailinator.com doesn not generate direct user revenue.</span></div>
<div class="MsoNormal" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<br /></div>
<div class="MsoNormal" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<span style="font-family: Arial; font-size: 10pt;">During a recent site redesign we incorporated a new mascot of sorts – an amiable looking caped crusader we dubbed “Mailinator Guy” - the Anti-Spam Superhero. </span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgmc-M-pBWoNRWtI1jtuOO78GUyqGMPHf3Rc3s5TYMd95DOUCEu4olKkZiVIQAish40etUf9ZsIT2DIRWWlKV29LrT8dOWpKZIp-gY-hK_pvZCodK1hM9Zq1QA5pJKxq5FBS9_TWs33SNtK/s1600/comic2_clipped_rev_1.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="201" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgmc-M-pBWoNRWtI1jtuOO78GUyqGMPHf3Rc3s5TYMd95DOUCEu4olKkZiVIQAish40etUf9ZsIT2DIRWWlKV29LrT8dOWpKZIp-gY-hK_pvZCodK1hM9Zq1QA5pJKxq5FBS9_TWs33SNtK/s320/comic2_clipped_rev_1.png" width="320" /></a></div>
<span style="font-family: Arial; font-size: 10pt;"> </span></div>
<div class="MsoNormal" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<span style="font-family: Arial; font-size: 10pt;">The more we lived with Mailinator Guy, the more often we started asking: what if we launched a web comic based on Mailinator Guy? Would a comic be appropriate content? </span><span style="font-family: Arial; font-size: 10pt;">If you've ever read Mailinator's FAQ you know the tone of the site is already quite tongue-in-cheek - we at least amuse ourselves. Frankly the thought of bringing Mailinator Guy to life was simply too fun an experiment not to try.</span></div>
<div class="MsoNormal" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<span style="font-family: Arial; font-size: 10pt;"><br /></span><span style="font-family: Arial; font-size: 10pt;">ComicRocket has over 35,000 web comics indexed on their site alone, so competition for user attention is stiff. However we found Facebook ads for the comic to be highly successful in engaging a new audience. We think it works well because they are demographically targetable, colorful, eye catching and fun. A thumbnail or portion of the strip is always displayed in the posts and ads.</span></div>
<div class="MsoNormal" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<br /></div>
<div class="MsoNormal" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<span style="font-family: Arial; font-size: 10pt;">We also engage our existing audience through the Mailinator Facebook page (with >30000 likes). There is an announcement on the page when a new comic is published. Additionally, the colorful thumbnail version of the strip is shown on the landing page and on mailbox pages </span><span style="font-family: Arial; font-size: 10pt;">at </span><u style="font-family: Arial; font-size: 10pt;"><span style="color: #1155cc;"><a href="http://mailinator.com/" style="color: #1155cc;" target="_blank">mailinator.com</a></span></u><span style="font-family: Arial; font-size: 10pt;">. We have found a</span><span style="font-family: Arial; font-size: 10pt;">s we release a comic each week, we are slowly habituating our audience to check </span><u style="font-family: Arial; font-size: 10pt;"><span style="color: #1155cc;"><a href="http://mailinatorguy.com/" style="color: #1155cc;" target="_blank">mailinatorguy.com</a></span></u><span style="font-family: Arial; font-size: 10pt;"> on a weekly basis – and if they are following us on twitter or are a Facebook fan, we have an excuse to contact them.</span></div>
<div class="MsoNormal" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<br /></div>
<div class="MsoNormal" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<span style="font-family: Arial; font-size: 10pt;">The feedback loop of the two sites is strong. Mailinator.com provides the free disposable email utility that brings new users by itself. The site's design reinforces the Mailinator Guy character and encourages our primary demographic, arguably also the primary demographic for web comics, to follow the comic. The MailinatorGuy.com site provides a weekly enticement for a visit. That visit keeps the Mailinator brand fresh in our user's mind causing the loop.</span></div>
<div class="MsoNormal" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjyWlG0NSWJK9VKYQXA7XhyphenhyphenaqNe6o9JDlc2zJYHJW05JqPA9t2hrrYBmBlgsbwGNzFdbnifflt7RKvgASPo886TsrXlMFrPKOoQM_ZiXAg4xH8tFP_kLLc2Az-XbnMtBcqOmXDo2zB4FW9A/s1600/Slide1.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="171" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjyWlG0NSWJK9VKYQXA7XhyphenhyphenaqNe6o9JDlc2zJYHJW05JqPA9t2hrrYBmBlgsbwGNzFdbnifflt7RKvgASPo886TsrXlMFrPKOoQM_ZiXAg4xH8tFP_kLLc2Az-XbnMtBcqOmXDo2zB4FW9A/s320/Slide1.jpg" width="320" /></a></div>
</div>
<div class="MsoNormal" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<br />
<span style="font-family: Arial; font-size: 10pt;">Of course some users of Mailinator will never care about the comic. And some users of the comic will never care about </span><u style="font-family: Arial; font-size: 10pt;"><span style="color: #1155cc;"><a href="http://mailinator.com/" style="color: #1155cc;" target="_blank">mailinator.com</a></span></u><span style="font-family: Arial; font-size: 10pt;">. And that's just fine - both sites have inherent stand-alone value.</span></div>
<div class="MsoNormal" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<br /></div>
<div class="MsoNormal" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<span style="font-family: Arial; font-size: 10pt;">At the time of this writing we have published six episodes of The Adventures of Mailinator Guy. Our advertising efforts, both paid and direct referrals from <u><span style="color: #1155cc;"><a href="http://mailinator.com/" style="color: #1155cc;" target="_blank">mailinator.com</a></span></u>, have driven traffic that averaged 1,500 unique weekly users the first three weeks, to over 3,000 unique weekly visitors the following three weeks. The most traffic occurs on Mondays when we have usually posted and announced a new episode. Along with an increase in users, producing the comic has become more efficient and economical, and still pretty darn fun.</span></div>
<div class="MsoNormal" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<br /></div>
<div class="MsoNormal" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<span style="font-family: Arial; font-size: 10pt;">Importantly - traffic to <u><span style="color: #1155cc;"><a href="http://mailinator.com/" style="color: #1155cc;" target="_blank">mailinator.com</a></span></u> has increased 15% in the 6 weeks since the launch of the comic and continues to grow. </span></div>
<div class="MsoNormal" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<br /></div>
<div class="MsoNormal" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<div style="font-style: normal;">
<span style="font-family: Arial; font-size: 10pt;">These numbers are very encouraging but honestly the most important outcome is the increase in Mailinator brand recognition. The site's brand value will continue to grow and that's the real goal of this exercise. </span></div>
<div style="font-style: normal;">
<span style="font-family: Arial; font-size: 10pt;"><br /></span></div>
<span style="font-family: Arial; font-size: 10pt;">At the end of the day - everyone is happy. More people know about the Mailinator brand - this makes us happy - and have either avoided some spam by using our free disposable email service, have had a laugh at the expense of Mailinator Guy - and his sidekick (<i>you'll have to read the strip to find out about him</i>), or both.</span><br />
<span style="font-family: Arial; font-size: 10pt;"><br /></span></div>
<div class="MsoNormal" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
If you're up for following the adventures of Mailinator Guy yourself - here's the link to get you started:<br />
<br />
<a href="http://www.mailinatorguy.com/mailinatorguy/2-spam-attack-2/">The Adventures of Mailinator Guy!</a> </div>
<div class="MsoNormal" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://www.mailinatorguy.com/mailinatorguy/2-spam-attack-2/">
<img border="0" src="http://www.mailinatorguy.com/wp-content/uploads/2013/10/comic2-212x300.png" />
</a>
</div>
<div class="MsoNormal" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<br /></div>
<div class="MsoNormal" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<br /></div>
<div class="MsoNormal" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<br /></div>
<div class="MsoNormal" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<br /></div>
<div class="MsoNormal" style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<br /></div>Anonymoushttp://www.blogger.com/profile/06931197076592662323noreply@blogger.com0tag:blogger.com,1999:blog-8523046672726455213.post-39027496922809920632012-05-02T10:43:00.002-07:002012-05-02T10:43:41.560-07:00How to get your resume "Silicon Valley Ready" - Part IPer my last <a href="http://paultyma.blogspot.com/2012/04/why-you-should-join-start-up-and-maybe.html">post</a>, I've been given the opportunity to review a nice pile of resumes. As I am prone to, this got me to obsess a tad over how the resumes were put together and more importantly, what each told me.
<br><br>
What I perceived as issues are, in retrospect, my fault, not the resume owner's. That's because, per the entire point of my last post, the start-up environment is radically different than the corporate IT department. And the latter is where many of these resumes came from (which is exactly what I wanted and asked for).
<br><br>
In many cases, I received a resume from someone that included the regular set of data - experience, education, skills, etc. But the ones that got me excited were the ones where the person included in the email links to the websites or mobile-apps they had built. As I've said - the number one selling point for you as an engineer to get a job in Silicon Valley is that <b>you love this stuff</b>. There's an age-old conundrum of new grads who say "Employers want me to have experience before they'll hire me - but how do I get experience if I can't get a job?"
<br><br>
In our business I'm happy to say - that problem does not exist. Simply because you don't need anyone to give you a job to build something. A website. A mobile app. Heck, a program that finds <a href="http://paultyma.blogspot.com/2010/11/google-interviewing-story.html">smaller sets of strings in larger ones</a>.
<br><br>
I realized that's probably the number one thing I'm looking for. You can show me, with no doubts, that as a software engineer - you can build something. Start to finish.
<br><br>
Interestingly, I like to think that I also don't put that much weight into whether a project was a commercial success. If it was, that's nice but and maybe it's because you are not only a great engineer but you have an awesome product sense - who knows (it just might mean you were lucky too). And unless it wasn't a commercial success <b>because</b> it was poorly engineered, that's not really the point. The point is that you built it. Or at least some non-trivial part of it.
<br><br>
With that in mind - this article sprang forth on what I like to see in resumes. I'll point out that this isn't very different from what I looked for when I was on a hiring committee at Google (so there are at least some current Google engineers that are partially there because of these thoughts).
<br><br>
First - I propose a new section to resumes - at least for software engineers. In addition to Experience, Education, Skills, Interest, and References (not suggesting we remove any of those) - I propose we add <b>Cool Stuff I Have Built</b>.
<br><br>
If your resume is going to go over one page (which, personally, I don't mind) - I'm hoping it's because of this section.
<br><br>
Any project you did solo or had a major hand in - whether paid or not paid, million users or just your mom, I'd love to know about. Websites, iphone apps, android apps, desktop apps, open-source projects, github accounts - you name it. Solo or as part of a team (indicate that). But it has to be, in one way or another "finished". Even if your iphone app was rejected by the app store - you can point me to a link to see it. It doesn't have to be a product either - maybe its an open-source project. The bottom line is it is something that you "finished". You executed. Your idea became a living breathing application or piece of code that in some way some how you could show to people.
<br><br>
The section might be broken up into individual projects with bullet points about each. For example:
<br><br>
<div style='font-family: courier;font-size:.9em'>
<b>Project Name:</b> Mailinator
<br>
<b>Technologies used:</b> Java, tomcat, (no database)
<br>
<b>Team Size</b>:5'11", 175lbs (haha)
<br>
<b>Implementation details/challenges:</b>Custom server architecture built as an experimental test-bed for highly-multithreaded server design. Custom SMTP server. No database as emails are stored in RAM with a custom compression scheme.
<br>
<b>Notable Metrics:</b> up to 25MM emails per day, ~20k users per day, runs on a single server (on purpose as part of a personal experiment to optimize the system).
<br>
<b>relevant links:</b> www.mailinator.com, http://mailinator.blogspot.com/2012/02/how-mailinator-compresses-email-by-90.html
</div>
<br><br>
Surely you could add other bullet points too (and suggestions welcome, leave a comment to this post). But you get the idea.
<br><br>
My previous post resonated strongly with some people - that is, they were in "IT departments" feeling like they weren't growing technically. And as you can imagine, a resume telling me you did a payroll system is great - but it's not what (most) start-ups are building. But what if you haven't built anything? And your "Cool Stuff I've Built" section is empty?
<br><br>
Well .. fix that.
<br><br>
No one is stopping you from building something. No one said you're ready for a transition out of your current job today and as with much of life it's up to you to take yourself to the next level. But nearly any person that already writes software with a penchant for learning and some ambition can spend the next few months of nights and weekends learning and building. (<i>And it's absolutely possible that your day job accomplishments belong in that cool stuff section too</i>).
<br><br>
So if by day you're a payroll guy, but by night you're in iphone ninja - you've got my attention. Not only because you have the skills that I'm looking for - but that in your spare time, you're out doing great things. And instead of going out every night drinking with your friends - at least some of those nights, you chose to stay home and learn and build cool stuff. And why would you do something like that? Simple - you love this stuff.
<br><br>
<i>(My start-up is located in Palo Alto and I am right now interviewing for the initial engineering team. We're well-funded, building cool stuff, and plan to change, ya know, the world. No matter where you are - if you're a software engineer, willing to relocate to San Francisco/Silicon Valley (and of course, love to build great things) send me your resume. paul@refresh.io or check out <a href='http://www.refresh.io/jobs'>www.refresh.io/jobs</a>)</i>paulhttp://www.blogger.com/profile/11412172362500455307noreply@blogger.com3tag:blogger.com,1999:blog-8523046672726455213.post-77417576738357369772012-02-21T07:51:00.001-08:002012-02-23T07:47:53.130-08:00How Mailinator compresses email by 90%Given the title of this article, the first thing that should pop into your mind is probably - "well, use a compression algorithm - right?".<br />
<br />
Right! Well, yes, well, not exactly. Read on.<br />
<br />
Your second thought might also have been - "Why bother? Just buy more disks." Which in the big picture is also not a bad answer. But for <a href='http://www.mailinator.com'>Mailinator</a> that doesn't work - if you have read previous Mailinator tech articles you might know that Mailinator stores all it's email in RAM.<br />
<br />
There were good reasons for that when Mailinator started. One was the use case - which was always disposable email that lasts a few hours (rather longer nowadays). Secondly, when Mailinator started, disks and datastores weren't as sophisticated/fast as they are now.<br />
<br />
Also, Mailinator is/was always a free service so keeping costs down was always important. To this day, Mailinator runs on a single server. It averages about 4-5Terabytes of bandwidth a month and the peak incoming email rate I've seen is about 3500 emails/sec (this is just a production observation, server limit is bandwidth, not CPU).
<br />
<br />
And finally - last but not least - to me, much of web and application development today is utterly devoid of any fun algorithms. I spend a non-trivial amount of time in interpreted/dynamic scripting languages that do a fantastic job of hiding (or at least lure me away from thinking about) algorithmic complexity. I've probably inadvertently written more n^3 algorithms than, um, (n^3)-for-some-large-value-of-n.<br />
<br />
Mailinator has always been my test bed for trying fun ideas, algorithms, and datastructures. In other words - I probably didn't need to do all the work I'm writing about here - but I definitely did have fun doing it (probably should have been out talking to girls, but alas).<br />
<br />
<b>Compression</b><br />
<br />
Ok - so back to 90% compression.<br />
<br />
So to start testing, I grabbed a few hundred megs of the Mailinator stream and ran it through several compressors. Mostly just stuff I had on hand 7z, bzip, gzip, etc. Venerable zip reduced the file by 63%. Not bad. Then I tried the LZMA/2 algorithm (7z) which got it down by 85% !<br />
<br />
Well. OK! Article is over! Everyone out! 85% is good enough.<br />
<br />
Actually - there were two problems with that result. One was that, LZMA, like many compression algorithms build their dictionary based on a fixed dataset. As it compresses it builds a dictionary of common sequences and improves and uses that dictionary to compress everything thereafter.<br />
<br />
That works great on static files - but Mailinator is not a static file. Its a big, honking, several gigabyte cache of ever changing email. If I compressed a million emails, and then some user wanted to read email #502,922 - I'd have to "seek" through the preceding half-million or so to build the dictionary in order to decompress it. That's probably not feasible. And, as I said, the Mailinator cache is constantly throwing out old emails and putting in new ones.<br />
<br />
In other words, an algorithm that relies on previous entries to build a dictionary can't work given that we keep purging the front of the stream never to be seen again.<br />
<br />
Hence, we cannot compress emails "together". But we can compress them individually. Sadly, this hurts our compression ratio - and by a lot. The algorithm now must start building a new dictionary with each email. And emails are small so the dictionary isn't very mature by the time we're done compressing in many cases.<br />
<br />
We can help this situation by giving the compression algorithm a pre-built dictionary. That is, scan a typical piece of data to be compressed, find common sequences and create a list of them. Then we give that dictionary to the compressor/decompressor as it takes off.<br />
<br />
Woopsie. Again, the Mailinator stream is a living and breathing entity that's always changing. One minute might be a few million viagra spams, the next minute might be all about fake rolex watches. In other words, there is no "typical piece of data" - a static dictionary built off a sample of emails will be obsolete in relatively short order.<br />
<br />
So, the first idea was to build a sliding dictionary builder. Each email is scanned for string occurrences and we keep a count of them. Then every so often (minutes or hours), the compressor switches to using the most recently constructed dictionary. Every compressed email is given a reference to its dictionary so when/if it needs to be decompressed, it knows what dictionary to give the decompressor. Many thousands of emails share the same dictionary so RAM to store dictionaries isn't particularly significant.<br />
<br />
Well, that's great and does restore LZMA back to about 60-70% but remember I mentioned I had another problem with LZMA? Speed.<br />
<br />
The C++ version of LZMA by Igor Pavlov compresses at about 1.7MB/s per CPU core on my test machine. Um. no. Firstly, Mailinator can pull down tens of MB per sec at times. Secondly, no component of our processing pipeline can be allowed to take up this much CPU (my rule, not yours). We need our CPU for other things when large volumes of mail arrive. (The java version by the way was about the same speed).<br />
<br />
Simply - LZMA is pretty awesome - but it's too slow for this purpose.<br />
<br />
So the for the moment, I fell back to using a fast but simpler compression (zlib/LZW) on individual emails - and we sink down to about 40-50% savings from compression.<br />
<br />
<b>A Bigger Idea of a "Dictionary"</b><br />
<br />
The next step for me was to think about email composition. We get lots of different types of email - but we get lots of the same types too. For example, we get lots of newsletters (people send them to Mailinator then read them via POP or RSS).<br />
<br />
The nice thing for us is that a newsletter email blast could be 10,000 emails that are, all the same. Well, ok, not exactly - no two emails are ever the "same" because headers have times, dates, message-id's, etc. within them. But if we remove the headers, you can get 10,000 emails going into 10,000 different inboxes that all have the same message "body". Are you thinking what I'm thinking?<br />
<br />
Right - store each email with it's own headers plus a pointer to ONE system-wide byte-array containing the newsletter body. What's the "compression" ratio of that? Well over 90%. And just to be a snot we can then apply compression to that byte array to eek out another few percent. We're reusing memory here so it's not exactly "compression" but we are reducing the size of the data sent to by some fantastic amount for this happy use case.<br />
<br />
This isn't a revolutionary idea (online music libraries do the same thing) but it does fit pretty nicely in the Mailinator paradigm. Sadly apart from newsletters, not many other email sets, spam or otherwise have email bodies that are identical. In fact, spammers specifically change the subject line and destination url of every email they send for tracking and spam-detection-thwarting purposes. So what you get is something like this (headers omitted):<br />
<br />
Email 1:<br />
<span style="font-family: 'Courier New', Courier, monospace;">Buy vi4gra now!</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">http://rrr4.somerandomthing.com/?3jwow33oo</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">Happy man are you will be!</span><br />
<br />
Email 2:<br />
<span style="font-family: 'Courier New', Courier, monospace;">Buy vi4gra now!</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">http://1rr220.somerandomthing.com/?ajo200kko</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">Happy man are you will be!</span><br />
<br />
So much for simply detecting identical email bodies. And this goes for less nefarious things too. Sign-up emails from websites will contain the same surrounding text with different names and validation urls inside.<br />
<br />
What we could use here is a <a href="http://en.wikipedia.org/wiki/Longest_common_substring_problem">Longest Common Substring</a> (LCS) algorithm. Basically, it would compare the two email bodies and be able to break them up as:<br />
<br />
Common string 1:<br />
<span style="font-family: 'Courier New', Courier, monospace;">Buy vi4gra now!\r\nhttp://</span><br />
<br />
Disparate strings:<br />
<span style="font-family: 'Courier New', Courier, monospace;">rrr4.somerandomthing.com/?3jwow33oo</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">1rr220.somerandomthing.com/?ajo200kko</span><br />
<br />
Common string 2:<br />
<span style="font-family: 'Courier New', Courier, monospace;">\r\nHappy man are you will be!</span><br />
<br />
Nice .. each email is stored as 3 (compressed) byte arrays where 2 of those can be shared.<br />
<br />
Unfortunately, classic LCS algorithms are expensive. Comparing two sequences is an <span style="font-family: 'Courier New', Courier, monospace;">O(nm)</span> algorithm. And we're not interested in comparing two sequences, we're interested in comparing each new sequence (er.. each new email) with the few million that preceded it. Also, the LCS algorithm is also very memory expensive in the creation of trie datastructures - again, scaling to millions of emails just doesn't fit in our parameters.<br />
<br />
Generally speaking, there are a lot of tricks I've noticed in analyzing algorithms. A few off the top of my head are: if you see an easy <span style="font-family: 'Courier New', Courier, monospace;">O(n^2)</span> algorithm, it's rather likely there's an <span style="font-family: 'Courier New', Courier, monospace;">O(nlogn)</span> one hiding in there somewhere. In contrast, if your dataset is small, you might be better off sticking to algorithms that make your CPU's cache and instruction pipeline happy instead of worrying about algorithmic running time (i.e. bubblesort > quicksort for small data). Lastly - if you can make assumptions about your data, you can often short-cut the classic algorithm with an good approximation.<br />
<br />
<b>Caching Lines</b><br />
<br />
Cool, so let's assume something about the data. For emails, as it turns out, disparate parts of emails often occur on line boundaries (as you see in lines 1 & 3 above). A few same lines, a different one, a few more same. Instead of looking for common sequences based on individual characters, we can treat individual lines as units. Then we can attempt to find multiple occurrences of those lines. It cannot be as precise as LCS proper as in our above example (we would not find the identical portion "http://" in line 2) but we're basically settling for a greedy approximation, and one that works pretty well.<br />
<br />
How do we store it though? LCS's tries would kill us. I know - let's use an LRU cache. Those darn things work for everything! We can use an LRU cache that caches full email-lines. It will inherently flush out old email lines as the spam stream evolves (nice!) and will provide quick look- ups to compares thousands of lines at once (happy!). Specifically in Java, an LRU-cache is a synchronized <span style="font-family: 'Courier New', Courier, monospace;">LinkedHashMap</span> with <span style="font-family: 'Courier New', Courier, monospace;">true</span> as the last constructor parameter and an overridden <span style="font-family: 'Courier New', Courier, monospace;">removeEldestEntry</span>.<br />
<br />
So we store a few 10's of thousands of email lines in an LRU cache and then as each new email comes in, we check to see if that line is in the cache. If it is, we reuse the one in the cache instead of creating new storage for this email. By assuming all common sequences are bounded at newlines, we remove the boundary-discovery work LCS must do. Strictly speaking, we're cheating and losing some opportunity, but it's a good enough guess for this type of data.<br />
<br />
This had a dramatic effect on our "compression" (again, it's slighty dubious to call it compression but, as you consider the big picture, our entire machinery of the LRU cache and bastardized LCS-in-spirit algorithm is creating a reuse-dictionary, it might not actually be compression - but it goes through several of the motions).<br />
<br />
<b>Caching Multi-lines</b><br />
<br />
Caching lines is great - but what about caching multi-lines? Say we have a few emails - for brevity, assume each character in the following examples are email "lines":<br />
<br />
Email 1:<br />
<span style="font-family: 'Courier New', Courier, monospace;">ABC1</span><br />
<br />
Email 2:<br />
<span style="font-family: 'Courier New', Courier, monospace;">ABC2</span><br />
<br />
Email 3:<br />
<span style="font-family: 'Courier New', Courier, monospace;">ABC3</span><br />
<br />
Email 4:<br />
<span style="font-family: 'Courier New', Courier, monospace;">ABC4</span><br />
<br />
So the first 3 lines are all the same in each email (<span style="font-family: 'Courier New', Courier, monospace;">ABC</span>), the 4th lines are numbers which are not the same. Our algorithm:<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">1) Load a <b>LINE</b> and see if it's in the cache (if no more lines, quit)</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">2) .. if it's not there, put <b>LINE</b> in the cache, and store <b>LINE</b> in the email - GOTO 1</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">3) .. If it IS there:</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">4) .... see if <b>LINE + NEXT_LINE</b> is in the cache</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">5) .... if its not there, put <b>LINE + NEXT_LINE</b> into the cache and store <b>LINE</b> (which is a cache hit) in our email - GOTO 1</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">6) .... if it IS there, <b>LINE = LINE + NEXT_LINE</b>, - GOTO 4;</span><br />
<br />
So if we run our 4 emails above through this algorithm. We get the following:<br />
<br />
Running through all of email 1 - we get:<br />
- Cache HITS stored in email: none<br />
- Cache MISSES stored in email: <span style="font-family: 'Courier New', Courier, monospace;"><b>A,B,C,1</b></span><br />
- Cache contents afterwards (lru order):<span style="font-family: 'Courier New', Courier, monospace;"> <b>1,C,B,A</b></span><br />
<br />
Running through all of email 2 - we get:<br />
- Cache HITS stored in email: <span style="font-family: 'Courier New', Courier, monospace;"><b>A,B,C</b></span><br />
- Cache MISSES stored in email: <span style="font-family: 'Courier New', Courier, monospace;"><b>2</b></span><br />
- Cache contents afterwards (lru order): <span style="font-family: 'Courier New', Courier, monospace;"><b>2,C2,C,BC,B,AB,A,1</b></span><br />
<br />
(notice how '<span style="font-family: 'Courier New', Courier, monospace;">1</span>' (which didn't cache hit) has worked itself to the end)<br />
<br />
Running through all of email 3 - we get:<br />
- Cache HITS stored in email: <span style="font-family: 'Courier New', Courier, monospace;"><b>AB,C</b></span><br />
- Cache MISSES stored in email: <span style="font-family: 'Courier New', Courier, monospace;"><b>3</b></span><br />
- Cache contents afterwards (lru order): <span style="font-family: 'Courier New', Courier, monospace;"><b>3,C3,ABC,AB,A,2,C2,BC,B,1</b></span><br />
<br />
Running through all of email 4 - we get:<br />
- Cache HITS stored in email: <span style="font-family: 'Courier New', Courier, monospace;"><b>ABC</b> <-- very cool result, note coolness</span><br />
- Cache MISSES stored in email: <span style="font-family: 'Courier New', Courier, monospace;"><b>4</b></span><br />
- Cache contents afterwards (lru order):<span style="font-family: 'Courier New', Courier, monospace;"> <b>4,ABC,AB,A,3,C3,C,2,C2,BC,B,1</b></span><br />
<br />
So what happened? The system has realized that <span style="font-family: 'Courier New', Courier, monospace;">ABC</span> is cacheable and is now pointing to that. All subsequent emails with the set-of-lines <span style="font-family: 'Courier New', Courier, monospace;">ABC</span> will reuse the same memory. Note that the disparate lines <span style="font-family: 'Courier New', Courier, monospace;">1,2,3,</span> and <span style="font-family: 'Courier New', Courier, monospace;">4</span> will always be stored separately, but the algorithm will then pick-up any common line-sets later in the email too (if there were any).<br />
<br />
This elaborate system to find equal email lines and reuse them drags out compression of the entire flowing email stream down to about 80%. What about 90%? Well.. one more trick.<br />
<br />
<b>Back to LZMA</b><br />
<br />
Remember LZMA from above that we abandoned because it was too slow to happen inline? As you'd guess, the biggest impact it had was on bigger emails. And although it's a CPU hog, we do actually have a few cores laying around. So let's give it one (but seriously, just one).<br />
<br />
We setup one core (i.e. thread) to trail behind and scan incoming email for ones that are over some size (say 20k) and re-compress those using the sliding dictionary LZMA we mentioned earlier. While 3 of our cores average 5-10% utilization by receiving, analyzing, and storing incoming email - the 4th core sits at 100% re-compressing emails where it will find benefit. If it gets too far behind, it simply leaps ahead and leaves some compression on the table.<br />
<br />
(Note that empirically, LZMA is an order of magnitude faster decompressing than compressing, otherwise that would have been a new problem as it could take too long when someone wanted to read an email)<br />
<br />
Voila. 90%. (Two notes: 1: that's a reasonable average at least... sometimes better, sometimes worse and 2: I realize I'm not exactly sure what "Voila" means, looking that up now).<br />
<br />
There are also some other important notes. Storing a byte array in Java costs something. The pointer alone (64bit) is 8bytes. Then there is the byte length field, padding, etc. In other words, I limited the system to never store email lines under 64 bytes. Small lines get concatenated together straight away.<br />
<br />
Second, there are more email-idiomatic tweaks we can do to improve the situation. Base64-encoded attachments are effectively un-cacheable, so we pass over those.<br />
<br />
Third, although from our cheeky example it may seem like we're finding optimal line sets (i.e. <span style="font-family: 'Courier New', Courier, monospace;">ABC</span>). We're not. We could end up caching <span style="font-family: 'Courier New', Courier, monospace;">ABC</span> and destroying an opportunity for a more optimal <span style="font-family: 'Courier New', Courier, monospace;">BCDXYZ</span> or something. I'm guessing this doesn't happen often but would be an interesting future consideration.<br />
<br />
<i>Edit: Wow, sincere thanks to an Anonymous commenter for making me reconsider the above algorithm. I had originally stated it was O(n^2). My first version was indeed O(n^2) (which wasn't written about) and after a few changes it became O(n) and I failed to see that. I find its very easy to find tech reviewers once an article hits Hackers News, before then though - not so much. :) My apologies for the error.</i>
<br />
<br />
So for the end-user, this whole diatribe simply means little except their emails are sticking around longer. They have no idea that when they click to read an email we may be LZW or LZMA decompressing tens of byte arrays shared by thousands of emails with a custom-sliding dictionary built by scanning emails that arrived hours ago and then catenating them together so they can be shown on their webpage all in a few milliseconds. And they likely don't care, they're probably too busy signing up for Minecraft or something.<br />
<br />
But that's ok. I know.<br />
<br />
And if you got this far, you know too.<br />
<br />
Ok.. now back to real work. What was I doing again? Oh yeah, writing some slick one-liners in Ruby. No clue on the running times - probably like <span style="font-family: 'Courier New', Courier, monospace;">O(n^4)</span> or something, but if I fiddle with it a bit more - I bet I can cut the character count of the code by half!paulhttp://www.blogger.com/profile/11412172362500455307noreply@blogger.com39tag:blogger.com,1999:blog-8523046672726455213.post-61646652967725704772009-06-09T05:53:00.000-07:002010-02-04T08:17:34.975-08:00A Beautiful Race ConditionI recently gave a keynote at the ICIS 2009 conference in Shanghai. The topic was why multithreaded programming seemed so easy, yet turned out to be so hard. The fun part was I investigated (per my last post and this one) several old, personal concurrency demons I knew existed but wanted to know more about.<br /><br />One of those was, indeed, my favorite race condition. It doesn't escape me that its probably wholly unhealthy to even *have* a favorite race condition (akin to having a favorite pimple or something) - but nonetheless, the elegance of this one still makes my heart aflutter.<br /><br />The scenario of this race is that we assume, not terribly inaccurately, that race conditions at times, can cause corrupted data. However, what if we have a situation where we sort of don't mind some corrupted data? A "good enough" application as it were.<br /><br />The dangerous part of all this is if we assume (without digging in) what kind of data corruption can happen. As you'll see, you might just not get the type of data corruption you were hoping for (which is one of the sillier sentences I've ever written).<br /><br />The particular instance of this kind of happy racing I've encountered is where someone uses a java.util.HashMap as a cache. I've never done such a thing myself, but I heard about this race and thus this analysis. They may use it with a linked-list or maybe just raw, but the baseline is that they figure a synchronized HashMap will be expensive - and in their case, a race condition inside the HashMap will just lose (or double up on) an entry now and then.<br /><br />That is - a race condition between two (or more) threads might accidentally drop an entry causing an extra cache miss - no biggie. Or, it may cause one thread to re-cache an entry that didn't need it. Also no biggie. In other words, a slightly imprecise, yet very fast cache is ok by them. (of course, this assumption is dead wrong - don't do that - read on for why!)<br /><br />So they setup a HashMap in some global manner, and allow any number of nefarious threads bang away on it. Let them put and get to their hearts content.<br /><br />Now if you happen to know how HashMap works, if the size of the map exceeds a given threshold, it will act to resize the map. It does that by creating a new bucket array of twice the previous size, and then putting every old element into that new bucket array.<br /><br />Here's the core of the loop that does that resize:<br /><br /><span style="font-family: courier new;font-size:75%;"><br /><b>1:</b> // Transfer method in java.util.HashMap -<br /><b>2:</b> // called to resize the hashmap<br /><b>3:</b> <br /><b>4:</b> for (int j = 0; j < src.length; j++) {<br /><b>5:</b> Entry<K,V> e = src[j];<br /><b>6:</b> if (e != null) {<br /><b>7:</b> src[j] = null;<br /><b>8:</b> do {<br /><b>9:</b> Entry<K,V> next = e.next; <br /><b>10:</b> int i = indexFor(e.hash, newCapacity);<br /><b>11:</b> e.next = newTable[i];<br /><b>12:</b> newTable[i] = e;<br /><b>13:</b> e = next;<br /><b>14:</b> } while (e != null);<br /><b>15:</b> }<br /><b>16:</b> } <br /></span><br /><br />Simply, after line 9, variable <span style="font-family: courier new;font-size:85%;">e</span> points to a node that is about to be put into the new (double-wide) bucket array. Variable <span style="font-family: courier new;font-size:75%;"><br />next</span> holds a reference to the next node in the existing table (because in line 11, we'll destroy that relation). <br /><br />The goal is that nodes in the new table get scattered around a bit. There's no care to keep any ordering within a bucket (nor should there be). HashMap's don't care about ordering, they care about constant time access.<br /><br />Graphically, let's say we start with the HashMap below. This one only has 2 buckets (the default of java.util.HashMap is 16) which will suffice for explanatory purposes (and save room).<br /><br />As our loop starts, we assign <span style="font-family: courier new;font-size:85%;">e</span> and <span style="font-family: courier new;font-size:85%;">next</span> to A and B, respectively. The A node is about to be moved, the B node is next.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhN5vRvRXZwb00aU_plmJB51pyq68Bxc7ib8qjSAeXO5aHy_onGVPhwcQ_o0evSGJGIjYxyO3_vHWEiWHHMQlt3l4evlwz8DLVJhVuxh59b054Ee3NFEZnWqvJeZpxW5iTDtHaRSHonkJk/s1600-h/originalmap.gif"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 400px; height: 118px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhN5vRvRXZwb00aU_plmJB51pyq68Bxc7ib8qjSAeXO5aHy_onGVPhwcQ_o0evSGJGIjYxyO3_vHWEiWHHMQlt3l4evlwz8DLVJhVuxh59b054Ee3NFEZnWqvJeZpxW5iTDtHaRSHonkJk/s400/originalmap.gif" border="0" alt=""id="BLOGGER_PHOTO_ID_5345319645122653778" /></a><br /><br />We have created a double-sized bucket array (in this case size=4) and migrate node A in iteration 1.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5tZ061GLxeFY-qiQ8jIE-Z3igGP0I09Xprt5jhmZBWuFezQaa1VhBvKe7pt6LTzMCZfLCvSkq3KxWDLmI9dYlqZzlOc4KSjsNHhrTh5bbbNtacSM3DIbK8_pgO1sJGAH8X4shd-tgx0s/s1600-h/iteration1.gif"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 217px; height: 374px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5tZ061GLxeFY-qiQ8jIE-Z3igGP0I09Xprt5jhmZBWuFezQaa1VhBvKe7pt6LTzMCZfLCvSkq3KxWDLmI9dYlqZzlOc4KSjsNHhrTh5bbbNtacSM3DIbK8_pgO1sJGAH8X4shd-tgx0s/s400/iteration1.gif" border="0" alt=""id="BLOGGER_PHOTO_ID_5345320399681179794" /></a><br /><br />Iteration 2 moves node B and Iteration 3 moves node C. Note that next=null is the ending condition of our while loop for migrating any given bucket (read that again, its important to the end of the story).<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi7GCGb6mrUorHJ2DnzZnAKqvZFkVJuQoRaYhWUdJX7fxaxQEgcAaKEAovaFie6EeuMauE9X8FSapdxL0V4ryG4NTojvcP-QSRuVgEBQRXMcH1KV7z868spf49Zj1-ZBw0QiY163GBNhlA/s1600-h/iteration2.gif"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 344px; height: 238px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi7GCGb6mrUorHJ2DnzZnAKqvZFkVJuQoRaYhWUdJX7fxaxQEgcAaKEAovaFie6EeuMauE9X8FSapdxL0V4ryG4NTojvcP-QSRuVgEBQRXMcH1KV7z868spf49Zj1-ZBw0QiY163GBNhlA/s400/iteration2.gif" border="0" alt=""id="BLOGGER_PHOTO_ID_5345321489485319746" /></a><br /><br />Also important to the story, note that the migration inverted the order of Node's A and B. This was incidental to the smart idea of inserting new nodes at the top of the list instead of traversing to find the end each time and plunking them there. A normal put operation would still have to check that its inserting (and not replacing) but given a resize can't replace, this saves us a lot of "find the end" traversals.<br /><br />Finally, after iteration 3, our new HashMap looks like this:<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgO0KPWQiImoJNXdo-31CKb5aj_H0_tdk2vhVrvGV2qKb1ooSgIT29c1DO1hca6KKIXh7gFK5gItqQjPVdbwUVIamPcY7wmZEMu5oQDD7IbBYy2ru2GIoUqwZkgFPa4Ma5jH75EyhHWTAE/s1600-h/iteration3.gif"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 393px; height: 292px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgO0KPWQiImoJNXdo-31CKb5aj_H0_tdk2vhVrvGV2qKb1ooSgIT29c1DO1hca6KKIXh7gFK5gItqQjPVdbwUVIamPcY7wmZEMu5oQDD7IbBYy2ru2GIoUqwZkgFPa4Ma5jH75EyhHWTAE/s400/iteration3.gif" border="0" alt=""id="BLOGGER_PHOTO_ID_5345321806289535250" /></a><br /><br />Our resize accomplished precisely the mission it set out to. It took our 3-deep bucket and morphed it into a 2-deep and 1-deep one. <br /><br />Now, that's all well and good, but this article isn't about HashMap resizing (exactly), its about a race condition.<br /><br />So, let's assume that in our original happy HashMap (the one above with just 2 buckets) we have two threads. And both of those threads enter the map for some operation. And both of those threads simultaneously realize the map needs a resize. So, simultaneously they both go try to do that.<br /><br />As an aside, the fact that this HashMap is unsynchronized opens it up to a scary array of unimaginable visibility issues but that's another story. I'm sure that using an unsynchronized HashMap in this fashion can wrack evil in ways unlike man has ever seen, I'm just addressing one possible race in one possible scenario.<br /><br />Ok.. back to the story.<br /><br />So two threads, which we'll cleverly name Thread1 and Thread2 are off to do a resize. Let's say Thread1 beats Thread2 by a moment. And let's say Thread1 (by the way, the fun part about analyzing race conditions is that nearly anything can happen - so you can say "Let's say" all darn day long and you'll probably be right!) gets to line 10 and stops. Thats right, after executing line 9, Thread1 gets kicked out of the (proverbial) CPU.<br /><br /><br /><span style="font-family: courier new;font-size:75%;"><br /><b>1:</b> // Transfer method in java.util.HashMap -<br /><b>2:</b> // called to resize the hashmap<br /><b>3:</b> <br /><b>4:</b> for (int j = 0; j < src.length; j++) {<br /><b>5:</b> Entry<K,V> e = src[j];<br /><b>6:</b> if (e != null) {<br /><b>7:</b> src[j] = null;<br /><b>8:</b> do {<br /><b>9:</b> Entry<K,V> next = e.next; <br /> <b> // Thread1 STOPS RIGHT HERE</b><br /><b>10:</b> int i = indexFor(e.hash, newCapacity);<br /><b>11:</b> e.next = newTable[i];<br /><b>12:</b> newTable[i] = e;<br /><b>13:</b> e = next;<br /><b>14:</b> } while (e != null);<br /><b>15:</b> }<br /><b>16:</b> } <br /></span><br /><br />Since it passed line 9, Thread1 did get to set its <span style="font-family: courier new;font-size:85%;">e</span> and <span style="font-family: courier new;font-size:85%;">next</span> variables. The situation looks like this (I've renamed <span style="font-family: courier new;font-size:85%;">e</span> and <span style="font-family: courier new;font-size:85%;">next</span> to <span style="font-family: courier new;font-size:85%;">e1</span> and <span style="font-family: courier new;font-size:85%;">next1</span> to keep them straight between the two threads as both threads have their own <span style="font-family: courier new;font-size:85%;">e</span> and <span style="font-family: courier new;font-size:85%;">next</span>).<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgbEYO90odkCdEoALXMRTVSgH9ncyImoH8E1G-JxFW2NrszaUpfmlVxPm76mj6AfoR9J9Wa8N22h84TN45sTqbwuOCqdAroMxxPs8s5L8-tSN3rd6ZQgJrGRjKji4aa7YWkVQZ6aYp-Rmw/s1600-h/e1next1.gif"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 400px; height: 122px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgbEYO90odkCdEoALXMRTVSgH9ncyImoH8E1G-JxFW2NrszaUpfmlVxPm76mj6AfoR9J9Wa8N22h84TN45sTqbwuOCqdAroMxxPs8s5L8-tSN3rd6ZQgJrGRjKji4aa7YWkVQZ6aYp-Rmw/s400/e1next1.gif" border="0" alt=""id="BLOGGER_PHOTO_ID_5345324797645937650" /></a><br /><br />Again, Thread1 didn't actually get to move any nodes (by this time in the code, it did allocate a new bucket array).<br /><br />What happens next? Thread2, that's what. Luckily, what Thread2 does is simple - let's say it runs through the full resize. All the way. It completes.<br /><br />We get this:<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjF_6CvmhN3h-6slZXg1gFVpz2hASdXXIzwtyJKmIzQWxDgy4dt3GPCQJPefThA5oXcRs32yL2gIdZ5y5FzT4ohHa5M1BN3cb0JyqOWCImmAiDqNkS72QEASeZrhK2491Z3v7ZeWiVu0_M/s1600-h/thread2done.gif"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 400px; height: 241px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjF_6CvmhN3h-6slZXg1gFVpz2hASdXXIzwtyJKmIzQWxDgy4dt3GPCQJPefThA5oXcRs32yL2gIdZ5y5FzT4ohHa5M1BN3cb0JyqOWCImmAiDqNkS72QEASeZrhK2491Z3v7ZeWiVu0_M/s400/thread2done.gif" border="0" alt=""id="BLOGGER_PHOTO_ID_5345325729454137970" /></a><br /><br />Note that <span style="font-family: courier new;font-size:85%;">e1</span> and <span style="font-family: courier new;font-size:85%;">next1</span> still point to the same nodes. But those nodes got shuffled around. And <b>most importantly</b> the next relation got reversed.<br /><br />That is, when Thread1 started, it had node A with its next as node B. Now, its the opposite, node B has its next as node A.<br /><br />Sadly (and paramount to the plot of this story) is that Thread1 doesn't know that. If you're thinking that the invertedness of Thread1's <span style="font-family: courier new;font-size:85%;">e1</span> and <span style="font-family: courier new;font-size:85%;">next1</span> are important, you're right.<br /><br />Here's Thread1's next few iterations. We start with Thread2's bucket picture because thats really the correct "next" relations for our nodes now.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbc-EZ9K_R9MgCVx9VReZrlJLcSfqrI0CKlG0jQ95BCKCAuOwqwHGm6wqrtYE4vAOam17_1E-yDaA3OzfMHdO_r3N0IXSYzJWg8lArPJYpCAzW3pg-AHZMp3Ar00cmKHhoCrGoV3iMhbc/s1600-h/thread1-1.gif"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 400px; height: 209px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbc-EZ9K_R9MgCVx9VReZrlJLcSfqrI0CKlG0jQ95BCKCAuOwqwHGm6wqrtYE4vAOam17_1E-yDaA3OzfMHdO_r3N0IXSYzJWg8lArPJYpCAzW3pg-AHZMp3Ar00cmKHhoCrGoV3iMhbc/s400/thread1-1.gif" border="0" alt=""id="BLOGGER_PHOTO_ID_5345328287976839890" /></a><br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3O5wKCNoQUe-Vg87vJ9nVEfYFCxO5k9O3GWFRu9kKd9LRwC9JKHD4OTihHGS4IAtqDYkk8D4n1xMlLhZbnlZebeEPzHI6DA8bZbYRwhO0z1Bfenpc1QGD_JRZf4pv5fD-LTh9BF4taNI/s1600-h/thread1-2.gif"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 331px; height: 301px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3O5wKCNoQUe-Vg87vJ9nVEfYFCxO5k9O3GWFRu9kKd9LRwC9JKHD4OTihHGS4IAtqDYkk8D4n1xMlLhZbnlZebeEPzHI6DA8bZbYRwhO0z1Bfenpc1QGD_JRZf4pv5fD-LTh9BF4taNI/s400/thread1-2.gif" border="0" alt=""id="BLOGGER_PHOTO_ID_5345328511609856674" /></a><br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgWbXuVKTMMPP96dBMxQsQPE3bzIWRRHWi5fhtk0VV86jyUxgjQwjEytIR5yNfoWbURoa8X4PDW76lS39jghg4Cb0hSqGKexWUikLoio54V9kqcrI0Y2RaP04dtnV3NYtkpC7u7baNsfak/s1600-h/thread1-3.gif"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 400px; height: 255px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgWbXuVKTMMPP96dBMxQsQPE3bzIWRRHWi5fhtk0VV86jyUxgjQwjEytIR5yNfoWbURoa8X4PDW76lS39jghg4Cb0hSqGKexWUikLoio54V9kqcrI0Y2RaP04dtnV3NYtkpC7u7baNsfak/s400/thread1-3.gif" border="0" alt=""id="BLOGGER_PHOTO_ID_5345328755695978306" /></a><br /><br />Everything sort of looking ok.. except for our <span style="font-family: courier new;font-size:85%;">e</span> and <span style="font-family: courier new;font-size:85%;">next</span> at this point. The next iteration will plunk A into the front of the bucket 3 list (it is after all, next). And will assign its next to whatever happens to already be there - that is, node B.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjmA_fkB7CoD_rHN2iOnZOt7xDDPoFtMFsM1SmhuDF6vCiDgBIso7EuXv1AQDoIpoBG78JDOiOwfR-tu7Fmb49INzYq4pJPr4WDdXBu5pILP5XQlJkGzh7harwGMZxg2FO8P-Kt4_PEXFA/s1600-h/thread1-4.gif"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 400px; height: 192px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjmA_fkB7CoD_rHN2iOnZOt7xDDPoFtMFsM1SmhuDF6vCiDgBIso7EuXv1AQDoIpoBG78JDOiOwfR-tu7Fmb49INzYq4pJPr4WDdXBu5pILP5XQlJkGzh7harwGMZxg2FO8P-Kt4_PEXFA/s400/thread1-4.gif" border="0" alt=""id="BLOGGER_PHOTO_ID_5345329178416880242" /></a><br /><br />Woah. Thar she blows.<br /><br />So right about now Thread1 goes into, what we like to call in the biz, an "infinite loop". Any subsequent get operations that hit this bucket start searching down the list and, go into, yep - an infinite loop. Any put operation that first needs to scan the nodes to see if its going to do a replace, will, you guessed it, go into an infinite loop. Basically, this map is a pit of infinite loopeness.<br /><br />If you remember we noted that race conditions cause data corruption. Well, yeah, thats what we have here - just very unlucky data corruption on pointer structures. I'm the first to admit this stuff is tricky - if you found errors in my analysis I'll happily fix this post.<br /><br />Now I had the happy fortune for a time of sharing an office with Josh Bloch who wrote java.util.HashMap. When I innocently mentioned he had a bug in his code given this behavior, he did indeed (to use Josh's word's) go non-linear on me.<br /><br />And he was right. This is not a bug. HashMap is built specifically for its purpose and this implementation is not intended as threadsafe. There's a gaggle of ways to make it threadsafe, but in plain, vanilla, (and very fast) form - its not. And needless to say, you shouldn't be using it that way.<br /><br />Race conditions are nothing to mess with and the worst ones are the ones that don't crash your program but let it wander down some unintended path. Synchronization isn't just for fun you know.<br /><br />And nefarious uses of HashMap aside, I still attest - this is, indeed, a beautiful race.<br /><br />Addendum: I've been yelled at a few times for calling any race condition "beautiful". I'll defend myself by our apparently human nature to generally call intricate complexity beautiful (i.e. waves crashing on a shore, nature in general). <br /><br />Most races end up being about data-corruption. This one is data-corruption that manifests as control-flow-corruption. And it does so fantastically without an error (infinite loops notwithstanding).<br /><br />As the evolution analogy goes, if you drive a needle into a pocket watch - chances are you'll simply destroy it. But there's a tiny chance you'll actually make it a better watch (clearly, not the case here). And another tiny chance you'll simply make it something "different" - but still, per se, functioning. <br /><br />Again, my use of "beautiful" might be more appropriate as "a complex mutation with surprising non-destruction" :)paulhttp://www.blogger.com/profile/11412172362500455307noreply@blogger.com17tag:blogger.com,1999:blog-8523046672726455213.post-40979082001615729292009-06-08T08:00:00.000-07:002009-06-08T13:29:01.457-07:00On Java VisibilityA semi-famous example of broken Java synchronization is this:<br /><span style="font-family: courier new;font-size:85%;"><br />class SomeClass {<br /><br /> private boolean keepGoing = true;<br /><br /> public boolean get() {<br /> return keepGoing;<br /> }<br /><br /> public synchronized boolean set(boolean x) {<br /> keepGoing = x;<br /> }<br />}<br /></span><br />I believe this example is in Josh Bloch's book "Effective Java" (which I now notice I somehow lost my copy in my most recent move - don't tell Josh).<br /><br />The idea is here that someone (ostensibly) thought that they'd save some performance by not synchronizing a getter and just synchronizing the setter. There definitely is a cost to synchronization and needless to say, getting doesn't change anything - so why bother paying that cost for gets?<br /><br />As has been pointed out long before this post, without a synchronize on a getter, there is no guarantee of visibility when doing the get. That is, if one thread calls set(false), there's no guarantee that any other thread will know it happened.<br /><br />Consider code that might use the code above:<br /><br /><span style="font-family: courier new;font-size:85%;"><br />class SomeOtherClass {<br /><br /> SomeClass keepGoing = new SomeClass();<br /><br /> class Thread1 implements Runnable {<br /> public void run() {<br /> while (keepGoing.get()) x++;<br /> System.out.println("done1");<br /> }<br /> }<br /><br />class Thread2 implements Runnable {<br /> public void run() {<br /> keepGoing.set(false);<br /> }<br /> }<br />}<br /></span><br /><br />Let's say you start a Thread1 running. And of course, keepGoing.get() is true, so it just keeps looping along. Then lets say an eternity (or maybe 2 seconds) later you start Thread2.<br /><br />If we had reliable visibility, Thread1 would end the moment after Thread2 sets keepGoing to false.<br /><br />If you've read Josh's book, its no surprise that it doesn't. Specifically, Thread1 doesn't end. It keeps going. Thread2 ends happily and Thread1 never ends.<br /><br />The only interesting part to me was that this always worked. Always. Adding to the complexity of visibility concerns is that memory is "leaky". That is, even without guaranteed visibility you often get unreliable visiblity. <br /><br />So, I dug a little deeper. If you're adventurous enough to grab a debug-version of the JDK and run it with the -XX:+PrintOptoAssembly option. You get to see the optimizations the JIT are making. That is, you see the assembly code version of your Java code - post-optimization. Check out <a href=http://weblogs.java.net/blog/kohsuke/archive/2008/03/deep_dive_into.html>Koshuke Kawaguchi's Blog</a> for some instructions.<br /><br />So here's Thread1's loop code after JIT optimization at runtime:<br /><span style="font-family: courier new;font-size:75%;"><br />02c movq R10, precise klass manybrain/test/Main: 0x0000000040a50968:Constant:exact * # ptr<br />036 movsbl R8, [R10 + #596 (32-bit)] # byte ! Field manybrain/test/Main.keepGoing<br />03e testl R8, R8<br /><b>041 je,s B4 P=0.000000 C=147944.000000</b><br />041<br /><b>043 B2: # B3 <- B1 Freq: 1</b><br />043 movl R8, [R10 + #592 (32-bit)] # int ! Field manybrain/test/Main.x<br />043<br />04a B3: # B3 <- B2 B3 top-of-loop Freq: 1e-35<br />04a incl R8 # int<br />04d movl [R10 + #592 (32-bit)], R8 # int ! Field manybrain/test/Main.x<br />054 testl rax, [rip + #offset_to_poll_page] # Safepoint: poll for GC # manybrain.test.Main$Thread1::run @ bci:14 L[0]=_<br /> # OopMap{r10=Oop off=84}<br /><b>05a jmp,s B3</b><br />05a<br />05c B4: # N53 <- B1 Freq: 4.76837e-07<br />05c movl RSI, #27 # int<br />061 nop # 2 bytes pad for loops and calls<br /></span><br /><br />If you're old school, welcome home.<br /><br />If you're not, then this might look like a lot of goop. So let's parse out just the interesting parts.<br /><br />Line 41 is an conditional jump. Basically, if keepGoing (per the comment in line 36) is false, we jump to line 5C (label B4) and end the code segment. You and I know that keepGoing started true, so basically that jump isn't followed.<br /><br />Lines 43-5a are the loop that does x++.<br /><br />And checkout line 5a. That is an unconditional jump back to the top of the loop. So what does all this mean? That the JIT did some very aggressive optimization. In fact, remember our original loop from Thread1?<br /> <br /><span style="font-family: courier new;font-size:85%;"><br />while (keepGoing.get()) x++<br /></span><br /><br />The JIT has optimized this to:<br /><span style="font-family: courier new;font-size:85%;"><br />if (keepGoing.get()) {<br /> while (true) x++;<br />}<br /></span><br /><br />No wonder the loop never ends. You've got bigger problems now than a little leaky visibility issue. I'm positive I'm oversimplifying, but effectively the JIT saw your get method wasn't synchronized and made the assumption that it could optimize as such. If you didn't guarantee visibility, it didn't need to either. Obviously, add the synchronization modifier to the get method and all this badness won't happen.<br /><br />Moral of the story is much like the inevitable topics discussed at a lunch with <a href=http://jeremymanson.blogspot.com>Jeremy Manson</a> - you can't optimize away correct synchronization; as all you'll probably do is optimize the "correctness" part away.paulhttp://www.blogger.com/profile/11412172362500455307noreply@blogger.com5tag:blogger.com,1999:blog-8523046672726455213.post-53481387594405845882008-02-18T14:39:00.000-08:002008-02-18T14:49:51.248-08:00Kill the myth please. NIO is *not* faster than IOI'm giving 3 talks at the Software Developer's Conference West in Santa Clara, CA on March 3-7.<br /><br />One of them is a tutorial on how to interview in Silicon Valley (fun stuff).<br /><br />Another is how to write high-performance servers in Java. Specifically, why NIO is really not the best way anymore (post linux 2.6 kernel and NPTL) and multithreaded I/O is the new old-way of doing things. Its a fun talk and largely discusses the internals of Mailinator and how it runs a few thousand simultaneous threads without breaking a sweat.<br /><br />On top of that, as it turns out, in pure throughput, IO smokes NIO in all tests I tried. And I'm not alone - Rahul Bhargava of Rascal Systems did a very nice analysis of this and posted it, sadly, in some forums at theserverside.com.<br /><br />The post is solid gold and I'm posting it below just for posterity as I'd hate to see those forums come down some day and I lose access to that post. Incidentally, I disagree with Rahul that "fewer threads are easier to debug". It only takes 2 to make a deadlock or race condition. <br /><br />The original URL for this post is:<br /><span style="font-size:78%;">http://www.theserverside.com/discussions/thread.tss?thread_id=26700</span><br /><br />And like I said, it was written by Rahul Bhargava of Rascal Systems. Here it is:<br /><br />-------------------------------------------------<br /><br />I have been benchmarking Java NIO with various JDKs on Linux. Server is<br />running on a 2 CPU 1.7 GHz, 1GB RAM, Ultra160 SCSI 36GB disk<br /><br />With Linux kernel 2.6.5 (Gentoo) I had NPTL turned on and support for<br />epoll compiled in. The server application was designed to support<br />multiple disptach models :<br /><br />1. Reactor with Iterative Disptach with multiple selector threads. Essentially<br />the accepted connections were load-balanced between varying number of<br />selector threads. The benchmark then applied a step function to experimentally<br />determine the optimal # of threads and connection per selector ratio.<br /><br />2. Also a simple concurrent blocking disptach model was supported. This is<br />essentially a reader thread per connection model.<br /><br />Client application opens concurrent persistent connections to the server<br />and starts blasting messages. Server just reads the messages and does<br />basic un-marshalling to ensure message is ok.<br /><br />Results were interesting:<br /><br />1. With NPTL on, Sun and Blackwidow JVM 1.4.2 scaled easily to 5000+ threads. Blocking<br />model was consistently 25-35% faster than using NIO selectors. Lot of techniques suggested<br />by EmberIO folks were employed - using multiple selectors, doing multiple (2) reads if the first<br />read returned EAGAIN equivalent in Java. Yet we couldn't beat the plain thread per connection model<br />with Linux NPTL.<br /><br />2. To work around not so performant/scalable poll() implementation on Linux's we tried using<br />epoll with Blackwidow JVM on a 2.6.5 kernel. While epoll improved the over scalability, the<br />performance still remained 25% below the vanilla thread per connection model. With epoll<br />we needed lot fewer threads to get to the best performance mark that we could get out of NIO.<br /><br />Here are some numbers:<br /><br />(cc = Concurrent Persistent Connections, bs = Is blocking server mode on Flag,<br />st = Number of server threads, ct = Connections handled per thread,<br />thruput = thruput of the server )<br /><br />cc, bs,st,ct, thruput<br />1700,N,2,850,1379<br />1700,N,4,425,1214<br />1700,N,8,212,1240<br />1700,N,16,106,1140<br />1700,N,32,53,1260<br />1700,N,64,26,1115<br />1700,N,128,13,886<br />1700,N,256,6,618<br />1700,N,512,3,184<br />1700,Y,1700,1,1737<br /><br />As you can see the last line indicates vanilla blocking server (thread per connection)<br />produced the best thruput even with 1700 threads active in the JVM.<br /><br />With epoll, the best run was with 2 threads each handling around 850 connections in<br />their selector set. But the thruput is below the blocking server thruput by 25%!<br /><br />Results shows that the cost of NIO selectors coupled with OS polling mechanism (in<br />this case efficient epoll VS selector/poll) has a significant overhead compared to<br />the cost of context switching 1700 threads on an NPTL Linux kernel.<br /><br />Without NPTL of course it's a different story. The blocking server just melts at 400 concurrent<br />connections! We have run the test upto 10K connections and the blocking server outperformed<br />NIO driven selector based server by same margin. Moral of the story - NIO arrives at the scene<br />a little too late - with adequate RAM and better threading models (NPTL), performance gains<br />of NIO don't show up.<br /><br />Sun's JVM doesn't support epoll() so we couldn't use epoll with it. Normal poll() based<br />selector from Sun didn't perform as well. We needed to reduce the number of connections<br />per thread to a small number (~ 6-10) to get comprabale numbers to epoll based selector.<br />That meant running lot more selector threads kind of defeats the purpose of multiplexed IO.<br />The benchmarks also dispell the myth created by Matt Welsh et al (SEDA) that a single<br />threaded reactor can keep up with the network. On a 100Mbps ethernet that was true: network<br />got saturated prior to server CPUs but with > 1Gbps network, we needed multiple selectors<br />to saturate the network. One single selector's performance was abysmal (5-6x slower than<br />concurrent connections)<br /><br />For application that want to have fewer number of threads for debuggability etc, NIO may be<br />the way to go. The 25-35% performance hit may be acceptable to many apps. Fewer threads<br />also means easier debugging, it's a pain to attach a profiler or a debugger to a server hosting<br />1000+ threads :-) . Bottom line with better MT support in kernels (Linux already with NPTL), one<br />needs to re-consider the thread per connection model<br /><br />Rahul Bhargava<br />CTO, Rascal Systemspaulhttp://www.blogger.com/profile/11412172362500455307noreply@blogger.com8tag:blogger.com,1999:blog-8523046672726455213.post-1140760753351114812008-01-21T14:49:00.000-08:002013-07-29T07:13:32.009-07:00Your own private MailinatorAs you probably know, Mailinator has a always-growing list of alternate domains. However, those are just the domains sponsored by us.<br />
<br />
In truth - Mailinator accepts ANY email you fling at it*. In other words - you can setup your OWN personal domains simply by setting your MX record (i.e. through your registrar or DNS server) to point your mail delivery to Mailinator!<br />
<br />
Simply put - if you own YOURDOMAIN.com, you can simply change that domain's MX record to point to <span style="font-weight: bold;">mail.mailinator.com</span> and all email you send to @YOURDOMAIN.com will then end up in Mailnator.<br />
<br />
You can publicize this fact - or you can keep your personal alternate domain a secret and have your own private Mailinator !<br />
<br />
*pursuant to normal draconian Mailinator email rejection criteria including but not limited to hate email, unlawful email, etc.paulhttp://www.blogger.com/profile/11412172362500455307noreply@blogger.com20tag:blogger.com,1999:blog-8523046672726455213.post-20774552418660566322007-01-19T11:39:00.000-08:002007-05-17T21:54:55.846-07:00The Architecture of MailinatorAlmost 3.5 years ago I started the Mailinator(tm) service. I got the bulk of the idea from my drunk roommate at the time and the first incarnation took me all of about 3 days to code up. In some senses it was a crazy idea. As far I know, it was the first site of its kind. A web-based email service that allowed any incoming email to create an inbox. No sign-up. No personal information. Send email first, check email later.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://photos1.blogger.com/x/blogger/1228/848/1600/362151/zzz.jpg"><img style="float:right; margin:0 0 10px 10px;cursor:pointer; cursor:hand;" src="http://photos1.blogger.com/x/blogger/1228/848/400/842744/zzz.jpg" border="0" alt="" /></a><br /><br />This became ridiculously handy for things like signing up for websites that send you one confirmation email, then save or sell or spam your email address forever. And of course, it *is* very handy for users. But think about it from mailinator's side. Its basically signing up to receive spam for that address forever. That's a tall order and one that seems to have the possibility of a terrible demise. Someday, enough email could come in that will simply smush Mailinator. But, as of this writing, that day isn't today.<br /><br />I have in that 3.5 years received hundreds of "thank you" emails, a pile of "it doesn't work" emails, a radio interview, articles in the Washington Post, New York Times, and Delta Skymiles magazine, 1 call from both Scotland Yard and the LAPD, and a total of 4 subpoenas (1 of those being a Federal Grand Jury subpoena issued by the FBI).<br /><br />At this point, Mailinator averages approximately 2.5million emails per day. I have seen hourly spikes that would result in about 5million in a day. (<i>Edit: Feb 2007 - One month later we're averaging 4.5million emails a day with spikes over 6million</i>) In addition, the system also services several thousand web users and several thousand RSS users per day.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://photos1.blogger.com/x/blogger/1228/848/1600/543895/hasreceived.jpg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;" src="http://photos1.blogger.com/x/blogger/1228/848/400/194492/hasreceived.jpg" border="0" alt="" /></a><br />In the world of email services, this probably isn't all that much. The most interesting part to me is that the complete set of hardware that mailinator uses is one little server. Just one. A very modest machine with an AMD 2Ghz Athlon processor, 1G of ram (although it really doesn't need that much), and a boring IDE, 80G hard drive (Check <a href=http://www.serverbeach.com/catalog/index.php?REF=BA83A6U6H2>ServerBeach's</a> Category 1 Powerline 2100 for the exact specs). And honestly, its really not very busy at all. I've read the blogs of some copycat services of Mailinator where their owners were upgrading their servers to some big iron. This was really the impetus for me writing down this document - to share a different point of view.<br /><br />Mailinator easily handling a few million emails a day wasn't always the case. The initial mailinator system was quite busy. And in fact, got overwhelmed about a year ago when email traffic started topping 800,000 a day (that's my recollection anyway). In an effort to squeeze life out of the<br /><a href="http://www.amazon.com/gp/search?ie=UTF8&keywords=server&tag=mailinator-20&index=pc-hardware&linkCode=ur2&camp=1789&creative=9325">server</a><img src="http://www.assoc-amazon.com/e/ir?t=mailinator-20&l=ur2&o=1" width="1" height="1" border="0" alt="" style="border:none !important; margin:0px !important;" /> and as an exercise in putting together some principles I always championed about server development, I rewrote the system from scratch. I have no idea what the current limit of the existing system is, but at 2.5million a day, its not even breaking a sweat.<br /><br />If you don't know what Mailinator is, take a small tour through the (rather funny) <a href="http:///www.mailinator.com/faq.jsp">FAQ</a>.<br /><br /><b>Lossy lossie lossee</b><br /><br />There is a very important point to note about the Mailinator service. And that is, that it is indeed - free. Although it might not seem like it, it has an immense impact on the design (as you'll see). This allowed me to favor performance across the board of the design. This fact influenced decisions from how I dealt with detecting spam all the way down to how I synchronized some code blocks. No kidding.<br /><br />The basic tenet is that I do not have to provide perfect service. In order to do that, my hardware requirements would be much higher. Now that would all be fine and dandy if people were paying for the service. I could then provide support and guarantees. But given its free I instead went for, in order, these two design decisions:<br /><br />1) Design a system that values survival above all else even users (as of course, if its down, users aren't really getting much out of it)<br />2) Provide 99.99% uptime and accuracy for users.<br /><br />If you wonder what I mean about "survival" in the first line, it basically means that Mailinator is attacked on literally a daily basis. I wanted to make a system that could survive the large majority of those attacks. Note - I'm not interested in it surviving all of them. Because again, if some zombie network decided to Denial-of-service me - I really have no chance of thwarting it without some serious hardware. The good news is that if someone goes to all the trouble of smashing Mailinator (again referencing the fact we're lossy), I really don't lose much sleep over it. It sucks for my users - but there really isn't anything I can do anyway. I'm not trying to be cavalier about this - I went to great lengths to handle attacks, I'm just saying its a cold reality that I simply cannot stop them all. Thus I accept them as part of the game.<br /><br /><b>The platform</b><br /><br />The original Mailinator used a relatively standard<br /><a href="http://www.amazon.com/gp/search?ie=UTF8&keywords=unix&tag=mailinator-20&index=pc-hardware&linkCode=ur2&camp=1789&creative=9325">unix</a><img src="http://www.assoc-amazon.com/e/ir?t=mailinator-20&l=ur2&o=1" width="1" height="1" border="0" alt="" style="border:none !important; margin:0px !important;" /> stack of applications including a Java based web application running in Tomcat. Mailinator is and was of course, always just a hobby. I had a day job (or 2) so months of development just was never an option. I chose Java for no other reason than I knew Java better than anything else. For email, it used sendmail with a special rule that directed any incoming email to mailinator.com to one single mailbox.<br /><br />Sendmail --> disk --> Mailinator <-- Tomcat Servlet Engine<br /><br />The Java based mailinator app then grabbed the emails using IMAP and/or POP (it changed over time) and deleted them. I should have used an mbox interface but I never got around to implementing that. The system then loaded all emails into memory and let them sit there. Mailinator only allowed about 20000 emails to reside in memory at once. So when a new one came, the oldest one got pushed out.<br /><br />The FAQ advertises that emails stick around for "a couple of hours." And that was true, but exactly how long mattered on the rate of incoming emails. You'll also note an interesting side effect that since all emails lived in memory, if the server came down - all emails were lost! Talk about exploiting the fact that my service was free huh? This may seem dubious but the code was really quite stable and ran for weeks and months without downtime.<br /><br />I thought about saving emails into a database of course but honestly, all this bought me was emails that stuck around longer. And, that in and of itself sort of went against my intent for mailinator. The ideas was, sign-up for something, goto Mailinator, click the link, and forget about it. If you want a<br /><a href="http://www.amazon.com/gp/search?ie=UTF8&keywords=mailbox&tag=mailinator-20&index=pc-hardware&linkCode=ur2&camp=1789&creative=9325">mailbox</a><img src="http://www.assoc-amazon.com/e/ir?t=mailinator-20&l=ur2&o=1" width="1" height="1" border="0" alt="" style="border:none !important; margin:0px !important;" /><br /> where emails last a few days, thats fine, but there are many other alternatives out there - that's not what Mailinator is about. I forgot the database idea and now shoot for mails that last somewhere around 3-4 hours.<br /><br />This all worked fabulously for awhile. It pretty much filled up all 1G of ram of the server. Finally when the incoming email rate started surpassing 800,000 a day, the system started to break down. I believe it was primarily the disk contention between unix mail apps and the Java app locking mailboxes. Regardless, there were many issues with that system that bugged me for a long time. The root of most of those problems really boiled down to one thing - the disk. The disk activity of sendmail, procmail, logging and whatever else was a silly bottleneck. And it needed to go.<br /><br />More than a year ago now I did a full rewrite. Much of the anti-spam code that I'll describe later was already in this code-base but was improved and extended for the new system.<br /><br /><b>Synchronous vs. Asynchronous I/O</b><br /><br />I've read a fair number of articles on the wonders of asynchronous I/O (java's NIO library). I don't doubt them but I decided against using it. Primarily, again, because I did a great deal of work in multithreaded environments and knew that area well. I figured if I had performance issues later, I could always switch over to NIO as a learning experience.<br /><br />The biggest thing I knew I needed to do with Mailinator was to remove the unix application components. Mailinator needed to stop outsourcing its email receipt and do it itself. This basically meant I needed to write my own SMTP server. Or at least, a subset of one. Firstly, Mailinator has never had the ability to send email so I didn't need to code that part up. Second, I had really different needs for receiving email. I wanted to get it as fast as possible -or- refuse it as fast as possible.<br /><br />SMTP has a rich dialog for errors but I chose to only support one error message. And that error is, appropriately enough - "User Unknown". That's a touch ironic since Mailinator accepts any user at all. Simply said, if you do anything that the Mailinator server doesn't like - you'll get a user unknown error. Even if you haven't sent it the username yet.<br /><br />I looked at Apache James as a base which is a pure java SMTP server but it was way too comprehensive for my needs. I really just found some code examples and the SMTP specs and wrote things basically from scratch. From there, I was able to get an email, parse it, and put it right into memory. This bypassed the old system's step of writing it to disk all the way. From wire to user, mailinator mail never touches the disk. In fact, the Mailinator server's<br /><a href="http://www.amazon.com/gp/search?ie=UTF8&keywords=disk&tag=mailinator-20&index=pc-hardware&linkCode=ur2&camp=1789&creative=9325">disk</a><img src="http://www.assoc-amazon.com/e/ir?t=mailinator-20&l=ur2&o=1" width="1" height="1" border="0" alt="" style="border:none !important; margin:0px !important;" /> is pretty darn idle all things considered.<br /><br />Now to address persistence concerns right away - Mailinator doesn't run diskless, but it does run very asynchronously with regards to the disk. Emails are not written to disk EVER unless the system is coming down and is instructed to write them first (so it can reload them upon reboot). This little fact has been very handy when I've been subpoenaed. I simply do not have access to any emails that were sent to Mailinator in the past. If it is possible that I can get an email - so can you just by checking that inbox. If you can't get it then that means its long deleted from memory and nothing is going to get it back.<br /><br />Mailinator also used to do logging (again, shut-off because of pesky subpoenas). But it did it very "batchy". It wrote several thousand logs lines to memory before doing one disk write. In effect we never want to have contention based on the incredibly slow disk.<br /><br />Now if this all sounds a bit shaky, as in we might just lose an email now and then - you're right. But remember, our goal is 99.99% accuracy. Not 100%. That's an important distinction. The latest incarnation of Mailinator literally runs for months unattended. We do lose emails once in awhile - but its rare and usually involves a server crash. We accept the loss and by far most users never encounter it.<br /><br /><b>Emails</b><br /><br />The system now is one unit. The web application, the email server, and all email storage run in one JVM.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://photos1.blogger.com/x/blogger/1228/848/1600/694219/zzz1.jpg"><img style="float:right; margin:0 0 10px 10px;cursor:pointer; cursor:hand;" src="http://photos1.blogger.com/x/blogger/1228/848/320/569451/zzz1.jpg" border="0" alt="" /></a><br />The system uses under 300 threads. I can increase that number but haven't seen a need as of yet. When an email arrives (or attempts to arrive) it must pass a strong set of filters that are described below. If it gets past those filters it is then stored in memory - however, it is first compressed to save in-memory space. Over 99% of emails that arrive are never looked at, so we only ever decompress an email if someone actually "looks" at them.<br /><br />Because of this, I am able to store many more emails than the original system's 20000. The current mailinator stores about 80000 emails and uses under 300M or ram. I probably should increase this number as plenty of ram is just sitting around. The average email lifespan is about 3-4 hours with this pool. The amount of incoming email has gone way up, so even by increasing this pool, we're largely staying steady as far as email lifespan. I could probably kick that up to 200,000 or so and increase the lifespan accordingly but I haven't seen a great need yet.<br /><br />Another inherent limit that the system imposes is on mailboxes themselves. Popular mailboxes such as joe@mailinator.com and bob@mailinator.com get much more email than average. Every inbox is limited to only 10 emails. Thus, popular boxes inherently limit themselves on the amount of email they can occupy in the pool. Use of popular inboxes is discouraged anyway and generally become the creme de la cesspool of spam.<br /><br />Two more<br /><a href="http://www.amazon.com/gp/search?ie=UTF8&keywords=memory&tag=mailinator-20&index=pc-hardware&linkCode=ur2&camp=1789&creative=9325">memory</a><img src="http://www.assoc-amazon.com/e/ir?t=mailinator-20&l=ur2&o=1" width="1" height="1" border="0" alt="" style="border:none !important; margin:0px !important;" /> conserving issues is that no incoming email can be over 100k and all attachments are immediately discarded. That latter feature was in years ago but obviously really ruins this whole new wave of image spam (if you see a few seemingly "empty" emails in some popular boxes, they might have been image spam that got their images thrown away).<br /><br /><b>Spam and Survival</b><br /><br />I'd like to emphasize here that Mailinator's mission is NOT to filter spam. If you want penis enlargement or sheep-of-the-month club emails, that's pretty much what Mailinator is good for. We are clear in the FAQ. Mailinator provides pretty good anonymity - but we do NOT guarantee it. We also do NOT guarantee ANY privacy. Its really easier that way for us. Still, it does a pretty damn good job even so. We might log you (used to and it might get turned on again someday, never know) and we DO respond to subpoenas (that whole "jail" thing is a strong motivator).<br /><br />So, in essence I have no real interest in filtering out spam. I do however, have a great deal of interest in keeping Mailinator alive. And spammers have this nasty habit of sending Mailinator so much crap that this can be an issue. So - Mailinator has a simple rule. If you do anything (spammer or not) that starts affecting the system - your emails will be refused and you may be locked out.<br /><br />In the new system I created a data structure I call an AgingHashmap. It is, as it indicates a hashmap (String->int) that has elements that "age".<br /><br />The first type of spammer I encountered was one machine blasting me with thousands of emails. So, now, every time an email arrives, its senders IP is put into an AgingHashmap with a counter of 1. If that IP does not send us anymore email for (let's say) a minute, then that entry automatically leaves the AgingHashmap. But, let's say that IP address sends us another email 2 seconds later. We then find the first entry in the AgingHashmap and increase that counter to 2. If we see another email from that IP, it goes to 3 and so on. Eventually, when that counter reaches some threshold we ban all emails from that IP for some amount of time.<br /><br />We can put this in words as so (values are examples):<br />If any IP address sends us 20 emails in 2 minutes, we will ban all email from that IP address for 5 minutes. Or more precisely, we will ban all email from that IP until it stops trying to send to us for at least 5 minutes.<br /><br />This is really what the AgingHashmap is good for. We can setup some parameters and detect frequency of some input, then cause a ban on that input. If some IP address sends us email every second for 100 days straight, we'll ban (or throw away) every last email after the first 20.<br /><br />Here's a graph of an average 24 hours of banned IP address emails. Notice at 10am and 11am some joker (i.e., some single IP address) sent us over 19000 emails per hour.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://photos1.blogger.com/x/blogger/1228/848/1600/815054/badip.jpg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;" src="http://photos1.blogger.com/x/blogger/1228/848/400/201852/badip.jpg" border="0" alt="" /></a><br /><br />I do have some code that has Java talk back to unix's iptables system to do very hard blocking of IP addresses but its not on right now. Partially because there's no need (yet) and partially because I like to see the stats.<br /><br />The funny part of this is the error Mailinator gives. Remember the "User Unknown"? Once an IP address is banned and then it tries to open a new connection it will send the SMTP greeting of "HELO". Mailinator will then reply "User Unknown" and close the connection. Of course, it didn't even get the username yet.<br /><br /><b>Zombies</b><br /><br />The next problem came from zombie<br /><a href="http://www.amazon.com/gp/search?ie=UTF8&keywords=networks&tag=mailinator-20&index=pc-hardware&linkCode=ur2&camp=1789&creative=9325">networks</a><img src="http://www.assoc-amazon.com/e/ir?t=mailinator-20&l=ur2&o=1" width="1" height="1" border="0" alt="" style="border:none !important; margin:0px !important;" />. Now we were getting spam from thousands of different IPs all sending the same message. We could no longer key in on IP address. As a layer of defense past IP we created an AgingHashmap based on subjects. That is, if we get (again, example numbers) something like 20 emails with the same subject within 2 minutes, all emails with that subject are then banned for 1 hour.<br /><br />Here's a similar graph. Keep in mind these emails got past the IP filter - so basically they are "same subject" emails from many disparate sources.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://photos1.blogger.com/x/blogger/1228/848/1600/410953/subjban.jpg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;" src="http://photos1.blogger.com/x/blogger/1228/848/400/853950/subjban.jpg" border="0" alt="" /></a><br /><br />You could argue we should ban them forever, but then we'd have to keep track of them and the Mailinator system is inherently transient. Forgetting is core to what it does. This blocking is more expensive than IPs as comparing subjects can be costly. And of course, we have to have enough of a conversation with the sending server to actually get the subject.<br /><br /><b>Pottymouth!</b><br /><br />Finally, we ran into some issues on emails that just weren't cool. As I said, I'm far more interested in keeping Mailinator alive than blocking out your favorite porn newsletter. But, some unhappy people used Mailinator for some really not happy purposes. Simply put, as a last layer, subjects are searched for words that indicate hate or crimes or just downright nastiness.<br /><br /><b>Boing</b><br /><br />Another major influx that happened early on was a plethora of bounce messages. Now thats sort of odd isn't it? I mean Mailinator doesn't send email. In fact, it CAN'T send email so how could it get bounce messages? Well, some spammy type folks thought it'd be neat to send out spam from their servers using forged Mailinator addresses as a return address. Thus when those emails bounced, the bounce came here.<br /><br />What's worse, is I still get email from people who think Mailinator sent them<br /><a href="http://www.amazon.com/gp/search?ie=UTF8&keywords=spam&tag=mailinator-20&index=pc-hardware&linkCode=ur2&camp=1789&creative=9325">spam</a><img src="http://www.assoc-amazon.com/e/ir?t=mailinator-20&l=ur2&o=1" width="1" height="1" border="0" alt="" style="border:none !important; margin:0px !important;" />. Its very frustrating to defend myself against people who are ignorant of how email works ready to crucify me for sending them spam (especially ironic is that I run a free, anti-spam website). As I've said in my FAQ - please feel free to add mailinator.com to the tippy tippy tippy top of your spam blacklists. If you EVER get an email from mailinator.com, its a forged spam.<br /><br />The good news is that bounces are very easy to detect, and are really the first line of our defense. Bouncing SMTP servers aren't particularly evil, they're just doing their job so when I say "user unknown" they believe me and go away.<br /><br />On an abstract level, here is what happens to an email as it enters the system.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://photos1.blogger.com/x/blogger/1228/848/1600/990134/zzz3.jpg"><img style="float:right; margin:0 0 10px 10px;cursor:pointer; cursor:hand;" src="http://photos1.blogger.com/x/blogger/1228/848/320/132541/zzz3.jpg" border="0" alt="" /></a><br /><br />(and to be fair, there might just be another layer or two thats not on that diagram!)<br /><br /><b>Anti-Spam revolt</b><br /><br />There are 2 more, somewhat conflicting features of the Mailinator server that should be noted. For one, its a clear fact that when we're busy, we're busy. An easy DoS against us would be to open a socket to our server and leave it open. This is an inherent vulnerability in any server (maybe especially multithreaded servers). So, as a basic idea Mailinator closes all connections if they are silent for more than a second or two. Actually, the amount of time is variable (read below). Clearly, we are DoS'able by sending us many many connections, but this blocks at least one trivial way of bringing us down.<br /><br />Secondly, although we demand servers talking to us are very speedy. We reserve the right to be very NOT speedy. Here's the logic. When Mailinator is not terribly busy, we still demand responses quickly, but we give responses slowly. In fact, the less busy we are, the slower we give responses. It is possible that sending an email into the Mailinator SMTP server could take a very long time (like 10 or 20 or 30 seconds) even for a very small amount of data.<br /><br />Why? Well.. think about it. Let's say you're spamming. You want to send out a zillion emails as fast as possible. You want every receiving SMTP server to get your email, deliver it to the poor sod who wants (or doesn't want) weener enlargement and then close the connection so you can go on to the next. If you encounter some darn SMTP server that takes 20 seconds to receive your email, the speed at which you can send out your emails diminishes. You might just even think about avoiding such SMTP servers.<br /><br />It might be a pipe dream to think this is slowing down any spammers, but this does tend to keep my quieter times lasting longer. And it doesn't really hurt me - or my users. And if we eventually get terribly busy, those delays are scaled down to make sure we don't lose any emails.<br /><br /><b>Sites will ban it</b><br /><br />Every time I read some comment about Mailinator, someone always points out something like "Yeah, well sites will start banning any email from Mailinator and then it will be worthless". Guys. Its been 3 years. A handful of sites have indeed blocked email from Mailinator, but my user base and the number of read emails has only gone up. Clearly, people are finding Mailinator more useful than ever.<br /><br />I have added at times additional domains (like sogetthis.com and fakeinformation.com) that point to mailinator. Often if a site bans mailinator.com proper, you can use one of those to same effect.<br /><br /><b>Overall</b><br /><br />Many copycat sites have appeared over the years which is pretty reasonable. This idea itself is obvious. The only real hurdle was that it seemed impossible to do given the amount of useless email you'd get. But the copycats had the advantage of seeing that Mailinator actually does work, so they knew what to shoot for. Only a few post their daily email numbers but I've yet to see any that come close to mailinator's incoming email (not that this is necessarily a good thing). I also see that many are using an architecture similar to Mailinator's original which is just fine so long as they either don't get any massive increases in email or are happy to keep buying bigger hardware.<br /><br />Overall, <a href="http://www.mailinator.com">Mailinator</a> has been a great experience. It was a terribly fun exercise in optimization, security, and generally making things work. Thousands of people use it everyday and its amazing how many people know about it when it comes up in conversation. I've thought many times about how to make a business around it, and there is always an angle, but I've just been to busy with other things.<br /><br />My hope is that its useful for you and that you tell your friends.<br /><br /><a href="http://digg.com/submit?phase=2&url=mailinator.blogspot.com/2007/01/architecture-of-mailinator.html&topic=tech_news"><br /><img src="http://digg.com/img/badges/180x35-digg-button.gif" width="180" height="35" alt="Digg!" /><br /></a><div style="clear:both; padding-bottom: 0.25em;"></div></p><div style="clear:both; padding-bottom: 0.25em;"></div></p><br /><br /> <br /><br /><i><br />Note: Eternal thanks to Jack Lawrence of Syracuse, NY who, in a drunken stupor gave me the core idea (story <a href="http://iggychaos.blogspot.com/2006/02/idea-about-ideas.html">here</a>), Nicci Gabriel of <a href="http://www.sideofsauce.com">www.sideofsauce.com</a> for the seriously cool web design, and to Brian Pipa of <a href="http://www.candyaddict.com">www.candyaddict.com</a> who, as a big fan of Mailinator, added the very cool "Spam Map" and the RSS feeds.<br /></i>paulhttp://www.blogger.com/profile/11412172362500455307noreply@blogger.com46