@@ -12,6 +12,8 @@ API functionality. As of July 2015, the FiveStreet team runs a
1212SuperIMAP cluster processing ~ 400k emails per day for thousands of
1313users.
1414
15+ SuperIMAP is written in Ruby and open sourced under the MIT license. [ Why Ruby?] ( #why-ruby )
16+
1517## Contents
1618
1719* [ Installation] ( #installation )
3638 * [ Manage a User] ( #apiv1connectionsimap_provider_codeuserstag )
3739* [ Operations] ( #operations )
3840 * [ Process Types] ( #process-types )
41+ * [ Why Ruby?] ( #why-ruby )
3942 * [ Environment Variables] ( #environment-variables )
4043 * [ Scaling] ( #scaling )
4144 * [ Monitoring] ( #monitoring )
@@ -369,11 +372,11 @@ updating the user (ie: a PUT request.
369372
370373#### Process Types
371374
372- SuperIMAP consists of 3 different processes:
375+ SuperIMAP consists of 3 different processes, all written in Ruby / Rails :
373376
374- + ' web' - Serves the admin interface and the API.
375- + ' imap_client' - Handles the task of connecting to IMAP providers and listening for email.
376- + ' worker' - Processes background jobs generated by the 'imap_client' process.
377+ + ` web ` - Serves the admin interface and the API.
378+ + ` imap_client ` - Handles the task of connecting to IMAP providers and listening for email.
379+ + ` worker ` - Processes background jobs generated by the 'imap_client' process.
377380
378381#### Environment Variables
379382
@@ -429,6 +432,24 @@ Apart from keeping an eye on these metrics, SuperIMAP should need no other regul
429432
430433You may also want to keep an eye out for any failing Delayed Job tasks. You can view these from the Admin site.
431434
435+ #### Tracer Emails
436+
437+ SuperIMAP has the ability to give you useful monitoring information
438+ through "tracer emails". The system will send a specially formatted
439+ email to an account, wait for the incoming email, and log the
440+ results. The logs can be accessed through the "Tracer Logs" tab.
441+
442+ To enable Tracer Emails, navigate to a user and check the "Enable
443+ Tracer" checkbox. It is recommended that you create a few dummy email
444+ addresses to use for tracer emails.
445+
446+ By default, a cluster of three tracers are sent every ten minutes from
447+ each ` imap_client ` instance to a random tracer-enabled user managed by
448+ that instance.
449+
450+ Keep in mind that this could generate a lot of email. Three emails
451+ every ten minutes works out to ~ 430 emails per day.
452+
432453#### Performance
433454
434455SuperIMAP's architecture makes judicious use of system resources:
@@ -464,23 +485,62 @@ users requires just 10 database connections, uses about 3GB of RAM,
464485and has a 0.50 load average. The work queue usually sits near 0, with
465486a latency of < 0.5 seconds.
466487
467- #### Tracer Emails
468-
469- SuperIMAP has the ability to give you useful monitoring information
470- through "tracer emails". The system will send a specially formatted
471- email to an account, wait for the incoming email, and log the
472- results. The logs can be accessed through the "Tracer Logs" tab.
473-
474- To enable Tracer Emails, navigate to a user and check the "Enable
475- Tracer" checkbox. It is recommended that you create a few dummy email
476- addresses to use for tracer emails.
477-
478- By default, a cluster of three tracers are sent every ten minutes from
479- each ` imap_client ` instance to a random tracer-enabled user managed by
480- that instance.
481-
482- Keep in mind that this could generate a lot of email. Three emails
483- every ten minutes works out to ~ 430 emails per day.
488+ #### Why Ruby?
489+
490+ At first glance, and from a purely technical point-of-view, Ruby is a
491+ poor choice for an application like SuperIMAP. SuperIMAP is highly
492+ concurrent, and Ruby is bad at concurrency.
493+
494+ Specifically, the ` imap_client ` process spans what could technically
495+ be described as a "boatload" of threads (2 threads per connected user,
496+ plus a handful of other threads). Ruby threads are heavyweight, so the
497+ interpreter has to burn significant resources just to create and
498+ schedule the threads before it can do any real work.
499+
500+ Using Erlang, Go, or Rust (all of which support lightweight threads
501+ and actor-style programming) would have made the concurrent bits of
502+ SuperIMAP less tricky to write, and would have required fewer
503+ computing resources, possibly allowing a single box to handle tens of
504+ thousands of active users.
505+
506+ So, why Ruby? A few reasons:
507+
508+ + ** FiveStreet uses Ruby** - The primary goal of SuperIMAP is to power
509+ a critical part of FiveStreet's application. FiveStreet is built and
510+ maintained by a small team of Ruby engineers. The entire FiveStreet
511+ application is written in Ruby. Adding a language would force the
512+ team to spend dozens of hours learning a new stack and maintaining a
513+ new development environment.
514+
515+ + ** Low barrier to entry** - A secondary goal of SuperIMAP is to become
516+ a healthy open-source project. Based only on language popularity, it
517+ is more likely that another team can use, troubleshoot, and contribute to
518+ a Ruby-based SuperIMAP than an Erlang/Go/Rust-based SuperIMAP.
519+
520+ + ** The concurrency is not complicated** - The concurrency in SuperIMAP is
521+ fairly straightforward -- one parent process, many child
522+ processes. It's a little painful to solve the problem in Ruby, but
523+ not impossible.
524+
525+ + ** For us, the cost savings are small** - FiveStreet's SuperIMAP
526+ cluster currently runs on three commodity servers and happily handles
527+ thousands of users. It's possible that if SuperIMAP were written in
528+ a different language, we could handle the load on a single machine,
529+ saving us a few hundred dollars a month. Not worth changing our
530+ stack for it.
531+
532+ + ** Ruby has mature IMAP and OAuth 2.0 libraries** - Ruby has a
533+ [ built-in IMAP library] ( http://ruby-doc.org/stdlib-2.0.0/libdoc/net/imap/rdoc/Net/IMAP.html ) ,
534+ and a
535+ [ widely-used OAuth 2.0 library] ( https://github.com/intridea/oauth2 ) . As
536+ of 2015, the IMAP and OAuth 2.0 libraries for other languages are
537+ [ far] ( https://github.com/alexcrichton/oauth2-rs )
538+ [ less] ( https://github.com/boorad/erlimap )
539+ [ mature] ( https://github.com/mxk/go-imap ) .
540+
541+ Side Note: This was a deeply considered choice. I (Rusty Klophaus, the
542+ author of SuperIMAP) spent about 4 years writing Erlang
543+ professionally. It's a fascinating language
484544
485545## Appendix
486546
0 commit comments