I couldn’t help but note that Eric’s posted about the technology powering Convore, and guess what little library is listed there?  That’s right, Django.

…

Seriously though, it’s nice to see someone using such a seemingly-oldschool stack to build a cutting-edge application and not being apologetic about it at all.  And this is a great illustration of the sweet spot for Eventlet: quickly creating a novel scalable network application.

Install it by one of these methods:

• easy_install eventlet
• pip install eventlet
• http://pypi.python.org/packages/source/e/eventlet/eventlet-0.9.14.tar.gz
• hg clone http://bitbucket.org/which_linden/eventlet/

To file issues/bugs:
http://bitbucket.org/which_linden/eventlet/issues/new/

Here’s the changes in this version:

• Many fixes to the ZeroMQ hub, which now requires version 2.0.10 or later. Thanks to Ben Ford.
• ZeroMQ hub no longer depends on pollhub, and thus works on Windows (thanks, Alexey Borzenkov)
• Better handling of connect errors on Windows, thanks again to Alexey Borzenkov and Nat Linden (#67, #69).
• More-robust Event delivery, thanks to Malcolm Cleaton
• wsgi.py now distinguishes between an empty query string (“”) and a non-existent query string (no entry in environ).
• wsgi.py handles ipv6 correctly (thanks, redbo)
• Better behavior in tpool when you give it nonsensical numbers, thanks to R. Tyler for the nonsense. :)
• Fixed importing on 2.5 (#73, thanks to Ruijun Luo)
• Hub doesn’t hold on to invalid fds (#74, thanks to Edward George)
• Documentation for eventlet.green.zmq, courtesy of Ben Ford

The md5sum for the tarball is dfc96ed14b27392fdc529abcafeed880
The sha256sum is
219207d165176b12565e7f8f40aaadb2965814ce5935dba1097083d0f183bf28
(that’s right, I totally went there!)

How get it:

• easy_install eventlet
• pip install eventlet
• Download directly from pypi
• hg clone http://bitbucket.org/which_linden/eventlet/

File issues/bugs here.

Full changelog:

• ZeroMQ hub, and eventlet.green.zmq make supersockets green. Thanks to Ben Ford! Check out the examples zmq_chat.py and
  distributed_websocket_chat.py.
• eventlet.green.MySQLdb added. It’s an interface to MySQLdb that uses tpool to make it appear nonblocking
• Greenthread affinity in tpool. Each greenthread is assigned to the same thread when using tpool, making it easier to work with
  non-thread-safe libraries.
• Eventlet now depends on greenlet 0.3 or later.
• Fixed a hang when using tpool during an import causes another import. Thanks to mikepk for tracking that down.
• Improved websocket draft 76 compliance, thanks to Nick V.
• Rare greenthread.kill() bug fixed, which was probably brought about by a bugfix in greenlet 0.3.
• Easy_installing eventlet should no longer print an ImportError about greenlet
• Support for serving up SSL websockets, thanks to chwagssd for reporting #62
• eventlet.wsgi properly sets ‘wsgi.url_scheme’ environment variable to ‘https’, and ‘HTTPS’ to ‘on’ if serving over ssl
• Blocking detector uses setitimer on 2.6 or later, allowing for sub-second block detection, thanks to rtyler.
• Blocking detector is documented now, too
• socket.create_connection properly uses dnspython for nonblocking dns. Thanks to rtyler.
• Removed EVENTLET_TPOOL_DNS, nobody liked that. But if you were using it, install dnspython instead. Thanks to pigmej and gholt.
• Removed _main_wrapper from greenthread, thanks to Ambroff adding keyword arguments to switch() in 0.3!

The md5sum of the tarball is 2273527ca1715bad018090fdd2d480cf.

Using signals is kind of appropriate for this sort of functionality, because signals are part of a sorta old-school way of implementing concurrency.  The nice thing about them versus, say, a watchdog thread, is that the Python interpreter really interrupts itself in the middle of whatever it was doing when the signal comes in, so the handler can print the lowest-level line of code that was blocking.

* Eventlet no longer uses the Twisted hub if Twisted is imported — you must call eventlet.hubs.use_hub(‘twistedr’) if you want to use it. This prevents strange race conditions for those who want to use both Twisted and Eventlet separately. * Removed circular import in twistedr.py
* Added websocket multi-user chat example
* Not using exec() in green modules anymore.
* eventlet.green.socket now contains all attributes of the stdlib socket module, even those that were left out by bugs.
* Eventlet.wsgi doesn’t call print anymore, instead uses the logfiles for everything (it used to print exceptions in one place). * Eventlet.wsgi properly closes the connection when an error is raised * Better documentation on eventlet.event.Event.send_exception
* Adding websocket.html to tarball so that you can run the examples without checking out the source

Enjoy!

The MD5 of the tarball is: 96edb9eb4f73863f3d19afb5d5c85854

0.9.10
======
* Greendns: if dnspython is installed, Eventlet will automatically use it to provide non-blocking DNS queries. Set the environment variable ‘EVENTLET_NO_GREENDNS’ if you don’t want greendns but have dnspython installed. * Full test suite passes on Python 2.7.
* Tests no longer depend on simplejson for >2.6.
* Potential-bug fixes in patcher (thanks to Schmir, and thanks to Hudson) * Websockets work with query strings (thanks to mcarter)
* WSGI posthooks that get called after the request completed (thanks to gholt, nice docs, too)
* Blocking detector merged — use it to detect places where your code is not yielding to the hub for > 1 second (this could use documentation) * tpool.Proxy can wrap callables
* Tweaked Timeout class to do something sensible when True is passed to the constructor

I hope you are having a sensational week!

Thus far, Chrome seems to be the major test browser for WebSockets (though Safari implements them as well), and they switched over to draft 76 a few weeks ago.  So the Chrome betas speak draft 75, and the nightlies speak 76.

Fortunately, it’s possible to write WebSocket servers that interoperate seamlessly with both versions, by detecting those new Sec-Websocket headers and doing the appropriate handshake for the appropriate version.  Andrew Godwin implemented this logic for Eventlet’s websocket module, so you can use it to serve sockets to any browser that’s out there today.  Find it in the development branch, which you can install by running easy_install eventlet==dev.

We’ll be keeping a weather eye on the spec as it proceeds.

So here’s the full changelist:

• A fix for monkeypatching with psycopg versions < 2.2.
• Improved support for chunked transfers in wsgi, plus a bunch of tests from schmir (ported from gevent by redbo)
• A fix for the twisted hub from Favo Yang

Enjoy!

• Support for psycopg v2.2′s async support, courtesy of Daniele Varrazzo’s great work. You use it by calling monkey_patch(psycopg=True).
• Websocket support moves to core. This means it’s easier than ever to use websockets with eventlet.wsgi. It’s, like, two lines now. Mad props to Ben Ford for providing 100% unit test coverage. Check out the example.
• eventlet.wrap_ssl exists! We talked about it a while back but never got around to implementing it until now.

Give it a go, see how you like it!

Here’s the full changelog:

• Support for psycopg2′s asynchronous mode, from Daniele Varrazzo
• websocket module is now part of core Eventlet with 100% unit test coverage thanks to Ben Ford. See its documentation at http://eventlet.net/doc/modules/websocket.html
• Added wrap_ssl convenience method, meaning that we truly no longer need api or util modules.
• Multiple-reader detection code protects against the common mistake of having multiple greenthreads read from the same socket at the same time, which can be overridden if you know what you’re doing.
• Cleaner monkey_patch API: the “all” keyword is no longer necessary.
• Pool objects have a more convenient constructor — no more need to subclass
• amajorek’s reimplementation of GreenPipe
• Many bug fixes, major and minor.

Eventlet meets Avro RPC in an unlikely (in my opinion) place: WSGI. Instead of building their own transport layer for RPC calls, Avro sits on top of HTTP for its transport layer, POST’ing binary data to the server and processing the response. Since Avro can sit on top of HTTP, we can use eventlet.wsgi for building a fast, simple RPC server.

I’ve never used Avro before, but its supporters seem to really like it.  It seems to have improved upon Protocol Buffers (while preceding them chronologically), and has a very efficient wire encoding.

subscribe

Categories

• Apture
• Cheetah
• Git
• Hudson
• JavaScript
• Miscellaneous
• Opinion
• Python
• Software Development

continuous blog

• Hudson 1.355 Released
• A new blog for Kohsuke
• Hudson 1.354 Released

more

Recent Comments

• oto kiralama You’ve managed to explain a really tricky subject well. I find that I sometimes have difficulty in getting my head round topics like this, but you’ve summed it up really well. I’ve found another…

  A Review: The Metamorphosis · 1 day ago

• R. Tyler Ballance You can consider it public domain for all I care, if there’s enough interest I can probably create a proper GitHub project for it :)

  Hudson notifications with libnotify · 1 week ago

• plinehan Great hack. Thanks!

  Reliable Locks in Hudson · 1 week ago

• alberto Can you put a license to this code? I would like to make some improvements to it. :)

  Hudson notifications with libnotify · 1 week ago

github activity

• rtyler created branch master at rtyler/eventlet-avro-example
• rtyler created repository eventlet-avro-example
• rtyler commented on hudson/Hudson-GIT-plugin
• rtyler started following fzzzy
• rtyler pushed to master at rtyler/Spawning

more

youtube activity

• Python on Hudson (Part 1)

more

upvoted

• …when they demanded that he choose between them, he told them that they would have to fight it out themselves, at which point the two women began to wrestle. Picasso described it “as one of his choicest memories”.
• Voyager 2 is experiencing very odd anomalies at the edge of the Solar System
• This is what I think of when I think of the Mission.
• Hundreds of SF felony convictions to be potentially overturned as dozens of cops’ prior criminal records were witheld from defense attorneys

more

Home » Blogs » R. Tyler Ballance’s blog

How-to: Using Avro with Eventlet

May 7, 2010 – 8:45am | by R. Tyler Ballance

Working on the plumbing behind a sufficiently large web application I find myself building services to meet my needs more often than not. Typically I try to build single-purpose services, following in the unix philosophy, cobbling together more complex tools based on a collection of distinct building blocks. In order to connect these services a solid, fast and easy-to-use RPC library is a requirement; enter Avro.

---

Note: You can skip ahead and just start reading some source code by cloning my eventlet-avro-example repository from GitHub.

---

Avro is part of the Hadoop project and has two primary components, data serialization and RPC support. Some time ago I chose Avro for serializing all of Apture’s metrics and logging information, giving us a standardized framework for recording new events and processing them after the fact. It was not until recently I started to take advantage of Avro’s RPC support when building services with Eventlet. I’ve talked about Eventlet before, but to recap:

Eventlet is a concurrent networking library for Python that allows you to change how you run your code, not how you write it

What this means in practice is that you can write highly concurrent network-based services while keeping the code “synchronous” and easy to follow. Underneath Eventlet is the “ greenlet” library which implements coroutines for Python, which allows Eventlet to switch between coroutines, or “green threads” whenever a network call blocks.

Eventlet meets Avro RPC in an unlikely (in my opinion) place: WSGI. Instead of building their own transport layer for RPC calls, Avro sits on top of HTTP for its transport layer, POST’ing binary data to the server and processing the response. Since Avro can sit on top of HTTP, we can use eventlet.wsgi for building a fast, simple RPC server.