Jason Rush: Dev

Last year, I started working on a JRuby port of the Google Wave robot API called Rave. I took a bit of a hiatus from working on Rave, since the Google Wave API was a bit unstable, and I had been spinning my wheels trying to solve some things that turned out to be unsolvable... Anyway, the Wave API has stabilized a bit, so I'm back at it. And I have some help - a couple of people have been contributing code to the project. Which is, by the way, awesome.

Anyway, I'll post more on Rave pretty soon. In the mean time, check it out on Github, for more info. Particularly the README.

I'm hanging out at JRubyConf today, and I got all inspired to contribute something to the community. So I decided to put together a little plugin for auto-loading jars into a JRuby on Rails project. One of the key advantages of using JRuby instead of MRI is the availability of all of the mature Java libraries. In JRuby, you have the ability to easily wrap those libraries and access them directly from your Ruby code. The process is pretty easy, you just require each jar that you want to use, just like you'd require a gem, or another .rb file. But there is no consistent pattern for this -- no convention for where to place those files. What I wanted was to have a jars folder under vendor where you could just drop jars and have them automatically required in your app. The end result is a ridiculously simple plugin I've imaginatively called Vendor Jars.

To install it, just cd to the top-level folder of your rails app and run

script/plugin install git://github.com/diminish7/vendor_jars.git

This will add the plugin to your vendor/plugins folder, and it will create a vendor/jars folder. Then, for any Java jar that you want to be accessible from your JRuby code, just drop the jar into that vendor/jars folder, and it will automatically get required at startup.

That's it. I know it's not much (it's like 4 lines of code or something) but I feel like I write that same code every time I do a new JRuby app, so, simple as it is, I think it'll be useful.

I've had a frustrating couple of weeks working on Rave. Rave is a Ruby port of the Python API for building robots in Google Wave. Wave is pretty unstable, and so I keep running into problems where I'll spend hours debugging my code only to determine that it's a problem with Wave. So I'm going to set it aside for a little while. In about a month, Google is going to release new versions of everything, and start letting the general public sign up for Wave accounts. I assume that they are planning on having a more stable version of Wave at that point. So, I'll revisit it then. As it stands, Rave is in a pretty good spot for building a lot of types of robots. The two big things I've been fighting with recently are that the capabilities versioning doesn't work correctly (so Wave caches your robot's capabilities and you can't add or remove capabilities in new versions), and cron events don't work. I'll revisit both of those in October once a more stable version of Wave is out. Until then, I need to stop spinning my wheels and start focusing on some side projects that I can make some actual progress on.

Google Wave
There's been a lot of talk about Google Wave lately, so what's Wave all about? Wave is a protocol, that extends XMPP. In practice, it contains elements of email, instant messaging and threaded discussion. Oh, and document management. Yeah, it seems to do just about everything. The problem with that is that it makes it really hard to describe. It always sounds like it is trying to be everything - to solve every problem. But really, it's all about online collaborative communication. Here, take a look at the demo from the Google I/O keynote this year. It makes more sense once you've seen it in action.

Wave is in an alpha release right now. As far as I know, only the attendees of Google I/O have accounts on the sandbox. Hopefully that will change before too long. Google's only saying that it will be released "before the end of the year" right now. In the mean time, there's a ton of information on the main Wave site, the wave protocol site, and the Google blog.

Some Terminology
There are three primary components to Google Wave: waves, wavelets and blips. Waves are the central component. Think of a wave as a conversation. A wave has one or more wavelets, which are sort of like sub-conversations, or threads of discussion under a conversation. Finally, blips are the actual messages in the discussion, so a wavelet can contain one or more blips. There's a better description of these terms in the protocol spec.

Wasn't This Post Supposed to be About Robots?
Here's what's really cool about Google Wave: there's an API for building extensions for it. There are three things that a developer can do with Wave: embed a wave on a website, build gadgets to be embedded in a wave, and build robots. There's a discussion on each of these on the Wave API site. My focus here is going to be on robots, because let's face it, robots are awesome.

Robots are automated participants in a wave. They can do basically anything that a human participant can do: they can read the conversation, add to the conversation, and they can edit a conversation. So you can, for example, build a spell checker that reads everthing you post to a wave and edits it to change typos as you type. There are some cool examples of things you can do with robots in the Google I/O keynote demo.

Google Wave Robot Tools
Google released two toolkits for building robots, one in Java, and one in Python. Furthermore, there is currently a restriction in place that requires robots to live on Google App Engine. So that means that a robot has to either be in Python or Java... or some other language implemented on the JVM. Happily, for those of us who prefer to code in Ruby, we've got JRuby. Now all we need is an implementation of the robot client API in Ruby.

Rave: A Google Wave robot client framework for Ruby
There are several parallel efforts to create a Ruby implementation of the robot API. The rest of this post is going to focus on Rave, but I just wanted to link to the other implementations that I'm aware of:

• Sam Ruby's Wave Robot Ruby Client
• Jack Danger's Wave
• Mike Sofaer's Wave Robot Sinatra Template

Warning!
Both Rave and Google Wave are super-alpha. There are a lot of things in the protocol that aren't implemented in Rave yet. And the protocol is changing, so the things that are implemented are likely to break at some point. I will try to keep up, but if you notice anything broken, or if you have a need for something that isn't implemented yet, shoot me an email: diminish7 at gmail dot com.

Okay. You've been warned.

The Basics
Since Google currently requires that all robots run on App Engine, this tutorial assumes you are using JRuby. You should have JRuby already installed, and you should have the App Engine SDK installed. You can find more information on JRuby at http://jruby.org/, and you can find more information about the App Engine SDK at http://code.google.com/appengine/downloads.html

What We're Going to Build
I hate it when people yell in writing. I want a robot that will moniter discussions and tone down any yelling. So, for example, this robot will turn a post like "I'M NOT YELLING!!!!" into "I'm not yelling." The full source for this robot is in git under examples/appropriate-casey/. Here's how you build it yourself:

Install Rave

   sudo jruby -S gem install rave

Start a new Rave Project
Rave comes with a "rave" executable that, among other things, sets up the project structure for you. Setting up a new project looks like this:

   jruby -S rave create [robot_name] [options]

The options include profile_url and image_url, which will set the URL for the robot's profile and avatar, respectively. Here's how the example project "appropriate casey" was set up:

   jruby -S rave create appropriate-casey

I haven't bothered to create an avatar for casey, but if I had, I would instead have done:

  jruby -S rave create appropriate-casey image_url=http://myimageurl/avatar.png

This will stub out a project called "appropriate-casey". It automatically creates a robot class AppropriateCasey, creates the config files for both Rack and Warbler, and creates the appengine-web.xml file that App Engine will need. For the App Engine file, it assumes that the App Engine project name is the same as the robot's name. So in this case, the application name is appropriate-casey, which means I have to have the URL http://appropriate-casey.appspot.com. If you name your robot something different than the App Engine application ID, just change the line in appengine-web.xml.

Build your robot!
The robot.rb file that Rave created contains a class that extends Rave::Models::Robot. All of the logic needed for your robot to talk to App Engine is included in Rave::Models::Robot, so all you really need to do now is define your robot's event listeners. Here is a list of the events that Google Wave can send your robot:

• WAVELET_BLIP_CREATED
• WAVELET_BLIP_REMOVED
• WAVELET_PARTICIPANTS_CHANGED
• WAVELET_TIMESTAMP_CHANGED
• WAVELET_TITLE_CHANGED
• WAVELET_VERSION_CHANGED
• BLIP_CONTRIBUTORS_CHANGED
• BLIP_DELETED
• BLIP_SUBMITTED
• BLIP_TIMESTAMP_CHANGED
• BLIP_VERSION_CHANGED
• DOCUMENT_CHANGED
• FORM_BUTTON_CLICKED
  

To add a listener to your robot that will respond to a given event, just define a method with the lower-case version of the event name, that accepts an event and a context. So, I want appropriate-casey to do something whenever a document is changed (the DOCUMENT_CHANGED event), so I add the following method to my robot:

   def document_changed(event, context)
      # Do some stuff
   end

The Event object contains the following properties:

• type (the type of event, "DOCUMENT_CHANGED" in this case)
• timestamp (the timestamp that the event occurred)
• modified_by (the user ID of the user who modified the document)
• properties (varies by event type, but this will be a range of affected text in this case)
  

The Context object contains the following properties:

• waves (a hash - the keys are the wave IDs, and the values are the waves)
• wavelets (a hash - the keys are the wavelet IDs, and the values are the wavelets)
• blips (a hash - the keys are the blip IDs, and the values are the blips)
• operations (an array of operations)

See the Google Wave protocol documentation for definitions of each of these things...

So, with Appropriate Casey, I want to create a robot that looks for people "shouting" in Waves. To do this, I just create a document_changed method in my robot that looks something like this:

   def document_changed(event, context)
     unless event.modified_by == "appropriate-casey@appspot.com" || 
            event.modified_by == "spelly@gwave.com"
        context.blips.values.each do |blip|
          if blip.content
            new_sentence = true
            blip.content.length.times do |index|
              range = index..index+1
              char = blip.content[index, 1]
              if char =~ /[A-Z]/ && !new_sentence
                blip.set_text_in_range(range, char.downcase)
              elsif char =~ /[a-z]/ && new_sentence
                blip.set_text_in_range(range, char.upcase)
              elsif char == "!"
                if new_sentence
                  blip.delete_range(range)
                else
                  blip.set_text_in_range(range, ".")
                end
              end
              new_sentence = (char =~ /\.!?/ || (char =~ /\s/ && new_sentence))
            end
          end
        end
      end
   end

For simplicity's sake, I'm brute-forcing this a bit. It would be better to look at the event properties and just change the range of text that the event is telling me about.

So that's it, my robot is now ready to go: just 17 lines of code.

Packaging for deploy
I'm assuming that you have an application set up on App Engine already. Again, for the time being, all robots have to be on App Engine to work with Google Wave.

First of all, we need to turn our project into the correct format for App Engine. There is a utility
in the "rave" executable for this, so from your robot's top-level folder, run:

   jruby -S rave war

This is basically just a wrapper around the warbler gem, but with some additional cleanup to get things in the right format for App Engine. For example, the complete JRuby jar is too large for App Engine, so Rave breaks it up into two jars. Once you've run "rave war", you'll see a tmp/ folder, and a .war file. You can ignore the .war file, as App Engine needs the unpacked version. The tmp/war folder is what will get deployed.

Testing
The worst part about App Engine is testing...

Rave includes a "server" command that starts up Rack for your application, but that is of limited use, since App Engine has many App Engine-specific issues. Better to use the App Engine SDK (which is still not a perfect match to the deployed environment, but it's better). The App Engine SDK comes with a dev_server command. From your project directory run:

   path/to/appengine_sdk/bin/dev_server tmp/war

This will start the development server on port 8080. You can now hit the following three URLs:

• http://localhost:8080/_wave/capabilities.xml
• http://localhost:8080/_wave/robot/profile
• http://localhost:8080/_wave/robot/jsonrpc

The first two are GETs, but the last one requires a POST. See the Google Wave protocol for the expected body of the POST if you want to test locally.

Deploying
Use the App Engine SDK to deploy. From your project directory run:

  path/to/appengine_sdk/bin/appcfg update tmp/war

Using the Robot
From a Google Wave client, start a new wave, and invite your robot as a participant. Your robot's user name will be [robot_name]@appspot.com. So Casey is at appropriate-casey@appspot.com.

The End
That's it. You now have a working robot! Enjoy!

We recently open-sourced Moonstone, our JRuby wrapper around Lucene, the Java search engine toolkit. Moonstone is simple, and easy to use - it lets you put together a powerful search engine quickly. It gives you the full capabilities of Lucene, but replaces the Java-isms with a more Ruby-ish syntax. I wanted to take a quick minute to run through the basics:

Installation

    sudo jruby -S gem install moonstone

Note that if you are on Windows, drop the "sudo". Also, for the record, no one's ever tested Moonstone on Windows. But since it's all JRuby and Java, there shouldn't be any issues...

If you want the source, you can get it from automatthew's github repo.

Engines

Creating an engine requires two things: a doc_from method and an analyzer.

  class SimplestEngine     def doc_from(record)
      doc = Lucene::Document::Doc.new
      doc.add_field('name', record[:name])
      doc.add_field('desc', record[:description])
      doc
    end
   def analyzer
     @analyzer ||= Lucene::Analysis::WhitespaceAnalyzer.new
   end
 end    
 # We can now instantiate the engine
 engine = SimplestEngine.new
 # and create an index of some data
 records = [
     {:name => "Moonstone", :description => "A simple JRuby wrapper around Lucene"},
     {:name => "Foo", :description => "A search engine based on Moonstone"}
   ]
 # By convention the index method accepts an Enumerable that yields hashes
 engine.index(records)

 #Search for data
 q = Lucene::Search::TermQuery.new("name", "Moonstone")
 documents = engine.search(q)
 documents.each {|d| puts d['name']}

That's it in a nutshell. There are some more examples in the README on github, and more documentation is coming soon.

Happy searching!

I started up a new project to create a template engine for Ruby web apps. The beginnings of it are on github here. It's very, very rough, so don't judge yet. :-)

I know there are a ton of template engines out there. But here's my complaint with them: often the person developing the front end of an app is primarily an HTML/CSS/JavaScript developer. They may not know Ruby at all. So template engines like Markaby and Hoshi are out. ERB is the most HTML-like template engine, but a) it's still got Ruby mixed into it, so it isn't ideal for strictly front-end developers, and b) it's a mixture of Ruby and HTML, which is ugly and spaghetti-ish.

Enter Beholder.

Beholder is a template engine comprised of HTML tags. Dynamic behavior is accomplished through the use attributes. So, for example, if you have a page and you want to show a chunk of it conditionally, you could do something like this:

<html>
<head>
<title>Hello World</title>
</head>
<body>
<span component="if" condition="true">
Body of 'if' component
</span>
<span component="elsif" condition="true">
Body of 'elsif' component
</span>
<span component="else">
Body of 'else' component
</span>
</body>
</html>

Which would be rendered as:

<html>
<head>
<title>Hello World</title>
</head>
<body>
Body of 'if' component
</body>
</html>

The 'component' attribute can either refer to a built in control flow structure, such as 'if', or 'foreach', or it can refer to a partial, so you can embed mini-beholder templates in your main template. There is also a 'yield' component, so each component can act like a layout, generating it's HTML, and then yielding the content within the component node in the template.

In addition, any attribute can be made dynamically by including the 'prop:' prefix. So if you have a template like this:

<html>
<head>
<title>Hello World</title>
</head>
<body>
<span component="simple_component" message_class="beholder" />
</body>
</html>

And a partial called simple_component.beh like this:

<p class="prop:message_class">What's up, world?</p>

It would be rendered like this:

<html>
<head>
<title>Hello World</title>
</head>
<body>
<p class="beholder">What's up, world?</p>
</body>
</html>

The 'prop:' prefixed attributes get evaluated as (in this order) a) the value of the parent component's attribute with that name, b) the value of a local variable of that name, or c) a method in the helper class. (Note that I've only implemented a) right now... so you'll just have to trust me that that's where I'm going.)

This is modeled after Tapestry templates (Java) and Kid templates (Python). Anyway, there's a lot of work to be done on it (particularly around the evaluation of 'prop:' attributes), but I like the idea of it a lot.

I've been thinking a bit about template engines in Ruby. There's a ton of them out there. Here's a good review of a bunch of them. We generally use either Hoshi or Markaby at my job right now. For our purposes, these have been great. We all work on all aspects of a project, from front to back, and we'd all rather write Ruby than HTML. However, in several jobs I've had, we've had a front-end developer doing all the HTML and CSS. The problem is that people who really specialize in front-end development often don't know Ruby. So Hoshi and Markaby are out. In those situations, we've used ERB templates. But then you've got Ruby and HTML mixed together, and that's never pretty. What I'd really like is a template engine that just looks like HTML, with no Ruby mixed into it. Something along the lines of Python's Kid, or Java's Tapestry templates. I've been looking around a bit and haven't found anything like that yet in Ruby. Sounds like time to spin up a new project! More to come...

The two position-based logic games (broadcast and piano_instructor) are working correctly now, with all rules and facts coming from the DSL, including the rules inferred by the combination of other rules.

It ain't pretty though... The inference of rules is very non-generic, and very spaghetti code-ish. I really just wanted to get the logic straight in my head first. I'll look at pretty-ing it up next. But it works! Better to have ugly, but working code today, right?

Made a bit more progress on the Logic Games project today. I have all of the rules for both the broadcast game and the piano instructor game set up generically, so that all of the base rules are specified in the DSL. Here's an example, taken from the broadcast game:

new_game
called "Broadcast"
described_as("Seven consecutive time slots... etc")

with_property "Position"
with_range 1, 7
for_entities "G", "H", "L", "O", "P", "S", "News"

new_question
described_as "If G is played 2nd... etc"
with_fact "G", "Position", :is, 2
determines ["News", "H", "L", "O", "S"], "Position", :is, 3

#L must be played immediately before O
new_rule "L", :before, "O", 1
#The news tape must be played at some time after L
new_rule "News", :after, "L"
#There must be exactly two time slots between G and P, regardless of
# whether G comes before P or whether G comes after P
new_rule "G", :separated_by, "P", 2

evaluate

That generates almost all the facts and rules for the game. There are a few rules that are implied by the combination of these rules - since "O" is immediately after "L", and "News" is sometime after "L", then "News" must be at least 2 after "L". So that's next: figure out how to imply rules from combinations of explicit rules.

I've started refactoring my logic games into a DSL that more closely approximates the English language logic games questions. Here's what I have so far:

#Set up game description
new_game
called "Broadcast"
described_as(
"Seven consecutive time slots for a broadcast, numbered in chronological order 1 through 7, will be filled by six song tapes - G, H, L, O, P, S - and exactly one news tape. Each tape is to be assigned to a different time slot, and no tape is longer than any other tape. The broadcast is subject to the following restrictions:

• L must be played immediately before O
• The news tape must be played at some time after L
• There must be exactly two time slots between G and P, regardless of whether G comes before P or whether G comes after P"
  

)
with_property "Position"
with_range 1, 7
for_entities "G", "H", "L", "O", "P", "S", "News"

#Create the questions
new_question
described_as "If G is played second, which one of the following tapes must be played third?"
with_fact "G", "Position", :is, 2
determines ["News", "H", "L", "O", "S"], "Position", :is, 3

This replaces the first 70 lines of the game. It sets up the game, and adds all the standard rules, like one-place-at-a-time rules, mutual-exclusion rules, and last-available-position rules.

My goal was to make this read more or less like the English version of the game. One interesting requirement for that has been that it needs to remember context. So if I say "new_question" on one line, then "described_as ..." on the next line, the game needs to remember that "described_as" applies to the new question we just created. I'm doing this by maintaining state. So each action keeps a record of the thing it acted upon, so that the next action knows what the context is. The alternative was to string together actions, like "new_game.described_as(description).with_fact(...)" etc. I didn't like that approach because it looks less like English, which was, again, my goal.

The next step in the DSL will be a simple way to specify the rules. I want it to be close to the English version of the rule, so

"Henry's lesson is later in the schedule than Janet's"

might turn into something like

new_rule "Henry", "Position", :after, "Janet"

Which will then get broken up into the 20 or so rules that that implies behind the scenes.

The Logic Games project now officially solves its first game! It's admittedly a pretty manual process right now. I still have to break the logic game up into all the initial rules and facts. But then it successfully runs through the rule engine and generates all the facts that can be inferred, and answers the question.

Here's the game it solves right now:

Seven consecutive time slots for a broadcast, numbered in chronological order 1 through 7, will be filled by six song tapes - G, H, L, O, P, S - and exactly one news tape. Each tape is to be assigned to a different time slot, and no tape is longer than any other tape. The broadcast is subject to the following restrictions:

• L must be played immediately before O
• The news tape must be played at some time after L
• There must be exactly two time slots between G and P, regardless of whether G comes before P or whether G comes after P
  

If G is played second, which one of the following tapes must be played third?

• The news
• H
• L
• O
• S

If you grab the code from github, and run

ruby logic_games.rb games

it will correctly tell you L is in the 3rd position.
Yay!

It was interesting to me that there were a lot of rules that I was using intuitively when I solved the problem by hand, but that I neglected to add to rule base because I never explicitly acknowledged the rule in my head. An example of this is the rules and facts generated by the combination of the fact that L must be played immediately before O, and the News must be played some time after L. What that implicitly means is that the News must be at least 2 slots after L, because the slot immediately after L is taken up by O. These types of rules will be the hardest to generalize, because they need to take external knowledge (previous statements) into account.

So the next step is to start generifying the logic for creating rules and facts. I would like to come up with some sort of DSL for specifying knowledge in a way that approximates the written game statements. So, for example, to specify that L must be played immediately before O, you might have something like

"L".before("O", 1)

which would translate into "The entity with the name of 'L', must be positionally before the entity named 'O', by 1 position." This would then generate the 20 or so rules that this implies, like "If L is in position 1, then O is in position 2", etc.

So, to accomplish this, I'm going to work on implementing several other games so I can start getting a feel for the types of generalizations I'll need to implement. Then I'll go from there.

Good times.