Recursive

ExtJS Custom Templates, data-binding and interactivity

2012-08-11T00:00:00-07:00

This is the second part in a series of articles on the strengths and weakness of Sencha Touch and ExtJS MVC. Part 1 shows the general approach for writing re-usable robust code This article delves into the weaknesses of this approach, particularly when using ExtJS custom templates. You might want to read this great complimentary article by a different author first.

sticking point: custom templates

Productivity in ExtJS comes from composing existing widgets. However, I find myself with many custom templates.

    Ext.define 'Yap.view.Feed',
        extend: 'Ext.view.View'
        itemSelector: 'div.entry'
        config:
            cls: 'feedlist',
            store: 'Feeds',

        tpl: Ext.create('Ext.XTemplate', """
          <tpl for=".">
            <div class="feed-entry">
              ...
            </div>
          </tpl>
          """)

It is nice that I can populate this view by putting data in a store, so I am happy up until the point the user needs to interact with my view. The standard way of mapping a user action to a controller function is to use the ref system in the controller. The problem is that this means changes to the view break the controller. Lets see an approach of referencing just the widget in the ref system:

    Ext.define "Yap.controller.Controller", Sencha.controllerCompatibility
      extend: "Ext.app.Controller"
      views: [ 'View' ],

      events:
        feed: 'start'

      config:
        control:
          view:
            itemtap:   'itemTapToPress'
            itemclick: 'itemClickToPress'

      itemTapToPress: (_1, _2, _3, record, e) -> @onItemPress(record, e)
      itemClickToPress: (_1, record, _2, _3, e) -> @onItemPress(record, e)

      onItemPress: (record, e) ->
        elTarget = Ext.get(e.target)
        
        if (elTarget.is('.add'))
          alert('add')
        
        else if (elTarget.is('.comment'))
          this.onComment(record, 'comment')
        
        ...

An alternative that I would prefer is to actually listen for events in the view itself and then fire up events to the controller. This can be done in Sench Touch, but not in ExtJS. It does not solve the problems outlined, but at least all responsibility for view changes exists in one file.

Note that the above code does work in both Sencha Touch and in ExtJS by normalizing the difference between click and tap. Since there are multiple elements that can be clicked, we are forced to know about the DOM to determine where the click came from.

I could certainly use an onclick handler to fire an event, but I also want to automatically fire up the record of data as above. The best I can really do is fire an id and then lookup the record from its id. But this is just read-only information. After all of this effort, I still cannot modify the view without writing code that knows about the DOM. We have these limitations because custom ExtJS templates are dumb strings.

Components, not templates

If instead of using custom templates we use ExtJS components we can achieve dom manipulation from controllers that have no knowledge of the DOM or the components. Lets take a look at a function that creates such a button component:

    eventedButton = (event, text) ->
      # make sure that button gets pressed from an event fire
      # uses our button override
      # for example this allows the application to start with a button pressed by firing an event
      listener = {}
      listener[event] = -> this.doToggle() if !this.pressed

      {
        ui: 'link'
        text: text
        handler: -> Yap.fire(event)
        eventListeners: listener
        allowDepress: false
        enableToggle: true
      }

Our component fires an event when clicked and listens for the same event. The benefit of firing and listening to the same event is that when the page is re-loaded a controller will restore the application state by firing an event corresponding to the button previously pressed. But the point is that a controller can simply fire an event, and the DOM will end up changing.

In a view, I have something like the following for the buttons:

      xtype: 'toolbar'
      ui: 'plain'
      cls: 'viewport-main-toolbar'
      items: [{
        xtype: 'textfield'
        width: 180 
      }
      ,   
        eventedButton 'event1', 'Text1'
      ,
        eventedButton 'event2', 'Text2'
      ]

I can turn these buttons into a full tabs implementation if I have a component named event1 and event2.

      Yap.UI.activateTab :: (Str) -> Any
      Yap.UI.activateTab = (tabname) ->
        unless tabnames = Yap.UI.registeredTabs[tabname]
          throw "unregistered tab: #{tabname}"
        for other_tabname in tabnames
          unless other_tabname == tabname
            Ext.getCmp(other_tabname).hide()
        Ext.getCmp(tabname).show()

      Yap.UI.registeredTabs = {}
      Yap.UI.registerTabs = (tabnames) ->
        for tabname in tabnames
          Yap.UI.registeredTabs[tabname] = tabnames

      Yap.UI.registerTabs ['event1','event2']

This is great, we have isolated components both properly updating their state and communicating with the world through events. but the data we are using to create multiple components is static. This brings us back to our custom templates that use dynamic data from a store and are automatically updated when a store changes.

So we can see a solution to data-binding in ExtJS: avoid custom templates and figure out how to hookup arbitrary components to the datachange of a store. But we can’t write HTML, we have to use just components for any element that requires interaction. And I don’t feel like figuring out the code to make this work: at this point I would be doing a lot of work that a framework is supposed to be doing for me and I have to work with a new abstraction instead of plain html.

ExtJS is powerful because of its re-usable components, but custom templates with interaction means we are not hitting the sweet spot of the framework. I see a few ways forward:

as mentioned, discover a way to use just components, and never any templates
enhance data binding with a small add-on like rivets.js. Find another add-on or build a library to enhance view-controller communication to be DOM independent.
Use an entirely different framework in place of custom templates.

Stay Tuned!

A follow up article on solving these issues by injecting an entirely different framework into ExtJS.

Robust, reusable Sencha Touch and ExtJS code

2012-08-06T00:00:00-07:00

Summary

We can make non-view code compatible between Sencha Touch & ExtJS by developing a compatibility layer specialized for our application. Events help de-couple controllers from views, allowing controllers to be re-used, and they can also be adapted to automatically restore application state.

Why Sencha Touch & ExtJS

The motivation for using these frameworks is to have managed re-usable UI components.

Sencha Touch really takes advantage of this to provide widgets that look native, are efficiently created and destroyed, and use full-screen layouts. Sencha Touch is well optimized for mobile and is one of the few proven mobile web development toolkits.

ExtJS has some powerful components like a grid view and a tree view. Once you are using Sencha Touch, ExtJS becomes attractive for re-using code.

We are getting ready to release some applications that use these frameworks. We do not yet have much man power behind these efforts (but you can help remedy that), so we need to use these frameworks in productive ways.

Writing robust and re-usable Sencha Touch/ExtJS

There is a big toolkit at your disposal and multiple ways to write code with the same functionality. What I am looking for are best practices that will lead to robust and flexible code that is re-usable.

Code re-use between frameworks

These frameworks have a lot of API compatibility. Sencha Touch has many advancements not yet present in ExtJS. My understanding is that internally ExtJS underwent big changes for 4.0, some of which were to pave the way for compatibility with Sencha Touch. Unfortunately, the performance of the new release was unexpectedly poor. Now much of the development effort is focused on improving ExtJS performance and API compatibility with Sencha Touch keeps getting delayed. So to achieve compatibility we need to take matters into our own hands.

True universal compatibility might be an impossible task. But If I just tackle the compatibility issues that my application needs, it does not seem that hard to achieve.

Models and Stores

Models and Stores are the most compatible between the two frameworks. The main difference appears to be that a lot of functionality that you would specify at the top level of an ExtJS model. I have one function that will do a proper translation called Sencha.modelCompatibility. Note that the code samples given in this blog are in coffee script.

    Ext.define("Yap.model.Country", Sencha.modelCompatibility({
      extend: "Ext.data.Model"
      config:
        idProperty: "_id"
        fields: [
            { name: '_id',                type:'int'     }
            { name: 'code',               type:'string'  }
            { name: 'name',               type:'string'  }
    }))

Here are our definitions, note there is one for Store also:

    window.Sencha = (->
      isTouch = !!Ext.getVersion("touch")?.version?.match(/2\./)
      {
        isTouch: isTouch
        isExtJS: !isTouch
      })()

    Sencha.modelCompatibility =
      if Sencha.isTouch then (x) -> x else
        (classConfig) ->
          if cfg = classConfig.config
            if fs = cfg.fields
              classConfig.fields = fs
              delete cfg['fields']
          classConfig

    Sencha.storeCompatibility =
      if Sencha.isTouch then (x) -> x else
        (classConfig) ->
          if classConfig.config
            for lifted in ['model','proxy']
              if val = classConfig.config[lifted]
                classConfig[lifted] = val
                delete classConfig.config[lifted]
          classConfig

That was pretty easy!

Controller/View compatibility

I have never tried to make a view compatible between the 2 frameworks. Mobile & desktop require different views most of the time. Even when the views are similar they are generally using components that do not easily translate between the 2 frameworks.

This brings us to controllers. While models and stores need properties to be moved under config, some of the controller aspects need some involved translation.

    Sencha.controllerCompatibility =
      (classConfig) ->
        compatibleRefs = (classConfig) ->
           if touchRefs = classConfig.config?.refs
              classConfig.refs =
                for ref, selector of touchRefs
                  {ref:ref, selector:selector}
              delete classConfig.config['refs']

        classConfig = makeEventHistory(classConfig)
        if !Sencha.isTouch
          compatibleRefs(classConfig)
          classConfig.getApplication = -> @application

          if classConfig.config?.control
            oldInit = classConfig.init
            classConfig.init = ->
              if @config.control
                control = {}
                for selector, eventDispatch of @config.control
                  dispatch = {}
                  for event, fn of eventDispatch
                    dispatch[event] = this[fn]
                    undefined
                  control[selector] = dispatch
                  undefined
                console.log(control)
                @application.control(control,undefined,this)
                delete @config['control']

              if oldInit
                oldInit.apply(this, arguments)
              else
                @callParent(arguments)

        classConfig

The code is almost all the code we needed to re-use code between our applications. Our API calls use the same code and go through the same Ext.data.JsonP.request function for both frameworks. We have a few overrides elsewhere and a few if Sencha.isTouch, statements, but this makes it possible to re-use everything but views. makeEventHistory in the code sample is an enhancement that leads us to the discussion of robust code.

Controllers, Views, and robust code re-use

We have already stated that view-reuse is a non-starter. That create a big problem for controllers because they are often written in a way that deeply couples them to the view. I do not think there is something fundamentally wrong with this approach, but one just needs to figure out how to still re-use code. One appraoch would be to have small view-controllers that know about the view and then communicate to other re-usable (controller) code. I think this is similar to what deftJS (an ExtJS/Sencha Touch extension) does.

My issue with this approach and with the normal controller approach is that the view references are inherently fragile and a poor way to communicate between views and controllers. Sencha Touch:

    refs: [
        {
            'mydiv' : 'div.my'
        }
    ],

    control: [
      mydiv: {
        onTap: 'myTapFn'
      }
    ]

Views change frequently, and every change risks silently breaking a DOM query used by both the controller DOM references and their associated control functions. Even if we have 100% test coverage and are confident we can correct all breakages, DOM queries are an unnecessarily complex approach to communicating with a view.

DOM queries are an indicator that the javascript framework is not high-level enough.

Fortunately we have a great tool at our disposal for communicating between the view and the controller: events.

Communicating through events

Our controllers and our views can listen for and fire events. We can create a controller that has no knowledge of the DOM by firing an event from the view.

    Ext.define "Yap.view.stuff",
      extend: "Ext.dataview.List"
      config:
        store        : 'Stuff'
        listeners:
          select: (list, record) -> Yap.app.fireEvent('choose-item', record)

I was very happy with this code. On the controller side we fill up a store with data, render the view, and then listen for an event that gives us the selection back. We have achieved 2-way data-binding, and we can swap out different views and re-use our controller between Sencha Touch and ExtJS. This Sencha Touch view (some more config options emitted) is producing a full-screen list that looks native, no styling required on my part.

changing the url hash based on events

Developing a UI becomes very tedious when a long sequence of clicks are required to reach the point in the UI that is being worked on. Since I am already firing events, I set the events as the url hash. Refreshing the browser fires the latest event, getting me close to the point in the UI that I was previously at. There is a limitation to this: a refreshed event must be able to recover all the state required. So I register all events that work with this principle: other events will not change the hash.

In controller I just register an event and a callback:

    Ext.define "Yap.controller.Controller", Sencha.controllerCompatibility
      extend: "Ext.app.Controller"
      events:
        start: 'onStart'

The above mentioned makeEventHistory processes this:

    # enhancement, not compatibility
    #
    # in config, specify an event map in the format of the normal config routes key
    #
    # The events will be listened to (getApplication().on), and the location hash will be changed to match the event.
    # Now add a catch-all route that fires events from the location hash.
    # This makes debugging a lot easier: refresh your browser to get back to where you were.
    # But this also means you need to figure out what to do if a fired event expects data.
    makeEventHistory = (classConfig) ->
      if events = classConfig.config?.events
        throw("did not expect before") if classConfig.before

        # whenever an event is triggered, change the hash location
        # Conveniently, a page refresh takes us back to where we were.
        # However, if there was data with the event, on refresh it will be empty
        hist_ev_map = {}
        for ev of events
          hist_ev_map[ev] = (_1,_2,event) ->
            window.location.hash = '!/' + ((event || _2).info?.eventName || ev)
          undefined

        # There is probably a way to do this without overriding init
        oldInit = classConfig.init
        classConfig.init = ->
          @getApplication().on hist_ev_map
          events.scope = @
          @getApplication().on events
          if oldInit
            oldInit.apply(this, arguments)
          else
            @callParent(arguments)

        delete classConfig.config['events']
      classConfig

Now we just need to use the url hash on startup. In an initialization function we need something like this:

    Yap.fire(window.location.hash.slice(3) || 'default')

I created this to ease my development, but I am keeping it in for our production applications. Most users won’t notice most of the time, but if the refresh their browser or click the back button it should put them close to where they were.

Stay Tuned!

In the next post in this series I discuss the limitations of this approach of communicating through events.

Forward Authentication

2012-07-23T00:00:00-07:00

The magic of forwarded authentication

I have been using ssh for a long time, and on my local system I have passwords on all my RSA keys. I use ssh-agent to not have to type my password multiple times a day.

In production (AWS) most transfers are between instances and S3 but every now and then we need to transfer information in between instances. This is generally awkward, for security reasons our ssh private keys are not installed in production.

Enter authentication forwarding. It is enabled by passing the -A switch to open ssh, here is the man page in all it’s glory.

Enables forwarding of the authentication agent connection. This
can also be specified on a per-host basis in a configuration file.

Agent forwarding should be enabled with caution. Users with the
ability to bypass file permissions on the remote host (for the
agent's UNIX-domain socket) can access the local agent through the
forwarded connection. An attacker cannot obtain key material from
the agent, however they can perform operations on the keys that
enable them to authenticate using the identities loaded into the
agent.

Example

An example will help clarify the use case. Let’s say you can connect from your dev system to server-1 and server-2. You need to move big files between the two production systems and you could achieve simply by using a command like the following on server-2:

scp -rC server-1:src dest

Unfortunately there is no ssh key setup between the two servers. The solution is simple, from your dev system execute:

ssh -A user@server-2 'scp -rC server-1:src dest'

This will take your key on dev and make it available on server-2 for the duration of that command. Your key does not actually leave your system, rather the authentication related operations are proxied back from server-2 to dev.

Put Your Apron On

2012-07-11T00:00:00-07:00

Invisible deployments

At yap.TV we are fans of continuous deployments, we want to be able to deploy to production anytime code is ready.

Our current setup

We deploy our code on EC2 and use Chef in a solo chef configuration.

At this time we are doing deployments in place with versioning and rollback. We deploy a new versions of our applications on the existing servers. We use various versioning mechanisms depending on the technology (ruby, scala, ejabberd). We have the ability to revert to earlier versions but we had seldom needed to do that over the last years.

ELB based deployment

For a long time we have used Amazon’s Elastic Load Balancers (ELB) in front of our API servers. The deployment mechanism involves:

remove a server from the ELB
run the chef recipes
run local tests to make sure that all the services have restarted and reconnected to shared resources in the system
add back the server in the load balancer

We have a script to automate most of the steps however that script is still driven by a human.

Limitations

There are a couple of limitations with this approach. First of all the ELBs rudely drop connections as soon as you remove an instance from the load balancer, see Eric Hammond’s thread . This means that with the approach above you will drop the existing API connections. This is mitigated by the fact that we have smart clients that retry connections, however we would like to eliminate the cases in which we knowingly drop connections.

Second of all it is hard to continue to automate this approach. Right now we have multiple AWS keys using different privilege levels through IAM. The production servers have read only access to the EC2 information however they do not have write access. We have been very reluctant to expose write access to the management infrastructure.

Enter Apron

Apron is series of scripts that keep your Chef from getting dirty during the deployment process. It is doing this by routing traffic away from the system being updated without requiring access to the management infrastructure. We present the basic mechanisms here but the exact integration will depend on your specific setup.

Apron pushes the routing logic to the local system being deployed. Each system has a buddy, this buddy is defined in a configuration file when running chef. Each server routes traffic to its buddy before starting deployment and then accepts traffic when it is done. This uses the built in command iptables. We have this working on Natty and Pangolin in production.

There are 3 simple scripts that need to be integrated in your deployment process. Before deployment the local server should start to deploy traffic to his buddy using apron-on.sh.

After the deployment and local testing you want to re-accept traffic using apron-off.sh

Should things go wrong you can look at things with apron-show.sh

Next steps

We are looking for ways to make the system more robust. The deployment needs to happen sequentially and we may leverage other existing mechanisms to enforce that, for example Zookeeper

Additionally we will enhance our monitoring script to detect failures. One option is to expose real IP addresses in the service monitoring phases and report mismatches between the server we connect to and the server actually serving the page.

Javascript Frameworks And Data Binding

2012-06-11T00:00:00-07:00

UI Widgets

One approach to frameworks is to try to find the one with the best set of UI widgets that fit your needs. UI widgets have powerful abstractions, but at some point we need to extend them, compose them, or create entirely new widgets, so I am looking for a foundational framework that a UI can be built with from the ground up or used to integrate with UI widgets.

jQuery

jQuery and other similar libraries largely insulate the programmer from browser inconsitencies and add convenience for DOM and event interaction. It is possible to build a UI with only jQuery, but in my experience this is not a productive option because jQuery apropriately concerns itself with only the lowest level of abstractions. JQuery plugins and UI Widgets tend to be very opaque data containers that are tied to the DOM. It can be difficult to communicate with the data they are holding or otherwise extend them for new use cases. jQuery is not an MVC framework, although you can try to use it in that style.

MVC frameworks

MVC frameworks crucially separate out the data from the UI. Data is accesible in the models rather than being retrieved directly from the DOM. This is crucial for an interactive UI because the display of data (always a string) is different than the actual data. When data changes we need to interact with the real data, not a display of it. Traditionally we might have interacted with the data by querying the server for even a minor update. However, a much more responsive application can be built if we have access to the data client side and can optimistically make changes in the UI before ever hearing back an OK from the server.

Backbone.js

The original and very popular javascript MVC framework is Backbone.js. I have used Backbone.js on a few projects now, and I believe it is definitely better to use Backbone than to not use it. The good part of Backbone is that it structures your data into models and collections separate from (but bound to) the UI with good REST integration. However, in my experience the UI integration is not good enough. There is too much boilerplate from its lack of 2-way data binding. In Backbone you have to wire your templates to re-render when data changes. You also have to retrieve your data back out of your forms to put it in your models/collections. This boilerplate data binding is tedious for the experienced, but also ends up being a stumbling block for less advanced programmers that lets them write bad code.

Let me explain the point about less advanced programmers. Normally I am of the mindset that you need to focus on hiring better and teaching your team how to code better. However, in a client heavy interactive app we may have UI focused programmers. They may be capable of creating great UIs but do not always create the best engineering quality code. This could be because they are an experienced CSS designers that just recently learned javascript. But it can also be because they care deeply about the end result, and the code that it takes to get there is not as interesting to them. There are probably good UI programmers that would be great to have on your team, but that you could struggle with to guide them to create high quality code (code that is understandable, low defect, and easy to extend in the future). In my opinion, Backbone asks for too many unecessary decisions and boilerplate code from inexperienced programmers that can easily lead to low quality code.

Knockout / Knockback

Knockout is one of the original frameworks with good 2-way data binding. Knockout is a bit lacking as a feature complete framework. There is a project called Knockback.js that brings in rich models from Backbone.js. However, while Knockback seems feature complete it is a heavyweight in size and there is still some boilerplate to declare powerful 2-way data bindings.

more frameworks with 2-way data binding

The remaining framework options I know of with 2-way data binding are then Angularjs, Ember.js, Batman.js, CanJS, and Serenade.js.

Another intriguing option is Derby.js and Meteor, which tries to bring data-binding all the way to the server side. Derby.js is designed to work with a server-side nodejs server, so although it may be possible to integrate with a non-node setup, I will wait for a clear demonstration of that before trying it out. Batman.js does target running on the server side also, but should work on the client side with other backends.

Event Binding

One difference in frameworks is their approach is event binding. CanJS uses CSS selectors for event binding:

var Todos = can.Control({
    'init' : function(){}
  },
  'li click' : function( li, event ) {
    console.log( 'You clicked', li.text() )
  }
})

This keeps logic out of the view, which has benefits. However, this means you have to make sure your css selector always matches the view html. For that reason, I prefer other approaches to event handling.

Templating

One difference between these frameworks is how they do templating. Angularjs, KnockoutJS, and Batman.js templates are valid html. Templating then involves using existing custom tags, setting tag attributes, and some creation of custom tags. I have personally never been a big fan of tag-based templating. However, for client-heavy templates it begins to make some sense. I used to think of templates as generated and read-only. Assign some values to some variable, stick them into your templates, its easy, why try and pretend we are using html? I still think this makes sense when you are writing a stateless webapp. However, when you begin to bind interactivity to templates you quickly run into new issues.

In Ember.js actions can be bound in the template:

<a href="#" {action "edit" on="click"}>Edit</a>

An Ember View can also listen for events. The problem for me with Ember style views is that the basics need to abandon html to be concise. For example, Emberjs has its own checkbox function

{view Ember.Checkbox checkedBinding="todo.done"}

compared with angularjs (Batman.js and Knockout are similar)

<input type="checkbox" ng-model="todo.done">

The Emberjs version is not verbose. But the html version is easier to understand, especially for a designer because we only added the binding, we did not use any additional abstractions. I would like an ember-style version:

<input type="checkbox" {checked=todo.done}>

However, without knowledge of the tag, Ember-style templates probably need to special case checked to figure out the correct behavior. For server side templating this wouldn not matter: I could do something like this:

<input type="checkbox" {todo.done ? "checked=checked" : ""}>

There is no updating, so we can just make sure to get it right the first time. But for automatic 2-way data binding this will not work unless we have a construct to recognize which part of the expression is a data binding (todo.done), and which part is just for generating the proper html. So things quickly get more complicated unless we make the choice of abandoning html for functions or having our template language have knowledge about the DOM (a benefit of having templates that are valid HTML).

Another approach to templating is to use a thin abstraction over html to make writing HTML nicer. As an example, the Joosy framework defaults to using HAML. As a creator of the Hamlet templating language, I don’t like HAML-inspired syntax. Hamlet gets rid of closing tags and lets you insert values into your HTML. HTML is actually pretty good once you get rid of the closing tags and add a few conveniences. The additional abstractions beyond that makes us learn something new for no reason.

For my javascript templates, I first create them server-side (Yesod or Rails) with Hamlet. In addition to being a more convenient way to write HTML, it also lets me stick data into them server side if I find it more convenient. On top of that, I can still do client-side templating with something like Mustache, and obviously valid html templates of Angularjs/Knockout/Batman.js work very well. Serenade.js uses a css-selector style abstraction for its template languages, so it isn’t forcing you to learn anything new if you are used to CSS, just to use css selectors rather than tags.

2-way Data-binding by example

I am truly impressed with the revolutionary hassle-free data binding of Angularjs. You can use plain-old javascript attributes for bound data instead of get and set. Their boilerplate-free usage of dependency injection means your code will always be easy to test.

To see how pain free the AngularJS data binding is, lets look at some examples of computed properties. A computed property is made up of one or more other bound properties.

We want to have a full name in a view that automatically changes when the first or last name of the person changes:

<span> Person: {person.fullName()}</span>

Ember:

Person = Ember.Object.extend({
   firstName: null,
   lastName:  null,
   fullName: function() { return firstName + ' ' + lastName; 
       return this.get('firstName') + ' ' + this.get('lastName');
     }.property('firstName', 'lastName'); 
});

Batman (Coffeescript):

class Person extends Batman.Object
   firstName: null,
   lastName:  null,
   @accessor 'fullName', ->
     @get('firstName') + ' ' + @get('lastName');

Knockout:

  function AppViewModel(first, last) {
    this.firstName = ko.observable(first);
    this.lastName = ko.observable(last);
    this.fullName = ko.computed(function() {
      return this.firstName() + " " + this.lastName();
    }, this);
  }

AngularJS:

function Person(first, last) {
  this.first = first;
  this.last = last;
}
Person.prototype.fullName = function() {
  return this.first + ' ' + this.last;
};

Choosing a framework

The downsides of Angularjs is that it is a heavyweight in size and its REST and collection support is more immature than what I am used to from Backbone.js.

If you are loooking for a small-sized framework, and you are not using jQuery already, Serenade.js may fit the bill, weighing in at only 9k with no dependencies. We should note that Serenade.js does not target IE6 support.

If you want to share code with a server side javascript application, Batman.js is designed for that. Knockback.js and Batman.js seem to have good collection/REST support and may be easier options when switching from Backbone.js.

Please understand that this is a very limited comparison. There are probably other frameworks out there with 2-way data binding. Getting data binding right or taking a particular approach to templating doesn’t mean a lot unless there is good documentation on many other well-implemented features and capabilities. I think I have enough experience to know there must be something better than Backbone, but I have little practical experience with the alternatives. One nice way to compare frameworks is the TodoMVC project.

Elegant Xml Parsing

2012-06-08T00:00:00-07:00

Elegant XML Parsing in Ruby

I blogged in more detail about this before. But I saw a Stack Overflow question about this, so I feel I need to keep up the public service announcements :)

If you are parsing big XML files you have to stream them using something like SAX. Does your code look like this?

    def start_element tag, attributes
      case tag
      when 'name'
        @attribute = 'name'
      when 'titles'
        @attribute = 'name'
      end
    end

    def end_element tag
      @model.save! if tag == 'name'
    end

    def characters text
      @model[@attribute] = text
    end

Please stop doing this!

Sax Machine was created to have an elegant API for parsing XML.

    class AtomEntry
      include SAXMachine
      element :title
      # the :as argument makes this available through atom_entry.author instead of .name
      element :name, :as => :author
      element "feedburner:origLink", :as => :url
      element :summary
      element :content
      element :published
    end

This is declarative coding, which is what we should be striving for no matter what programming tools we are using. The problem is that the official sax-machine gem loads everything into memory. The reason why is that it does not use a callback (yield) or fibers to process elements when they are immediately loaded.

At yap.TV, we have been using my fork of sax-machine for over a year now on gigantic XML files. This fork uses Ruby’s fibers to transfer control from the xml parsing back to your code. This is similar in concept to how yield works, but you don’t have to pass in a block right away. Instead you get back an Enumerator that will end up parsing on demand.

    class Atom
      include SAXMachine
      element :title
      elements :entry, :lazy => true, :as => :entries, :class => AtomEntry
    end

    feed = Atom.parse(xml_file_handle, :lazy => true)
    feed.entries # => #<Enumerator: #<Enumerator::Generator:0x00000004c41ea0>:each> 
    feed.entries.each do |entry|
      # every time the block is called the next entry is parsed- no memory blow up! 
      # This is probably where you save the entry to a database
    end

The only thing extra you need to do is add a :lazy attribute to your top level elements. Also checkout ezkl’s fork. He is taking over maintenance of sax-machine in the official repo now though and planning on pulling in my changes.

This solves the memory part of performance. However, if this is not fast enough for you, I suggest switching the library to use ox. From the ox documentation:

Unlike Nokogiri and LibXML, Ox can be tuned to use only the SAX callbacks that are of interest to the caller

The current Sax parser will pull everything out of the XML and bring it into Ruby land. Avoiding this could make the parsing an order of magnitude faster depending on how sparsely you are using the XML.

For our usage at yap (importing to a database) we of course found that database access became a big bottleneck. We added some indexes and caching, and made some queries use the raw driver rather than go through an ORM.

sax-machine is a great example of how Ruby’s advanced features of fibers and meta-programming can be used. Please use it to write declarative code!

Scala On Rails

2012-06-07T00:00:00-07:00

Now that we have our blog going we wanted to collect information on what we have been doing. This post originally published on Konrad’s blog

Scala on Rails

The JVM is awesome.

But back to the topic of this post: We’ll see how you can leverage both Scala and JRuby in one project. Each will doing what it does best:

Ruby will handle the FAST frontend development as one of Ruby’s strong points is obviously Rails which made it so popular. Though there are people implementing APIs in ruby, if you need pure performance, you may need some other lang (vide airbreak, twitter…).
Scala will be doing all the heavy lifting including some of our existing code which interacts with zookeeper or uses Akka actors to crunch some numbers.

Another great point here is the reuse of our existing scala codebase which has some nice libraries we’ve implemented during our day-to-day. Anyway, let’s see some code!

Start out with a plain Rails 3.2 project You know the drill. rvm, ruby, rails new. The only difference here is that we’ll use JRuby right from the start here, so:

# get JRuby and start a new rails app
rvm install jruby && rvm use jruby
 
gem install rails
rails new scala-on-rails

JRubify it!

Next we’ll add some more dependencies to our Gemfile:

# jruby dependencies in Gemfile
gem 'jruby-openssl' # simply required
gem 'jruby-scala' # a jruby scala bridge
gem 'warbler' # to create a war file that we'll deploy layer

Most important here is the jruby-scala gem, which acts as a bridge to allow the use of scala constructs in ruby, more naturally. For a longre read on this check out Daniel Spiewak’s blogposts about it.

Now do the usual bundle install (be sure that you’re running jruby (!)). If you don’t want to rvm use jruby, you could also run bundler like this: jruby -S bundle install.

One thing I’m not covering in this blogpost is how to throw out ActiveRecord from rails. But that’s easily googlable: Just don’t require it, as shown here: Rails 3 without ActiveRecord. Of course you CAN use active record with JRuby etc, but in my project we won’t need it, so I’m removing it.

Scalify it!

Ok, now that we have all the gems we need let’s proceed with putting scala into our lib/ directory. We’ll need it at runtime obviously… :-)

# copy scala into your rails /lib directory
cd lib/
cp $SCALA_HOME/lib/scala-library.jar .

One way to include jar dependencies into your project is by simply pasting them into the lib/ directory, so if you need some cool-java-library.jar, just drop it in there! It’ll also be automatically packaged with the app into the warfile by warbler – yay!

Require ‘java’!

Kk, now on to some code, let’s assume I have a nice “number formatter” I’d like to use in my rails app, that I’ve already implemented in some “commons” library in Scala. (Obviously that’s not the only thing from scala we’re using in our app but it’s just an example class ;-)). Here’s how we’d write our controller:

# app/controllers/AnyController.rb
require 'java'
require 'scala-library.jar'
require 'scala-cool-formatting.jar'

# here's how to obtain a nice name for something in the Java namespace
PrettyOrdinalNumber = Java::pl.project13.common.format.PrettyOrdinalNumber

class AnyController < ApplicationController
 
  def index
    n = 12
    nth = PrettyOrdinalNumber.new(n) # the constructor in Scala actually takes a Long – but no worries about types, JRuby can handle it
 
    @whoa_from_scala = "the number #{n} is the #{nth.ordinal} natural number!"
  end
end

By the way, PrettyOrdinalNumber is a trait in scala. There we use it via implicit conversions to nicely print numbers when counting servers etc ;-) A small thing but nice to have.

The view here is boring – we just print this one variable; So I won’t even put a code block here for it ;-)

Run it!

We still have not launched the app… Does it work at all? It does – but let’s check. The nice thing here is… You can still simply run the app via the good ol’…

# rails s works as you'd expect it to
rails s

command. (Just be sure you’re running JRuby.) Tests and everything just works as you’d expect it to - no worries here. One thing worth mentioning here is that currently assets pipeline may not be working with JRuby (or maybe it does but I didn’t care so I just generate assets before I build the war file). So you’ll want to disable it in production like this (in config/enrivoments/production.rb):

# Precompile assets to be included in the WAR file
config.assets.compile = true

Then in order to pre-generate assets (compile scss -> css, et cetera) you can run:

# precompile assets like this
RAILS_ENV=production bundle exec rake assets:precompile

Which will generate all assets and throw them into public/assets/. Oh and don’t panic if this takes a long time - yup, sometimes it does.

Ship it!

Now that we’ve got our app running let’s see how we can make a war from it… We’ll be using the (awesome) warbler gem here. So if you don’t want to tweak anything, you just run:

# let's create a WAR
bundle exec warbler

and it’ll create a war containing all your libraries from lib/ and obviously the entire rails app. (Warbler is “smart” and can detect if your project is a rails app etc). If you need more control over what warbler is including (and where) in your warfile you can run:

# for PROs who want to configure warbler
warble config

Which will create a warble.rb file in conf/. I’ll leave it up to you if you need more tweaking or not (you could exclude temp/ and other directories from the warfile for example).

So… We’ve got our shiny war file. How can we run it? I’ll show you two ways of dealing with this, first: on Glassfish 3 (a applicationserver) and on Jetty (a small and fast servlet container).

So, with glassfish installed you could just:

# deploy a jruby war file to glassfish
asadmin start-domain domain1
asadmin deploy scala-on-rails.jar
chromium-browser http://localhost:8080/scala-on-rails

You may want to set it’s context path to / instead of /scala-on-rails but as you can see, it started and you’re running JRuby… accessing Scala code, on a normal JEE server!

If you prefer a smaller server, which ain’t so easy to administer remotely (asadmin rocks) but is a bit smaller etc, you should check out Jetty. There is a gem that claims to run apps on jetty “just so”, but it didn’t work out for me. And all in all… Deploying to jetty is also just 1 line of code, so why would a need a gem for that? ;-) Here’s how to deploy your war to jetty:

# It's that easy to deploy an app to jetty (using jetty-runner)
curl http://repo2.maven.org/maven2/org/mortbay/jetty/jetty-runner/7.6.0.RC5/jetty-runner-7.6.0.RC5.jar &gt; jetty-runner.jar
java -jar jetty-runner.jar scala-on-rails.war

…and that’s it! Look for the app on http://localhost:8080/

Ruby ♥ Scala ♥ Java ♥ JVM => Best friends

That’s it - we’ve deployed our first app using JRuby and Scala libraries. Such setup is doing pretty well for us and we can gain from scala/java power inside our quickly developed rails app.

The only tradeoff we made here is the fact that some gems require native extensions, which won’t work with JRuby. But we gained a hell lot of powerful libraries (lessening code duplication among projects in the process too!) and “1 file deployments”, which are really a lot nicer than deploying rails apps (that’s a personal opinion there, don’t kill me ;-)).

That’s it folks. All hail to the JVM and keep on hakking!