Elegant XML Parsing in Ruby
If you are parsing big XML files you have to stream them using something like SAX. Does your code look like this?
def start_element tag, attributes case tag when 'name' @attribute = 'name' when 'titles' @attribute = 'name' end end def end_element tag @model.save! if tag == 'name' end def characters text @model[@attribute] = text end
Please stop doing this!
Sax Machine was created to have an elegant API for parsing XML.
class AtomEntry include SAXMachine element :title # the :as argument makes this available through atom_entry.author instead of .name element :name, :as => :author element "feedburner:origLink", :as => :url element :summary element :content element :published end
This is declarative coding, which is what we should be striving for no matter what programming tools we are using. The problem is that the official sax-machine gem loads everything into memory. The reason why is that it does not use a callback (yield) or fibers to process elements when they are immediately loaded.
At yap.TV, we have been using my fork of sax-machine for over a year now on gigantic XML files. This fork uses Ruby’s fibers to transfer control from the xml parsing back to your code. This is similar in concept to how yield works, but you don’t have to pass in a block right away. Instead you get back an Enumerator that will end up parsing on demand.
class Atom include SAXMachine element :title elements :entry, :lazy => true, :as => :entries, :class => AtomEntry end feed = Atom.parse(xml_file_handle, :lazy => true) feed.entries # => #<Enumerator: #<Enumerator::Generator:0x00000004c41ea0>:each> feed.entries.each do |entry| # every time the block is called the next entry is parsed- no memory blow up! # This is probably where you save the entry to a database end
The only thing extra you need to do is add a :lazy attribute to your top level elements. Also checkout ezkl’s fork. He is taking over maintenance of sax-machine in the official repo now though and planning on pulling in my changes.
This solves the memory part of performance. However, if this is not fast enough for you, I suggest switching the library to use ox. From the ox documentation:
Unlike Nokogiri and LibXML, Ox can be tuned to use only the SAX callbacks that are of interest to the caller
The current Sax parser will pull everything out of the XML and bring it into Ruby land. Avoiding this could make the parsing an order of magnitude faster depending on how sparsely you are using the XML.
For our usage at yap (importing to a database) we of course found that database access became a big bottleneck. We added some indexes and caching, and made some queries use the raw driver rather than go through an ORM.
sax-machine is a great example of how Ruby’s advanced features of fibers and meta-programming can be used. Please use it to write declarative code!