Project

Namespaces

Public Vars

pigpen.rx

A high performance dump operator. The default implementation in pigpen.core
uses lazy seqs, which can be inefficient on larger data. The rx implementation
uses rx-java to deliver a slightly more performance for large local datasets.

dump

added in 0.1.0

(dump query)
Executes a script locally and returns the resulting values as a clojure
sequence. This command is very useful for unit tests.

  Example:

    (->>
      (pig/load-clj "input.clj")
      (pig/map inc)
      (pig/filter even?)
      (pig-rx/dump)
      (clojure.core/map #(* % %))
      (clojure.core/filter even?))

    (deftest test-script
      (is (= (->>
               (pig/load-clj "input.clj")
               (pig/map inc)
               (pig/filter even?)
               (pig-rx/dump))
             [2 4 6])))

  Note: pig/store commands return the output data
        pig/store-many commands merge their results
view source