Here’s a taste of what you can do with DataFu:

• Run PageRank on a large number of independent graphs.
• Perform set operations such as intersect and union.
• Compute the haversine distance between two points on the globe.
• Create an assertion on input data which will cause the script to fail if the condition is not met.
• Perform various operations on bags such as append a tuple, prepend a tuple, concatenate bags, generate unordered pairs, etc.

I’m starting to notice a pattern here. Twitter is open sourcing pretty much everything they are doing related to data storage. Yahoo (now Hortonworks) and Cloudera are the forces behind the open source Hadoop and the tools to bring data to Hadoop. And LinkedIn is starting to open source the tools they are using on top of Hadoop to analyze big data.

What is interesting about this is that you might not get the most polished tools, but they definitely are battle tested.

Original title and link: DataFu: Open Source Apache Pig UDFs by LinkedIn (NoSQL database©myNoSQL)

via: http://engineering.linkedin.com/open-source/introducing-datafu-open-source-collection-useful-apache-pig-udfs

David F. Carr in an article about DJ Patil and his work on Big Data at LinkedIn:

That is what he means by data jujitsu, where jujitsu is the art of using an opponent’s leverage and momentum against him. In data jujitsu, you try to use the scope of the problem to create the solution—without investing disproportionate resources at the early experimental stage. That’s as opposed to data karate, which would be a direct frontal assault to hack your way through the problem.

Original title and link: Data Jujitsu and Data Karate (NoSQL database©myNoSQL)

via: http://www.informationweek.com/thebrainyard/news/strategy/231900611/web-20-expo-linkedins-big-data-lessons-learned

Another open source project from LinkedIn:

Kafka is a distributed publish-subscribe messaging system. It is designed to support the following:

• Persistent messaging with O(1) disk structures that provide constant time performance even with many TB of stored messages.
• High-throughput: even with very modest hardware Kafka can support hundreds of thousands of messages per second.
• Explicit support for partitioning messages over Kafka servers and distributing consumption over a cluster of consumer machines while maintaining per-partition ordering semantics.
• Support for parallel data load into Hadoop.

LinkedIn has open sourced a couple of exciting projects, but they haven’t been able to get enough attention and grow so far a community around these.

Original title and link: Kafka: LinkedIn’s Distributed Publish/Subscribe Messaging System (NoSQL databases © myNoSQL)

via: http://sna-projects.com/kafka/

Krati is a simple persistent data store with very low latency and high throughput. It is designed for easy integration with read-write-intensive applications with little effort in tuning configuration, performance and JVM garbage collection.

Sounds a bit like Bitcask. Anyone can point out at least on the major differences?

From the project page:

• supports varying-length data array
• supports key-value data store access
• performs append-only writes in batches
• has write-ahead redo logs and periodic checkpointing
• has automatic data compaction (i.e. garbage collection)
• is memory-resident (or OS page cache resident) yet persistent
• allows single-writer and multiple readers

Krati: A Persistent High-Performance Data Store originally posted on the NoSQL blog: myNoSQL

via: http://sna-projects.com/krati/

Probably one of the nicest taglines for Pig:

If Perl is the duct tape of the internet, and Hadoop is the kernel of the data center as computer, then Pig is the duct tape of Big Data.

And an advise on how to use Pig:

When I write Pig Latin code beyond a dozen lines, I check it in stages:

• Write Pig Latin in TextMate (Saved in a git repo, otherwise I lose code)
• Paste the code into the Grunt shell – Did it parse?
• DESCRIBE the final output and each complex step – Did it still parse? Is the schema what I expected?
• ILLUSTRATE the output – Does it still parse? Is the schema ok? Is the example data ok?
• SAMPLE / LIMIT / DUMP the output – Does it still parse? Is the schema ok? Is the sampled/limited data sane?
• STORE the final output and see if the job completes.
• cat output_dir/part-00000 (followed by a quick ctrl-c to stop the flood) – Is the stored output on HDFS ok?

via: http://blog.linkedin.com/2010/07/01/linkedin-apache-pig/