cascalog: All content tagged as cascalog in NoSQL databases and polyglot persistence
Monday, 5 December 2011
Looking for a Map Reduce Language
Java, Cascading, Pipes - C++, Hive, Pig, Rhipe, Dumbo, Cascalog… which one of these should you use for writing Map Reduce code?
Antonio Piccolboni takes them up for a test:
At the end of this by necessity incomplete and unscientific language and library comparison, there is a winner and there isn’t. There isn’t because language comparison is always multidimensional and subjective but also because the intended applications are very different. On the other hand, looking for a general purpose, moderately elegant, not necessarily most efficient, not necessarily mature language for exploration purposes, Rhipe seems to fit the bill pretty nicely.
Original title and link: Looking for a Map Reduce Language (©myNoSQL)
via: http://blog.piccolboni.info/2011/04/looking-for-map-reduce-language.html
Tuesday, 5 July 2011
ElephantDB and Storm Join the Twitter Flock
That’s to say BackType, creators of Cascalog, ElephantDB, and Storm , has been acquired by Twitter (which in case you didn’t know names most of their open source libraries and storage solutions using bird names).
The announcement is here . Looking forward to seeing Storm open sourced.
Original title and link: ElephantDB and Storm Join the Twitter Flock (©myNoSQL)
Wednesday, 11 May 2011
Simhashing in Hadoop with MapReduce, Cascalog and Cascading
Simhashing in MapReduce is a quick way to find clusters in a huge amount of data. By using Cascading and Cascalog we’re able to work with MapReduce jobs at the level of functions rather than individual map-reduce phases.
Original title and link: Simhashing in Hadoop with MapReduce, Cascalog and Cascading (NoSQL databases © myNoSQL)
Saturday, 19 February 2011
BackType’s ElephantDB
I didn’t know BackType’s ElephantDB is open source and available on GitHub, same as their Cascalog the Clojure based query language for Hadoop.
Original title and link: BackType’s ElephantDB (NoSQL databases © myNoSQL)
Thursday, 13 January 2011
Big Data Analysis at BackType
RWW has a nice post diving into the data flow and the tools used by BackType, a company with only 3 engineers, to deal and analyze large amounts of data.
They’ve invented their own language, Cascalog, to make analysis easy, and their own database, ElephantDB, to simplify delivering the results of their analysis to users. They’ve even written a system to update traditional batch processing of massive data sets with new information in near real-time.
Some highlights:
- 25 terabytes of compressed binary data, over 100 billion individual records
- all services and data storage are on Amazon S3 and EC2
- 60 up to 150 EC2 instances servicing an average of 400 requests/s
- Clojure and Python as platform languages
- Hadoop, Cascading and Cascalog are central pieces of BackType’s platform
- Cascalog, a Clojure-based query language for Hadoop, was created and open sourced by BackType’s engineer Nathan Marz
- ElephantDB, the storage solution, is a read-only cluster built on top of BerkleyDB files
- Crawlers place data in Gearman queues for processing and storing
BackType data flow is presented in the following diagram:
Included below is an interview with Nathan about Cascalog:
Original title and link: Big Data Analysis at BackType (NoSQL databases © myNoSQL)
via: http://www.readwriteweb.com/hack/2011/01/secrets-of-backtypes-data-engineers.php
Thursday, 11 November 2010
Quick Reference: Hadoop Tools Ecosystem
Just a quick reference of the continuously growing Hadoop tools ecosystem.
Hadoop
The Apache Hadoop project develops open-source software for reliable, scalable, distributed computing.
HDFS
A distributed file system that provides high throughput access to application data.
MapReduce
A software framework for distributed processing of large data sets on compute clusters.
Amazon Elastic MapReduce
Amazon Elastic MapReduce is a web service that enables businesses, researchers, data analysts, and developers to easily and cost-effectively process vast amounts of data. It utilizes a hosted Hadoop framework running on the web-scale infrastructure of Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Simple Storage Service (Amazon S3).
aws.amazon.com/elasticmapreduce/
Cloudera Distribution for Hadoop (CDH)
Cloudera’s Distribution for Hadoop (CDH) sets a new standard for Hadoop-based data management platforms.
ZooKeeper
A high-performance coordination service for distributed applications. ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.
HBase
A scalable, distributed database that supports structured data storage for large tables.
Avro
A data serialization system. Similar to ☞ Thrift and ☞ Protocol Buffers.
Sqoop
Sqoop (“SQL-to-Hadoop”) is a straightforward command-line tool with the following capabilities:
- Imports individual tables or entire databases to files in HDFS
- Generates Java classes to allow you to interact with your imported data
- Provides the ability to import from SQL databases straight into your Hive data warehouse
Flume
Flume is a distributed, reliable, and available service for efficiently moving large amounts of data soon after the data is produced.
archive.cloudera.com/cdh/3/flume/
Hive
Hive is a data warehouse infrastructure built on top of Hadoop that provides tools to enable easy data summarization, adhoc querying and analysis of large datasets data stored in Hadoop files. It provides a mechanism to put structure on this data and it also provides a simple query language called Hive QL which is based on SQL and which enables users familiar with SQL to query this data. At the same time, this language also allows traditional map/reduce programmers to be able to plug in their custom mappers and reducers to do more sophisticated analysis which may not be supported by the built-in capabilities of the language.
Pig
A high-level data-flow language and execution framework for parallel computation. Apache Pig is a platform for analyzing large data sets that consists of a high-level language for expressing data analysis programs, coupled with infrastructure for evaluating these programs. The salient property of Pig programs is that their structure is amenable to substantial parallelization, which in turns enables them to handle very large data sets.
Oozie
Oozie is a workflow/coordination service to manage data processing jobs for Apache Hadoop. It is an extensible, scalable and data-aware service to orchestrate dependencies between jobs running on Hadoop (including HDFS, Pig and MapReduce).
Cascading
Cascading is a Query API and Query Planner used for defining and executing complex, scale-free, and fault tolerant data processing workflows on a Hadoop cluster.
Cascalog
Cascalog is a tool for processing data on Hadoop with Clojure in a concise and expressive manner. Cascalog combines two cutting edge technologies in Clojure and Hadoop and resurrects an old one in Datalog. Cascalog is high performance, flexible, and robust.
github.com/nathanmarz/cascalog
HUE
Hue is a graphical user interface to operate and develop applications for Hadoop. Hue applications are collected into a desktop-style environment and delivered as a Web application, requiring no additional installation for individual users.
You can read more about HUE on ☞ Cloudera blog.
Chukwa
Chukwa is a data collection system for monitoring large distributed systems. Chukwa is built on top of the Hadoop Distributed File System (HDFS) and Map/Reduce framework and inherits Hadoop’s scalability and robustness. Chukwa also includes a flexible and powerful toolkit for displaying, monitoring and analyzing results to make the best use of the collected data.
Mahout
A Scalable machine learning and data mining library.
Integration with Relational databases
- Oracle
- Hadoop connector for Oracle Ora-Oop
- Hadoop and Oracle Parallel Processing
Integration with Data Warehouses
The only list I have is MapReduce, RDBMS, and Data Warehouse, but I’m afraid it is quite a bit old. So maybe someone could help me update it.
Anything else? Once we validate this list, I’ll probably have to move it on the NoSQL reference
Original title and link: Quick Reference: Hadoop Tools Ecosystem (NoSQL databases © myNoSQL)
Wednesday, 26 May 2010
Presentations on Hadoop, HBase, PIG and Cascalog from Hadoop Meet-Up
The Yahoo! Developer Network Blog has ☞ posted the materials presented at Hadoop’s monthly user group meeting. I’ve embedded these below for your convenience:
What’s New With Pig: Alan Gates
Pig is one of the solutions used for data processing/analysis in the NoSQL world. For example Pig is heavily used at Twitter.
Recently Pig has released ☞ two new versions (0.6.0 and 0.7.0) and this talk focuses on the new features included with these versions and a compatibility plan with Hadoop[1]
Cascalog: Powerful and easy-to-use data analysis tool for Hadoop: Nathan Marz
Cascalog is a Clojure-based query language solution for Hadoop-stored data analysis. Nathan Marz (BackType) is demoing this cool tool:
HBase and Pig: The Hadoop ecosystem at Twitter: Dmitriy Ryaboy
As already mentioned Twitter is extensively using HBase, Pig and Hadoop — in their words Cassandra is OLTP and HBase is OLAP — and Dmitriy provides an overview of their Hadoop-based ecosystem:
References
- [1] Yahoo! Developer Network Blog has an article on this topic ☞ Towards Enterprise-Class Compatibility for Apache Hadoop. Considering that after the last release HBase has become a top-level Apache project and that there’s a very strong userbase for HBase and Hadoop, ensuring a healthy ecosystem for all these projects is extremely important. (↩)
Thursday, 6 May 2010
Cascalog: Clojure-based Query Language for Hadoop
I’ve already made the point that storing the data is just the start of the “adventure” and sooner than later you’ll have to put your NoSQL stored data to work.
Cascalog, introduced in the linked article, is a query language for Hadoop featuring:
- Simple – Functions, filters, and aggregators all use the same syntax. Joins are implicit and natural.
- Expressive – Logical composition is very powerful, and you can run arbitrary Clojure code in your query with little effort.
- Interactive – Run queries from the Clojure REPL.
- Scalable – Cascalog queries run as a series of MapReduce jobs.
- Query anything – Query HDFS data, database data, and/or local data by making use of Cascading’s “Tap” abstraction
- Careful handling of null values – Null values can make life difficult. Cascalog has a feature called “non-nullable variables” that makes dealing with nulls painless.
- First class interoperability with Cascading – Operations defined for Cascalog can be used in a Cascading flow and vice-versa
- First class interoperability with Clojure - Can use regular Clojure functions as operations or filters, and since Cascalog is a Clojure DSL, you can use it in other Clojure code.
While I do really like Clojure[1] and its conciseness, I kind of agree with the point Kevin Weil from Twitter made during the nosql:eu conference:
I need less Java in my life, not more
in the sense that higher level tools are more productive. And that’s the reason Twitter is using PIG (nb: I’d strongly recommend checking Kevin Weil’s slides) and Cloudera Hadoop distribution will include PIG and Hive.
References
- [1] Disclaimer: I’m a Clojure absolute beginner though. (↩)
