Since direct integration of data flow and control flow is neither reasonable nor desirable, a heuristic is needed to productively combine the two. […] Compared to an approach that integrates control flow and data flow, such as PL/SQL, embedding in an existing scripting language is a much lower development and maintenance effort. It will also be much easier for users, who will be able to use existing development tools (IDEs, debuggers, etc.) to work with their scripts.

The first proposal—macro expansions— has already been committed and will be included in the next Pig Latin release.

Original title and link: Pig Latin Adds Macros as Part of Becoming Turing Complete (NoSQL databases © myNoSQL)

via: http://wiki.apache.org/pig/TuringCompletePig

We currently have nearly 100 people working on Apache Hadoop and related projects, such as Pig, ZooKeeper, Hive, Howl, HBase and Oozie. Over the last 5 years, we’ve invested nearly 300 person-years into these projects. […] Today Yahoo runs on over 40,000 Hadoop machines (>300k cores). They are used by over a thousand regular users from our science and development teams. Hadoop is at the center of our research in search, advertising, spam detection, personalization and many other topics.

I assume there’s no surpise to anyone I’m a big fan of Yahoo! open source initiatives.

Original title and link: The Backstory of Yahoo and Hadoop (NoSQL databases © myNoSQL)

via: http://developer.yahoo.com/blogs/hadoop/posts/2011/01/the-backstory-of-yahoo-a

Dmitriy Ryaboy¹ has a guest post on Cloudera blog covering the new features in Apache Pig 0.8.

Summarized:

• Support for user defined functions (UDF) in scripting languages
• Generic UDFs: allows invocation of static java methods
• PigUnit: as the name suggests, a testing tool for Pig scripts
• PigStats: once again the name should give you a hint of what it does: better visibility into Pig job through a series of stats, XML-based metadata injected into Map-Reduce jobs, and listeners for the Pig process
• Scalar values: simplifying access to single-row relations
• possibility to start a monitoring thread for long running executions
• HBaseStorage: works with HBase 0.20 releases only
• flow allows custom Map-Reduce jobs
• automatic merge of small files
• custom partitioners

The Pig 0.8 release includes a large number of bug fixes and optimizations, but at the core it is a feature release. It’s been in the works for almost a full year and the amount of time spent on 0.8 really shows.

You can also check Dmitriy’s presentations about the NoSQL ecosystem at Twitter: Twitter, Pig, and HBase and HBase and Pig: The Hadoop ecosystem at Twitter

Original title and link: Apache Pig 0.8: What is New (NoSQL databases © myNoSQL)

via: http://www.cloudera.com/blog/2010/12/new-features-in-apache-pig-0-8/

Rajarshi Guha about Pig Latin:

While the implementation of such code [SMARTS matching and pharmacophore searching] is pretty straightforward, it’s still pretty heavyweight compared to say, performing SMARTS matching in a database via SQL. On the other hand, being able to perform these tasks in Pig Latin, lets us write much simpler code that can be integrated with other non-cheminformatics code in a flexible manner.

Extensibility over compactness.

Original title and link: Pig and Cheminformatics (NoSQL databases © myNoSQL)

via: http://blog.rguha.net/?p=748

Matt Asay (GigaOm):

Where Cloudera shines, however, is in taking these different contributions and making Hadoop relevant for enterprise IT[8], where data mining has waxed and waned over the years. […] Cloudera, in other words, is banking on the complexity of Hadoop to drive enterprise IT to its own Cloudera Enteprise tools.

Additionally, I think what Cloudera is “selling” is a good set of tools — Hadoop, HBase, Hive, Pig, Oozie, Sqoop, Flume, Zookeeper, Hue — put together based on their expertise in the field.

Original title and link: Cloudera: All Your Big Data Are Belong to Us (NoSQL databases © myNoSQL)

via: http://cloud.gigaom.com/2010/09/14/cloudera-all-your-big-data-are-belong-to-us/

Posted on the MongoDB mailing list:

I have about 500M log file entries each representing an “ad impression” (we are an advertising company). Each “hit” has about 50 attributes to it (example: Country, State, City, Adsize, Browser, OS, etc) .. I want to load all 500M into some form of database and then run queries against this set.

As you could expect MongoDB is considered as a possibility. But I’d call that a biased vendor advise. I’ll be blunt: invest in your future by using Hadoop and Pig. Hive may fit too.

Original title and link for this post: MongoDB or Hadoop? (published on the NoSQL blog: myNoSQL)

via: http://groups.google.com/group/mongodb-user/browse_thread/thread/632d1648707e51d9/9a504c99168cf4e7

Fantastic post on Yahoo! Hadoop blog presenting a series of scenarios where using Pig and Hive makes things a lot better:

The widespread use of Pig at Yahoo! has enabled the migration of our data factory processing to Hadoop. With the adoption of Hive, we will be able to move much of our data warehousing to Hadoop as well. Having the data factory and the data warehouse on the same system will lower data-loading time into the warehouse — as soon as the factory is finished, it is available in the warehouse. It will also enable us to share — across both the factory and the warehouse — metadata, monitoring, and management tools; support and operations teams; and hardware. So we are excited to add Hive to our toolkit, and look forward to using both these tools together as we lean on Hadoop to do more and more of our data processing.

The use cases mentioned in the post:

• data preparation and presentation:

  Given the different workloads and different users for each phase, we have found that different tools work best in each phase. Pig (combined with a workflow system such as Oozie) is best suited for the data factory, and Hive for the data warehouse.

• data factories: pipelines (Pig + Oozie), iterative processing (Pig), research (Pig)
• data warehouse: business-intelligence analysis and ad-hoc queries

  In both cases, the relational model and SQL are the best fit. Indeed, data warehousing has been one of the core use cases for SQL through much of its history. It has the right constructs to support the types of queries and tools that analysts want to use. And it is already in use by both the tools and users in the field. The Hadoop subproject Hive provides a SQL interface and relational model for Hadoop.

Yahoo! gets way to little credit for its work on bigdata and its contributions to the open source.

Original title and link for this post: Pig and Hive at Yahoo! (published on the NoSQL blog: myNoSQL)

via: http://developer.yahoo.net/blogs/hadoop/2010/08/pig_and_hive_at_yahoo.html

Yet another contribution from Yahoo!:

Common metadata layer for Hadoop’s Map Reduce, Pig, and Hive

Howl: Unifying Metadata Layer for Hive and Pig originally posted on the NoSQL blog: myNoSQL

via: http://github.com/yahoo/howl

I didn’t know data warehousing strictly imposes a relational model:

From a philosophical standpoint, my largest problem with NoSQL databases is that they don’t respect relational theory. In short, they aren’t meant to deal with sets of data, but lists. Relational algebra was created to deal with the large sets of data and have them interact. Reporting and analytics rely on that.

I’d bet people building and using Hive, Pig, Flume and other data warehousing tools would disagree with Eric Hewitt.

NoSQL Databases and Data Warehousing originally posted on the NoSQL blog: myNoSQL

via: http://info.livelogic.net/customer-intelligence-project-success/bid/49227/Distributed-Hash-Tables-and-Concentrated-Geekdom

Probably one of the nicest taglines for Pig:

If Perl is the duct tape of the internet, and Hadoop is the kernel of the data center as computer, then Pig is the duct tape of Big Data.

And an advise on how to use Pig:

When I write Pig Latin code beyond a dozen lines, I check it in stages:

• Write Pig Latin in TextMate (Saved in a git repo, otherwise I lose code)
• Paste the code into the Grunt shell – Did it parse?
• DESCRIBE the final output and each complex step – Did it still parse? Is the schema what I expected?
• ILLUSTRATE the output – Does it still parse? Is the schema ok? Is the example data ok?
• SAMPLE / LIMIT / DUMP the output – Does it still parse? Is the schema ok? Is the sampled/limited data sane?
• STORE the final output and see if the job completes.
• cat output_dir/part-00000 (followed by a quick ctrl-c to stop the flood) – Is the stored output on HDFS ok?

via: http://blog.linkedin.com/2010/07/01/linkedin-apache-pig/