Mixer, the recently announced Yahoo’s new data service for news activities, uses Memcached and Sherpa for its data backend. Plus a combination of asynchronous libraries and task execution tools:

The data processing model and the clear separation between read and write data solutions is not only compelling, but essential for maintaining the SLA (max. 250ms/response):

Memcache maintains two types of materialized views: 1) Consumer-pivoted, and 2) Producer-pivoted. Consumer-pivoted views (e.g. user’s friends’ latest read activity) are refreshed at query time by refresh tasks. Producer-pivoted views (e.g. user’s latest read activity) are refreshed at update time (i.e. when “read” event is posted). And producer-pivoted views are used to refresh consumer-pivoted views.

Sherpa is Yahoo!’s cloud-based NoSql data store that provides low-latency reads and writes of key-value records and short range scans. Efficient range scans are particular important for the Mixer use cases. The “read” event is stored in the Updates table. The Updates table is a Sherpa Distributed Ordered Table that is ordered by “user,timestamp desc”. This provides efficient scans through a user’s latest read activity. A reference to the “read” record is stored in the UpdatesIndex table to support efficient point lookups. UpdatesIndex is a Sherpa Distributed Hash Table

Original title and link: Memcached and Sherpa for Yahoo! News Activity Data Service (NoSQL database©myNoSQL)

via: http://developer.yahoo.com/blogs/ydn/posts/2011/09/mixer-?-the-data-service-that-powers-yahoo-news-activity/

GigaOm breaks the news of the Yahoo! Hadoop engineering spinoff, HortonWorks:

By incorporating next-generation features and capabilities, HortonWorks hopes to make Hadoop easier to consume and better suited for running production workloads. Its products, which likely will include higher-level management tools on top of the core MapReduce and file system layers, will be open source and HortonWorks will try to maintain a close working relationship with Apache. The goal is to make HortonWorks the go-to vendor for a production-ready Hadoop distribution and support, but also to advance Yahoo’s repeated mission of making the official Apache Hadoop distribution the place to go for core software. Earlier this year, Yahoo discontinued its own Hadoop distribution, recommitting all that code and all its development efforts to Apache.

Judging by all the cool projects the Yahoo! engineering team has created, I’ve already said this makes a lot of sense to me.

Original title and link: Yahoo Launches Hadoop Spinoff (NoSQL database©myNoSQL)

via: http://gigaom.com/cloud/exclusive-yahoo-launching-hadoop-spinoff-this-week/

They are said to be building a proprietary replacement for the Hadoop Distributed File System that’s allegedly three times faster than the current open-source version. It comes with snapshots and no NameNode single point of failure (SPOF), and is supposed to be API-compatible with HDFS, so it can be a drop-in replacement.

Where can one get Mapr product from?

Considering Yahoo is now focusing on Apache Hadoop and their plans for the next generation Hadoop MapReduce, I wouldn’t hold my breath for Mapr improvements.

Original title and link: Mapr: a Competitor to Hadoop Leader Cloudera (NoSQL databases © myNoSQL)

via: http://gigaom.com/cloud/meet-mapr-a-competitor-to-hadoop-leader-cloudera/

New version of Cloudera’s Hadoop distro — complete release notes available here:

CDH3 Beta 4 also includes new versions of many components. Highlights include:

• HBase 0.90.1, including much improved stability and operability.
• Hive 0.7.0rc0, including the beginnings of authorization support, support for multiple databases, and many other new features.
• Pig 0.8.0, including many new features like scalar types, custom partitioners, and improved UDF language support.
• Flume 0.9.3, including support for Windows and improved monitoring capabilities.
• Sqoop 1.2, including improvements to usability and Oracle integration.
• Whirr 0.3, including support for starting HBase clusters on popular cloud platforms.

Plus many scalability improvements contributed by Yahoo!.

Cloudera’s CDH is the most popular Hadoop distro bringing together many components of the Hadoop ecosystem. Yahoo remains the main innovator behind Hadoop.

Original title and link: Cloudera’s Distribution for Apache Hadoop version 3 Beta 4 (NoSQL databases © myNoSQL)

via: http://www.cloudera.com/blog/2011/02/cdh3-beta-4-now-available

I’m not sure how many companies have already hit this limit, but Yahoo! is showing again its Hadoop leadership:

The Apache Hadoop MapReduce framework has hit a scalability limit around 4,000 machines. We are developing the next generation of Apache Hadoop MapReduce that factors the framework into a generic resource scheduler and a per-job, user-defined component that manages the application execution. Since downtime is more expensive at scale high-availability is built-in from the beginning; as are security and multi-tenancy to support many users on the larger clusters. The new architecture will also increase innovation, agility and hardware utilization.

There are way too many interesting aspects covered in the post to spoil the pleasure of diving into them.

Original title and link: The Next Generation of Apache Hadoop MapReduce (NoSQL databases © myNoSQL)

via: http://developer.yahoo.com/blogs/hadoop/posts/2011/02/mapreduce-nextgen/

We currently have nearly 100 people working on Apache Hadoop and related projects, such as Pig, ZooKeeper, Hive, Howl, HBase and Oozie. Over the last 5 years, we’ve invested nearly 300 person-years into these projects. […] Today Yahoo runs on over 40,000 Hadoop machines (>300k cores). They are used by over a thousand regular users from our science and development teams. Hadoop is at the center of our research in search, advertising, spam detection, personalization and many other topics.

I assume there’s no surpise to anyone I’m a big fan of Yahoo! open source initiatives.

Original title and link: The Backstory of Yahoo and Hadoop (NoSQL databases © myNoSQL)

via: http://developer.yahoo.com/blogs/hadoop/posts/2011/01/the-backstory-of-yahoo-a