Just a couple of examples: product and brand engagement, advertising.

A recent study we just published in the Proceedings of the National Academy of Sciences tells a new story about the way people adopt products and engage with them. The prevailing theories about this process suggested that what influences a person [to] adopt technologies is the number or percentage of friends who have already adopted the same technology, along with a person’s threshold for adopting such technologies. Our study shows that it’s less about the number of your friends who are using the technology, but more about their diversity. […] Some of the work we’re interested in understanding is how your friends influence your decisions to engage with advertising and brands.

Original title and link: What Big Data Is Used for at Facebook (NoSQL database©myNoSQL)

via: http://www.technologyreview.com/printer_friendly_article.aspx?id=40184

Sarah Lacy:

The technologies in question include things like memcached which was created in 2003 by LiveJournal and has been used longer than Facebook has been alive.[…]

Other examples include Open Compute, an open hardware project started by Facebook that focuses on low-cost, energy efficient server and data center hardware; Tornado a Python-based web server used for building real-time Web services; and HPHP, a source code transformer that turns PHP into C++.

I have no other details about this patent letter Yahoo sent Facebook, but I seriously doubt it targets these technologies separately. Most probably it refers to some sort of combinations of these and one that Facebook has mentioned as part of their IP.

Original title and link: Yahoo Patent Letter to Facebook Referring to Memcached and Other Open Source Technologies (NoSQL database©myNoSQL)

via: http://pandodaily.com/2012/05/03/yahoo-ups-its-battle-with-facebook-and-picks-one-with-the-open-source-movement/

Just whetting your apetite for this interview with Mark Callaghan about MySQL, InnoDB, and his work at Facebook:

Q: How do you make MySQL both “less slow” and “faster” at the same time?

A: I ask questions like, “If I can make it do 10 things per second today, can I make it do 20 things per second tomorrow?” For example, we used to use an algorithm that is very CPU intensive to check database pages. Another person on my team, Ryan Mack, modified it to use hardware support on X86 processors so we could profile the servers in production to see what they were doing in these computing checksums.  We then realized that the newest CPUs had a faster way to do that, so we modified the MySQL to use the CRC32 for checksums. The hard part there was upgrading the servers on the fly from using the old check zones to the new checksums without taking the site down.

Exciting and scary.

Original title and link: A Story of MySQL and InnoDB at Facebook Told by Mark Callaghan (NoSQL database©myNoSQL)

via: https://www.facebook.com/notes/facebook-engineering/mysql-and-database-engineering-mark-callaghan/10150599729938920

Joe Maguire referring to the Facebook talk embedded below MySQL and HBase:

if Facebook doesn’t need NoSQL, who does?

My answer: many of those that cannot employ a specialized team to hack the hell out of MySQL to make it work at that scale.

On the flipside, many other companies don’t have the time or engineering power to grow their product together with a NoSQL database.

via: http://josephmaguire.blogspot.com/2011/12/facebook-there-are-no-published-cases.html

Mixer, the recently announced Yahoo’s new data service for news activities, uses Memcached and Sherpa for its data backend. Plus a combination of asynchronous libraries and task execution tools:

The data processing model and the clear separation between read and write data solutions is not only compelling, but essential for maintaining the SLA (max. 250ms/response):

Memcache maintains two types of materialized views: 1) Consumer-pivoted, and 2) Producer-pivoted. Consumer-pivoted views (e.g. user’s friends’ latest read activity) are refreshed at query time by refresh tasks. Producer-pivoted views (e.g. user’s latest read activity) are refreshed at update time (i.e. when “read” event is posted). And producer-pivoted views are used to refresh consumer-pivoted views.

Sherpa is Yahoo!’s cloud-based NoSql data store that provides low-latency reads and writes of key-value records and short range scans. Efficient range scans are particular important for the Mixer use cases. The “read” event is stored in the Updates table. The Updates table is a Sherpa Distributed Ordered Table that is ordered by “user,timestamp desc”. This provides efficient scans through a user’s latest read activity. A reference to the “read” record is stored in the UpdatesIndex table to support efficient point lookups. UpdatesIndex is a Sherpa Distributed Hash Table

Original title and link: Memcached and Sherpa for Yahoo! News Activity Data Service (NoSQL database©myNoSQL)

via: http://developer.yahoo.com/blogs/ydn/posts/2011/09/mixer-?-the-data-service-that-powers-yahoo-news-activity/

The story of migrating 30PB of HDFS stored data:

The scale of this migration exceeded all previous ones, and we considered a couple of different migration strategies. One was a physical move of the machines to the new data center. We could have moved all the machines within a few days with enough hands at the job. However, this was not a viable option as our users and analysts depend on the data 24/7, and the downtime would be too long.

Facebook could have convinced part of the 750+ million users to help carry over the physical machines.

Luckily HDFS has replication built in and Facebook engineers have paired it with their own improvements:

Another approach was to set up a replication system that mirrors changes from the old cluster to the new, larger cluster. Then at the switchover time, we could simply redirect everything to the new cluster. This approach is more complex as the source is a live file system, with files being created and deleted continuously. Due to the unprecedented cluster size, a new replication system that could handle the load would need to be developed. However, because replication minimizes downtime, it was the approach that we decided to use for this massive migration.

Another bit worth emphasizing in this post:

In 2010, Facebook had the largest Hadoop cluster in the world, with over 20 PB of storage. By March 2011, the cluster had grown to 30 PB — that’s 3,000 times the size of the Library of Congress!

Until now I thought Yahoo! has the largest Hadoop deployment, at least in terms of number of machines.

Original title and link: Moving an Elephant: Large Scale Hadoop Data Migration at Facebook (NoSQL database©myNoSQL)

via: http://www.facebook.com/notes/paul-yang/moving-an-elephant-large-scale-hadoop-data-migration-at-facebook/10150246275318920

Starting from the architecture of Facebook’s realtime analytics presented in the paper Apache Hadoop Goes Realtime at Facebook and Dhruba Borthakur’s excellent posts HDFS: Realtime Hadoop and HBase Usage at Facebook, Nati Shalom describes an alternative approach for real-time analytics using data grids making the following assumptions:

They had some assumptions in design that centered around the reliability of in-memory systems and database neutrality that affected what they did: for memory, that transactional memory was unreliable, and for the database, that HBase was the only targeted data store.

What if those assumptions are changed? We can see reliable transactional memory in the field, as a requirement for any in-memory data grid, and certainly there are more databases than HBase; given database and platform neutrality, and reliable transactional memory, how could you build a realtime analytics system?

While a great read, I get the feeling there’s something wrong. Maybe this:

There are lots of areas in which you can see potential improvements, if the assumptions are changed. As a contrast to Facebook’s working system: […] We can consolidate the analytics system so that management is easier and unified. While there are system management standards like SNMP that allow management events to be presented  in the same way no matter the source, having so many different pieces means that managing the system requires an encompassing understanding, which makes maintenance and scaling more difficult.

and then:

One other advantage of data grids is in write-through support. With write-through, updates to the data grid are written asynchronously to a backend data store – which could be HBase (as used by Facebook), Cassandra, a relational database such as MySQL, or any other data medium you choose for long-term storage, should you need that.

Original title and link: An Alternative Approach for Big Data Real Time Analytics (NoSQL database©myNoSQL)

via: http://natishalom.typepad.com/nati_shaloms_blog/2011/07/real-time-analytics-for-big-data-an-alternative-approach.html

Not only is Mozilla celebrating the release of Firefox 4, but they took the time to set up a nice visualization for downloads.

glow.mozilla.org is powered by tailing logs and streaming data into HBase:

1. The various load balancing clusters that host download.mozilla.org are configured to log download requests to a remote syslog server.
2. The remote server is running rsyslog and has a config that specifically filters those remote syslog events into a dedicated file that rolls over hourly
3. SQLStream is installed on that server and it is tailing those log files as they appear.
4. The SQLStream pipeline does the following for each request:
   • filtering out anything other than valid download requests
   • uses MaxMind GeoIP to get a geographic location from the IP address
   • uses a streaming group by to aggregate the number of downloads by product, location, and timestamp
   • every 10 seconds, sends a stream of counter increments to HBase for the timestamp row with the column qualifiers being each distinct location that had downloads in that time interval
5. The glow backend is a python app that pulls the data out of HBase using the Python Thrift interface and writes a file containing a JSON representation of the data every minute.
6. That JSON file can be cached on the front-end forever since each minute of data has a distinct filename
7. The glow website pulls down that data and plays back the downloads or allows you to browse the geographic totals in the arc chart view

This sounds a lot like what Facebook is doing for the new Real-Time Analytics system.. The parts missing are Scribe and ptail.

Original title and link: Firefox Downloads Visualization Powered by HBase (NoSQL databases © myNoSQL)

via: http://blog.mozilla.com/data/2011/03/22/how-glow-mozilla-org-gets-its-data/