After posting about the proposal of adding a 128-bit K-Order unique id generator to Redis, I’ve realized I haven’t linked to Boundary’s post about Flake, their decentralized k-ordered unique id generator project:

All that is required to construct such an id is a monotonically increasing clock and a location 3. K-ordering dictates that the most-significant bits of the id be the timestamp. UUID-1 contains this information, but arranges the pieces in such a way that k-ordering is lost. Still other schemes offer k-ordering with either a questionable representation of ‘location’ or one that requires coordination among nodes.

Original title and link: Flake: An Erlang Decentralized, K-Ordered Unique ID Generator by Boundary (NoSQL database©myNoSQL)

via: http://blog.boundary.com/2012/01/12/Flake-A-decentralized-k-ordered-id-generation-service-in-Erlang.html

Drew Conway:

Today I am pleased to announce that we have worked out most of the bugs, and now have a reliable data set upon which to build. Better still, we are ready to share. Unlike our old system, the data now lives on a live CouchDB database, and can be queried for specific research tasks. We have combined all of the data available from Twitter’s search API with the information on each member from Sunlight Foundation’s Congressional API. […] But be forewarned, working with this system and CouchDB requires a mature understanding of several tools and languages; including but not restricted to; curl, map/reduce, Javascript, and JSON. And that’s before you have even done any analysis.

Original title and link: Interesting Data Sets and Tools: Monthly Twitter Activity for All Members of the U.S. Congress (NoSQL database©myNoSQL)

via: http://www.drewconway.com/zia/?p=2846

Twitter’s real-time URL fetcher, code named SpiderDuck, is an excellent example of how NoSQL databases fit in the architecture of today’s systems:

Metadata Store: This is a Cassandra-based distributed hash table that stores page metadata and resolution information keyed by URL, as well as fetch status for every URL recently encountered by the system. This store serves clients across Twitter that need real-time access to URL metadata.

SpiderDuck is also using memcached:

Memcached: This is a distributed cache used by the fetchers to temporarily store robots.txt files.

Original title and link: Twitter’s Real-Time URL Fetcher Using Cassandra and Memcached (NoSQL database©myNoSQL)

via: http://engineering.twitter.com/2011/11/spiderduck-twitters-real-time-url.html

Ask and you’ll be answered. Nathan Marz announces that Twitter will open source Storm, the Hadoop-like real-time data processing tool developed at BackType:

I’m pleased to announce that I will be releasing Storm at Strange Loop on September 19th!

Here’s a recap of the three broad use cases for Storm:

• Stream processing: Storm can be used to process a stream of new data and update databases in realtime. Unlike the standard approach of doing stream processing with a network of queues and workers, Storm is fault-tolerant and scalable.
• Continuous computation: Storm can do a continuous query and stream the results to clients in realtime. An example is streaming trending topics on Twitter into browsers. The browsers will have a realtime view on what the trending topics are as they happen.
• Distributed RPC: Storm can be used to parallelize an intense query on the fly. The idea is that your Storm topology is a distributed function that waits for invocation messages. When it receives an invocation, it computes the query and sends back the results. Examples of Distributed RPC are parallelizing search queries or doing set operations on large numbers of large sets.

Original title and link: Twitter Open Sourcing Storm at Strange Loop (NoSQL database©myNoSQL)

via: http://engineering.twitter.com/2011/08/storm-is-coming-more-details-and-plans.html

The Redis Twitter clone app is showing its age:

I’m looking at the Twitter Clone and noticed a N + 1 -like “get” in the code […] The above code seems rather suboptimal, if my understanding is correct.

At least three better approaches have been suggested, so who is up for experimenting with Redis and rewriting this app to use latest Redis features?

1. use pipelining to get all the posts in one server roundtrip (won’t change the code much and be much faster)
2. use SORT … GET semantics to get all the post data at once from the list of ids (should be somewhat faster than 1)
3. Use MGET to get all the post data at once.

Original title and link: Rewriting the Redis Twitter Clone (NoSQL databases © myNoSQL)

via: http://groups.google.com/group/redis-db/browse_thread/thread/a67aae56aca2bc84

J.Chris Anderson in an interview over ReadWriteWeb:

Klint Finley: Let’s start at the top: what exactly is Twebz? It’s described as a “decentralized Twitter client.” What exactly does that mean?

J Chris Anderson: The aim is to allow you to interact with Twitter when Twitter is up and you are online. But if Twitter is down for maintenance or you are in the middle of nowhere, you can still tweet. And when you can reach Twitter again, it will go through.

If lots of folks are using it, then they can see each other’s tweets come in even when Twitter is down.

Mostly the goal was to show the way on how to integrate CouchDB with web services and APIs.

A classical example of CouchDB powerful P2P replication capabilities. Dave Winer would probably be its ☞ biggest fan.

Original title and link: CouchDB Usecase: Decentralizing Twitter (NoSQL databases © myNoSQL)

via: http://www.readwriteweb.com/hack/2010/12/j-chris-anderson-interview.php