5 techniques and links to research papers about data deduplication using HDFS and MapReduce:

Some of the common methods for data deduplication in storage architecture include hashing, binary comparison and delta differencing. In this post, we focus on how MapReduce and HDFS can be leveraged for eliminating duplicate data.

Patrick Durusau

Original title and link: Data Deduplication Tactics With HDFS and MapReduce (NoSQL database©myNoSQL)

via: http://www.hadoopsphere.com/2013/02/data-de-duplication-tactics-with-hdfs.html

Steve Jin has put together a quick list of HDFS commands:

The first part hadoop fs is always the same for file system related commands. After that is very much like typical Unix/Linux commands in syntax. Besides managing the HDFS itself, there are commands to import data files from local file system to HDFS, and export data files from HDFS to local file system. These commands are unique therefore deserve most attention.

[-put ... ] [-copyFromLocal ... ] [-moveFromLocal ... ] [-get [-ignoreCrc] [-crc] ] [-getmerge [addnl]] [-copyToLocal [-ignoreCrc] [-crc] ] [-moveToLocal [-crc] ]

Original title and link: Quick Reference to Hadoop File System Commands (NoSQL database©myNoSQL)

via: http://www.doublecloud.org/2012/09/hadoop-file-system-commands/

In a reply to Cloudera’s defense of HDFS, Jeff Darcy comments about the portability of HDFS:

This is also not an HDFS exclusive. Any of the alternatives that were developed outside the Hadoopiverse have this quality as well. If you have data in Cassandra or Ceph you can keep it in Cassandra or Ceph as you go Hadoop-distro shopping. The biggest data-portability wall here is HDFS’s, because it’s one of only two such systems (the other being MapR) that’s Hadoop-specific. It doesn’t even try to be a general-purpose filesystem or database. A tremendous amount of work has gone into several excellent tools to import data into HDFS, but that work wouldn’t even be necessary with some of the alternatives. That’s not just a waste of machine cycles; it’s also a waste of engineer cycles. If they hadn’t been stuck in the computer equivalent of shipping and receiving, the engineers who developed those tools might have created something even more awesome. I know some of them, and they’re certainly capable of it. Each application can write the data it generates using some set of interfaces. If HDFS isn’t one of those, or if HDFS through that interface is unbearably slow because the HDFS folks treat anything other than their own special snowflake as second class, then you’ll be the one copying massive amounts of data before you can analyze it … not just once, but every time.

Original title and link: Attacking HDFS’s Defense: Why Does Cloudera *Really* Use HDFS? (NoSQL database©myNoSQL)

via: http://hekafs.org/index.php/2012/07/why-does-cloudera-really-use-hdfs/

Building on Eric Baldeschwieler’s defense of HDFS, Cloudera’s Charles Zedlewski adds a couple of HDFS advantages:

• Choice: Customers get to work with any leading hardware vendor and let the best possible price / performer win the decision, not whatever the vendor decided to bundle in.
• Portability: It is possible for customers running Hadoop distributions based on HDFS to move between those different distributions without having to reformat the cluster or copy massive amounts of data. When you’re talking about petabytes of data, this kind of portability is vital. Without it, your vendor has incredible leverage when it comes time to negotiate the next purchase.
• Shared industry R&D We at Cloudera are proud of our employee’s own contributions to HDFS, and they collaborate with their colleagues at Hortonworks. But today you will find that IBM, Microsoft and VMware are also contributing to HDFS to make it work better with their products. In the future I predict you’ll find hard drive, networking and server manufacturers also add patches to HDFS to ensure their technologies run optimally with it.

Original title and link: Defending Hadoop’s HDFS - Cloudera Version (NoSQL database©myNoSQL)

via: http://www.cloudera.com/blog/2012/07/why-we-build-our-platform-on-hdfs/

In reply to the attack to HDFS, Eric Baldeschwieler emphasizes the pros of HDFS:

• Extreme low cost per byte
• Very high bandwidth to support MapReduce workloads
• Data reliability

but also the state of the HDFS competition:

• not designed for Hadoop scale
• not using commodity hardware or open source software
• not meant for MapReduce
• unproven technology

Original title and link: Defending Hadoop’s HDFS - Hortonworks Version (NoSQL database©myNoSQL)

via: http://hortonworks.com/blog/thinking-about-the-hdfs-vs-other-storage-technologies/

Derick Harris for GigaOm:

Ironically, one of Hadoop’s biggest shortcomings now is also one of its biggest strengths going forward —the Hadoop Distributed File System.

But if the growing number of options for replacing HDFS signifies anything, it’s that HDFS isn’t quite where it needs to be.

No alternatives => vendor lock-in => bad

Multiple options => proof of weaknesses => bad

Confused? I am a bit.

Original title and link: Attacking Hadoop’s HDFS: 8 Ways to Replace HDFS (NoSQL database©myNoSQL)

via: http://gigaom.com/cloud/because-hadoop-isnt-perfect-8-ways-to-replace-hdfs/