cloudera: All content tagged as cloudera in NoSQL databases and polyglot persistence
Security is an enterprise feature
At Hadoop Summit, Merv Adrian (VP Gartner) has shown data about Hadoop’s adoption in the enterprise space over the last 2 years and the numbers were great (actually they weren’t even good).
Hadoop vendors are becoming more aggressive in adding features that would make Hadoop enterprise ready. In some sectors (e.g. government, financial and health services) data security is regulated and this makes security features a top priority for adopting Hadoop in these spaces.
The state of Hadoop Security
There’s a mix of activity on the open source and vendor proprietary sides for addressing the void. There are some projects at incubation stage within Apache, or awaiting Apache approval, for providing LDAP/Active Directory linked gateways (Knox), data lifecycle policies (Falcon), and APIs for processor-based encryption (Rhino). There’s also an NSA-related project for adding fine-grained data security (Accumulo) based on Google BigTable constructs. And Hive Server 2 will add the LDAP/AD integration that’s current missing.
What’s interesting to note is that many big vendors have been focusing on adding proprietary security and auditing features to Hadoop.
Cloudera’s post introducing Sentry also provides a short overview of security in Hadoop, by looking at 4 areas:
- Perimeter security: network security, firewall, and Kerberos authentication
- Data security: encryption and masking currently available through a combination of recent work in the Hadoop community and vendor solutions.
- Access security: fine grained ACL
- Visibility: monitoring access and auditing
Sentry: Role-based Access Control for Hadoop
Cloudera has announced Sentry a fine grained role-based access control solution for Hadoop meant to simplify and augment the current course-grained HDFS-level authorization available in Hadoop.
Sentry comprises a core authorization provider and a binding layer. The core authorization provider contains a policy engine, which evaluates and validates security policies, and a policy provider, which is responsible for parsing the policy. The binding layer provides a pluggable interface that can be leveraged by a binding implementation to talk to the policy engine. (Note that the policy provider and the binding layer both provide pluggable interfaces.)
At this time, we have implemented a file-based provider that can understand a specific policy file format.
According to the post, right now only Impala and Hive have bindings for Sentry. This makes me wonder how Sentry is deployed in a Hadoop cluster so other layers could take advantage of the Sentry ACL. I see such a security feature implemented very close to HDFS so it would basically work with all types of access to data stored.
For more details about Sentry, read the official post With Sentry, Cloudera Fills Hadoop’s Enterprise Security Gap.
There are also numerous rewrites of the announcement:
- Rachel King for ZDNet: Cloudera intros new authorization module for Hadoop | ZDNet
- Virginia Backaitis for CMSWire: Cloudera Delivers Sentry Security For Hadoop: Regulated Enterprises Can Now Ask Big Data Questions
- Justin Lee for TheWhir: Cloudera Introduces New Authorization Module for Hadoop
- Isaac Lopez for Dataname: Cloudera Adds a Sentry to Their Stack - Datanami
- Jordan Novet for GigaOm: Cloudera keeps sensitive data hidden from prying eyes with new authorization settings — Tech News and Analysis
- Doug Henshen for InformationWeek: Cloudera Brings Role-Based Security To Hadoop - Software -
- Nick Kolakowski for Slashdot: Cloudera’s Sentry Offers Access Security for Big Data
Tony Baer is a principal analyst covering Big Data at Ovum. ↩
Original title and link: Hadoop Security and Cloudera’s new Role Based Access Control Sentry project ( ©myNoSQL)
Original title and link: Cloudera Announces Cloudera Developer Kit, Enabling Developers to Build Hadoop Apps Faster ( ©myNoSQL)
- support for
REFRESHfor a single table
- Hints for specifying particular join strategies
- Dynamic resource management, allowing high concurrency for Impala queries
Question: if I remember correctly Impala uses a single process on each machine to execute queries.
- is it multi-threaded?
- does it do any memory/CPU management so one query is not completely exhausting any of these resources?
- what happens with the queries executing when this process fails?
Original title and link: Cloudera Impala 1.0 Release Notes and A Couple of Questions ( ©myNoSQL)
Where is MapR today?
- MapR raised a total of $59mil.
- According to John Schroeder (CEO) “92% of MapR customers pay primarely for licenses and not for ancillary services and support”.
- According to Wikibon, MapR had $23mil. revenue in 2012, 49% of which coming from services (nb: this seem to contradict the above point)
- Support for MapR installations is offered by Accenture and Booz Allen Hamilton
How will MapR use the new capital?
With the new funding, the company plans to invest in research & development, and expand into Asia.
How is MapR seeing its competitors?
John Schroeder (CEO):
“Our competitors’ model is very cash intensive and you have to wonder whether or not they’ll ever be cash-flow positive”.
Cloudera has raised until now $141mil:
- Series A: $5mil
- Series B: $6mil
- Series C: $25mil
- Series D: $40mil
- Series E: $65mil
According to this, Cloudera raised $36mil in the first 3 rounds. I couldn’t find any official data about the capital raised by Hortonworks, but the number I’ve seen in a couple of places is $50mil. So far MapR raised $59mil.
Sources for these bits:
- VentureBeat: MapR gets $30M to push Hadoop deeper into the enterprise
- AllThingsD: MapR Lands $30 Million Series C Led by Mayfield Fund - Arik Hesseldahl - Enterprise - AllThingsD
- CrunchBase: Cloudera | CrunchBase Profile
- Wikibon: Big Data Vendor Revenue And Market Forecast 2012-2017 - Wikibon
Original title and link: MapR Raises $30mil in Series C ( ©myNoSQL)
The short answer is there is only one Apache Hadoop distribution.
The long answer is that there are many distributions that include Apache Hadoop or are claiming compatibility with Apache Hadoop.
The oldest and probably most popular: Cloudera’s Distribution of Hadoop (CDH)
The 100% open source: Hortonworks Data Platform.
The prioprietary: MapR.
The blue one: IBM InfoSphere BigInsights.
There’s also the version Facebook’s running on their cluster which includes Facebook Corona: a different approach to job scheduling and resource management.
But this list is not complete as it doesn’t include appliances featuring Hadoop. In this category we have:
- Oracle’s Big Data appliance featuring Cloudera’s Distribution of Hadoop
- Netapp’s Hadooplers
- EMC Greenplum DCA
- Teradata Aster Discovery Platform featuring Hortonworks’s Hadoop Data Platform
- Data Direct Networks (DDN)
I hope I didn’t miss any important ones1. As a conclusion for this list, my question is: who is actually benefiting from all these distributions?
I left aside for now Hadoop-as-a-Service. ↩
Original title and link: How Many Hadoops? ( ©myNoSQL)