Bigdata Filesystem Support in Ubuntu
Rationale:
Hadoop HDFS is a relatively heavy weight solution written in Java which has performance issues as its stands today on lighter weight CPU architectures such as ARM.
Ubuntu should provide alternatives to Hadoop HDFS to enable use of Bigdata applications on non-x86 architectures.
Calxeda have already ported QFS to ARM (patches now upstream).
Ceph already have support for use with Hadoop (integration currently being improved with delivery targetted for late 2012).
Goal:
Package QFS for Ubuntu 12.10.
Update Ceph packaging as required to support use with Hadoop.
Update Hadoop charms to support these two options.
Blueprint information
- Status:
- Not started
- Approver:
- Dave Walker
- Priority:
- Low
- Drafter:
- Ubuntu Server
- Direction:
- Approved
- Assignee:
- James Page
- Definition:
- Approved
- Series goal:
- Accepted for raring
- Implementation:
-
Deferred
- Milestone target:
-
ubuntu-13.04-beta-2
- Started by
- Completed by
Whiteboard
User Stories:
Frank is deploying a hyperscale hadoop cluster for processing clickstream data from websites across the world. He selects QFS for its sequential write performance and native support on ARM as a backing store for Hadoop and he is able to easily deploy it using Juju and MaaS.
Jane wants to deploy a more random bigdata workload using HBase; she is easily able to deploy Ceph alongside HBase using Juju and MaaS.
Risks:
Support for ARM from Ceph or QFS proves non-performant
Test Plans:
Build support into Hadoop charms for QFS and Ceph, test ease of deployment.
Release Note:
Ubuntu 12.10 includes the Quantcast File System (QFS), a lightweight, high-performance alternative to Hadoop HDFS.
Ceph 0.5x includes support for using Ceph as an alternative to Hadoop HDFS.
Work Items
Work items:
Complete packaging of QFS in Debian and Ubuntu: TODO
[xnox] review and/or sponsor QFS packaging: TODO
Updates to Ceph packaging as required to support hadoop: TODO
Support for QFS in hadoop charms: TODO
Support for Ceph in hadoop and hbase charms: TODO
Bigtop charms for Bigtop 0.4.0 onwards as part of upstream: TODO
Look at ARM support for Apache Bigtop distribution: TODO
