A meta-repository of big data tools
Here are the source code for the major pieces of a data science platform (hadoop, pig, wukong, storm, kafka, etc), and their essential plugins.
Clone it so you have all the source at hand -- to track development, to steal ideas from, or because you're getting on an airplane in ten minutes. The browse directory links to the most-likely-to-be-interesting directory, so you don't spend time trying to figure out if it's src/main/java/org/with/lots/of/dirs or java/src/main or what.
- It's not a link to every tool in the space -- only repos we've found useful or promising.
- It doesn't build everything from scratch, or have a complete set of dependencies. (Pull request encouraged!)
- hadoop-common -- Hadoop (Core Framework)
- hadoop-mapreduce -- Hadoop (Distributed Computation)
- hadoop-hdfs -- Hadoop (Distributed File System)
- mahout -- machine learning on Hadoop
- hive -- High-level interface to hadoop
- crunch -- data science on Hadoop
- pig -- the tool itself
- piggybank -- the official contrib set of Pig UDFs
- piggychimp -- Pig UDFs from infochimps-labs
- sounder -- Pig UDFs from Jacob Perkins (@thedatachef)
- datafu -- Pig UDFs from linkedin
- elasticsearch -- full-text datastore of joy
- hbase -- store a billion of y'know whatever
- R -- statistics, tried and true, written by statisticians (unfortunately, written by statisticians)
- Julia -- statistics, exciting and new, written by programmers (unfortunately, exciting and new)
- kafka -- real-time data delivery
- storm -- real-time data analytics
- wukong-example-data -- useful tables and interesting datasets, from country codes to UFO sightings
Tools that are needed to make the other tools work
- addressable
- bundler
- guard, guard-rspec, guard-yard
- uuidtools
- htmlentities
- oj
(other dependencies: RedCloth forgery highline jeweler json kramdown multi_json perftools.rb pry rake rb-fsevent redcarpet rspec simplecov yard)