Installation

This page documents trendulo.com's initial single-server installation

Filesystem

The following will each be mounted EBS volumes

• /trendulo (2GB)
• /tweets (22GB)
• /mnt0ebs (30GB)
• /mnt1ebs1 (30GB)

Also add an ephemeral disk for logs

• /mnt0 (394GB)

Notes: After creating and attaching a new EBS volume to our instance, run the following to create the filesystem, replacing the device name as appropriate.

sudo mkfs -t ext4 /dev/xvdh

/trendulo

We will install and configure Accumulo, Hadoop, and Zookeeper in this directory

• cd /trendulo
• wget http://people.apache.org/~ecn/1.4.0rc6/accumulo-1.4.0-dist-RC6.tar.gz
• wget http://archive.cloudera.com/cdh/3/hadoop-0.20.2-cdh3u2.tar.gz
• wget http://archive.cloudera.com/cdh/3/zookeeper-3.3.3-cdh3u2.tar.gz
• Extract all tar.gz files
• cd /trendulo/accumulo-1.4.0/conf
• cp examples/1GB/native-standalone/* .
• Edit accumulo-env.sh
  • JAVA_HOME=/usr/lib/jvm/java-6-sun-1.6.0.24
  • HADOOP_HOME=/trendulo/hadoop-0.20.2-cdh3u2
  • ZOOKEEPER_HOME=/trendulo/zookeeper-3.3.3-cdh3u2
  • ACCUMULO_LOG_DIR=/mnt0/logs/accumulo
• Edit accumulo-site.xml
  • set logger.dir.walog to /mnt0ebs/walogs
• cd /trendulo/hadoop-0.20.2-cdh3u2/conf
• Edit core-site.xml
  • set fs.default.name to _hdfs://localhost
  • set hadoop.tmp.dir to /trendulo/tmp/hadoop-tmp
• Edit mapred-site.xml
  • set mapred.job.tracker to localhost:9001
• Edit hadoop-env.sh
  • set JAVA_HOME to /usr/lib/jvm/java-6-sun-1.6.0.24/
  • set HADOOP_LOG_DIR to /mnt0/logs/hadoop
• Edit hdfs-site.xml
  • set dfs.data.dir to /mnt0ebs/hdfs/data,/mnt1ebs/hdfs/data
  • set dfs.name.dir to /mnt1ebs/hdfs/name/
• cd /trendulo/zookeeper-3.3.3-cdh3u2/conf
• Edit zoo.cfg
  • set dataDir to _/trendulo/zookeeper-data
  • set maxClientCnxns to 100
• Edit log4j.properties
  • set zookeeper.root.logger to DEBUG,ROLLINGFILE
  • set zookeeper.log.dir to /mnt0/logs/zookeeper

Environment

Set the following in .bash_profile

if [ -f ~/.bashrc ]; then
    . ~/.bashrc
fi

CLOUD_HOME=/trendulo

HADOOP_VER=0.20.2
ACCUMULO_VER=1.4.0
ZK_VER=3.3.3
CDH_VER=cdh3u2

export HADOOP_HOME=${CLOUD_HOME}/hadoop-${HADOOP_VER}-${CDH_VER}
export ACCUMULO_HOME=${CLOUD_HOME}/accumulo-${ACCUMULO_VER}
export ZOOKEEPER_HOME=${CLOUD_HOME}/zookeeper-${ZK_VER}-${CDH_VER}
export JAVA_HOME=/usr/lib/jvm/java-6-sun-1.6.0.24

PATH=$PATH:${HADOOP_HOME}/bin
PATH=$PATH:${ACCUMULO_HOME}/bin
PATH=$PATH:${ZOOKEEPER_HOME}/bin
PATH=$PATH:${JAVA_HOME}/bin

export PATH