Setting up RDFS for CLion development

After completing the following process, CLion should be able to find all the header files and class definitions it needs to index the RDFS codebase. You would be able to click on any identifier (e.g. class names, function names, etc.) and jump to their declarations or definitions. On macOS, CLion wouldn't be able to build the project though, since there are some Linux-specific functions (e.g. be64toh) that don't exist on macOS. On Linux, you might well be able to build the entire project, but running anything meaningful would require more dependencies to be installed. The point of this write-up is not to encourage/enable you to run and debug in your host environment, but to make it easier to navigate and develop the codebase with the help of CLion. You should always run and debug things in the virtual machine.

At a high level, what we are going to do is the following:

• Install dependencies
  • protobuf compiler
  • ZooKeeper C client library
  • Boost
  • ASIO
  • GoogleTest & GoogleMock
• Ask CLion to invoke the protobuf compiler and generate proto classes.
• Reload CLion so that it can re-index everything from a clean slate.

In particular, note that there's no need to install hadoop. You may see (seemingly confusing) lines like using hadoop::hdfs::xxx at more than a few places in the codebase. Don't worry! It happens to be the case that the hadoop/hdfs developers defined the HDFS protos under the hadoop::hdfs namespace. The using statements are merely importing the C++ classes generated by the proto compiler, which we will be installing and invoking.

macOS

Prerequisites

• Make sure you have Homebrew. (MacPorts might work as well.)

Installing dependencies

Install the proto compiler

• brew install protobuf
• If you can find header files in /usr/local/include/google/protobuf/, then you're probably good. If not, you may want to try compiling and installing from source, as outlined in the corresponding section for Ubuntu systems below. Before you do that, make sure to do brew uninstall --force protobuf so that the system doesn't end up confused with multiple versions of protobuf.

Install the ZooKeeper C client library

• brew install zookeeper
• If you can find /usr/local/include/zookeeper/zookeeper.h, then you're probably good. If not, you may want to try compiling and installing from source, as outlined in the corresponding section for Ubuntu systems below. Before you do that, make sure to do brew uninstall --force zookeeper so that the system doesn't end up confused with multiple versions of zookeeper.

Install the Boost library

• brew install boost
• To check that Boost is properly installed, click on the CMake tab on the bottom of the window after CLion has loaded the project. If you see something along the lines of -- Boost version: x.y.z, then you are probably good.

Install the ASIO library

• ASIO might have been installed as part of Boost installation. If you can find /usr/local/include/asio.hpp, then you are probably good.
• If not, brew install asio

Install GoogleTest and GoogleMock

• Run the following commands from where you want to install google test. It doesn't have to be a system-wide installation.
  • wget https://github.com/google/googletest/archive/release-1.8.0.zip
  • unzip release-1.8.0.zip
  • rm release-1.8.0.zip
  • cd googletest-release-1.8.0/googletest
  • cmake .
  • make
  • cd ../googlemock
  • cmake .
  • make

Make CLion happy

• Open the project in CLion (File > Import Project). Be careful not to allow CLion to overwrite the existing CMakeLists.txt file in RDFS.
• Add GoogleTest and GoogleMock. Preferences > Build, Execution, Deployment > CMake
  • Add to environment variables:
    • Name: GTEST_ROOT
    • Value: <path-to-googletest>
    • Name: GMOCK_ROOT
    • Value: <path-to-googlemock>
• Ask CLion to build the project (Run > Build). This will likely fail, but that's fine. The point of this step is to invoke the proto compiler and generate C++ code from the proto definitions.
• File > Reload CMake Project
• Enjoy. (If the previous step didn't do the trick, try File > Invalidate Caches / Restart.)

Notes

If CLion is still unhappy after the above process, you may try some of the following workarounds that people have reported:

• I (Eddie) had to make sure CLion is using the cmake installed on my computer (NOT bundled cmake)
  • To install CMake, do brew install cmake. If you already have CMake, upgrade it with brew upgrade cmake.
  • Check Preferences > Build, Execution, Deployment > Toolchains
• I (Jessica) had to install the Protobuf Support plugin to coax CLion into generating and recognizing proto classes.
  • To install the plugin, go to Preferences > Plugins > Browse repositories, and search for Protobuf Support.

Ubuntu

Some of the commands might need to be run with superuser privilege.

Prerequisites

apt install -y git build-essential cmake automake autoconf libtool libcppunit-dev

Installing dependencies

Install the proto compiler

• git clone git@github.com:google/protobuf.git
• cd protobuf
• ./autogen.sh
• ./configure --prefix=/usr
• make
• Optional (if you are as paranoid as me): make check
• make install

Install the ZooKeeper C client library

• wget http://www.gtlib.gatech.edu/pub/apache/zookeeper/zookeeper-3.4.9/zookeeper-3.4.9.tar.gz
• tar -xf zookeeper*.tar.gz
• rm zookeeper*.tar.gz
• cd zookeeper-3.4.9/src/c
• ./configure
• make
• make install

Install the Boost library

• apt install libboost-all-dev

Install the ASIO library

• apt install libasio-dev

Install GoogleTest and GoogleMock

• Run the following commands from where you want to install google test. It doesn't have to be a system-wide installation.
  • wget https://github.com/google/googletest/archive/release-1.8.0.zip
  • unzip release-1.8.0.zip
  • rm release-1.8.0.zip
  • cd googletest-release-1.8.0/googletest
  • cmake .
  • make
  • cd ../googlemock
  • cmake .
  • make

Make CLion happy

• Open the project in CLion (File > Import Project). Be careful not to allow CLion to overwrite the existing CMakeLists.txt file in RDFS.
• Preferences > Build, Execution, Deployment > CMake
  • Add to environment variables:
    • Name: GTEST_ROOT
    • Value: <path-to-googletest>
    • Name: GMOCK_ROOT
    • Value: <path-to-googlemock>
• Ask CLion to build the project (Run > Build). This will likely fail, but that's fine. The point of this step is to invoke the proto compiler and generate C++ code from the proto definitions.
• File > Reload CMake Project
• Enjoy. (If the previous step didn't do the trick, try File > Invalidate Caches / Restart.)

Windows

• Chocolatey is a package installer comparable to Homebrew.

TODO: Windows folks, please summarize your experience here :).

Common

Please do the following regardless of the system you are running:

• See the lint page CLion cpplint plugin installation instructions.