How to install parquet-tools on Ubuntu 18.04 LTS without building from source

I've seen:

• Installing parquet-tools
• Cannot compile parquet-tools
• Could not transfer artifact (): Received fatal alert: protocol_version -> [Help 1]
• maven project failed to execute "maven-thrift-plugin"
• How to install libthrift-dev on Ubuntu?

And a few more on installing thrift. I would really prefer not to build thirft and then parquet-mr from source. All I want is parquet-tools.

I'm on:

$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 18.04.5 LTS Release: 18.04 Codename: bionic $ 

Things I've tried:

• Download source from github and apache

• Try to build from source as described here and here. I get many different errors.

• Build from master or build from some release tags like 1.11.x. Got various errors, e.g.

  org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default) on project parquet-generator: Error rendering velocity resource. at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215) ... Caused by: org.apache.maven.plugin.MojoExecutionException: Error rendering velocity resource. at org.apache.maven.plugin.resources.remote.ProcessRemoteResourcesMojo.processResourceBundles (ProcessRemoteResourcesMojo.java:1246) ... Caused by: java.lang.NullPointerException at java.util.Objects.requireNonNull (Objects.java:203) ... 

• Install thrift using sudo apt-get install thrift-compiler (which installs 0.9.x, which gives compilation errors while building parquet-mr)

  [DEBUG] (f) arguments = [-c, thrift -version | fgrep 'Thrift version 0.12.0' && exit 0; echo "================================================================================="; echo "========== [FATAL] Build is configured to require Thrift version 0.12.0 =========="; echo -n "========== Currently installed: "; thrift -version; echo "================================================================================="; exit 1] 

• Try to build thrift from source, I get some errors:

  checking whether we are cross compiling... configure: error: in `/home/kash/vm_share/thrift-0.13.0': configure: error: cannot run C compiled programs. 

• Tried to look for 0.12/13.0 of pre-built thrift but can't find it. Looks like for bionic there is only 0.9.0

Please! I just want to see the meta of a parquet file on command line.

1 Answer

So I finally managed to compile from source.

TL;DR

1. Compile trift with --host=x86_64.
2. Use apache-parquet-1.11.11 tag on parquet-mr repo instead of master.
3. Update trift dependency version from 12 to 13 in parquet-mr/pom.xml and add maven central repo (codehaus is dead):

+ <repository> + <id>mvnrepository</id> + <url> + </repository> ... - <thrift.version>0.12.0</thrift.version> + <thrift.version>0.13.0</thrift.version> 

# install dependencies as described here: # install thrift from source wget -nv tar xzf thrift-0.13.0.tar.gz cd thrift-0.13.0 chmod +x ./configure ./configure --host=x86_64 --disable-libs sudo make install # build parquet-tools from source git clone cd parquet-mr git checkout apache-parquet-1.11.11 # build only parquet-tools and it's dependencies # had to skip tests because one failed mvn package -pl parquet-tools -am -Plocal -Dmaven.test.skip=true # Use java -jar parquet-tools/target/parquet-tools-*.jar --help # Or if you're lazy like me: alias parquet-tools="java -jar $(realpath ./parquet-tools/target/parquet-tools-*.jar)" parquet-tools -h