Spark Thrift Server does not work

dsnospam1 2018-04-12 21:09:48 UTC #2

Hi Bitnami,

The error appears relate to the metastore not being created correctly in derby. Perhaps an issue with hive-site.xml?

jota 2018-04-13 09:18:08 UTC #3

Thank you for using Bitnami. In order to reproduce the issue, could you please let us know the steps you performed in the instance before running the command?

Did you modify any configuration files?
Did you install any package in the system?
Are you following any documentation guide that we can check?

Thanks

dsnospam1 2018-04-13 14:04:17 UTC #4

Hi Jota,

No sir, I didn't modify any configuration files or install any package in the system (other than updates). This occurred on a fresh install of Ubuntu 16.04 LTS.

Regarding documentation, all I've done to try to isolate is Google searches, primarily on the site Stackoverflow

"spark sql thrift server hive metastore_db derby org.datanucleus site:stackoverflow.com"

The only thing I found that seems remotely close to isolating the issue is this URL:

jota 2018-04-16 08:15:28 UTC #5

Hi @dsnospam1,

I just reproduced the issue you reported in your first post when using a frest Bitnami Hadoop instance. In your case, you posted a warning message that should not break the application, but I found a different error in the log file

2018-04-16 08:08:50 WARN  NativeCodeLoader:60 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
java.lang.ClassNotFoundException: org.apache.spark.sql.hive.thriftserver.HiveThriftServer2
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at org.apache.spark.util.Utils$.classForName(Utils.scala:235)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:836)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Failed to load main class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.
You need to build Spark with -Phive and -Phive-thriftserver.

I just created a task on our side and the team will check it and prioritize. We will update this case when we have more information.

Sorry for the inconvenience

dsnospam1 2018-04-16 17:20:53 UTC #6

Hi Jota,

Thank you very much for the update. For my work, it is critical that I am able to start the spark thrift server with no errors. I eagerly look forward to the Bitnami team's fix for this.

dsnospam1 2018-04-16 18:01:30 UTC #7

Hi Jota,

One additional note. You may want to look at my previous post about missing jars from the Spark installation.

It seems there are some existing configuration issues with Spark on the Bitnami Hadoop installer. Again, I look forward to the resolution of this issue.

Thanks.

abjimenez 2018-04-17 12:42:33 UTC #8

Hi @dsnospam1,

Thanks for the note. As @jota, said we will update this thread when we have more information.

Best regards,
Andrés Bono

dsnospam1 2018-04-23 14:00:14 UTC #9

Hi Bitnami,

I just wanted to check in to see if there were any updates on this topic? My BI tool can only connect to Hive via ODBC (not JDBC) and since Bitnami does not include an ODBC Hive connector in its installer, my only option is to get the Spark Thrift JDBC/ODBC server to work, and it doesn't work currently,

Thanks!

abjimenez 2018-04-24 07:47:18 UTC #10

Hi @dsnospam1,

Unfortunately this is still pending, we have at task included in our backlog. We will update this thread as soon as we have more news.

Best regards,
Andrés Bono

marcos 2018-04-24 08:29:43 UTC #11

Hi @dsnospam1,

We have found a workaround.

Spark is offered in two different formats, pre-bundled with Hadoop libraries and with a user-provided Hadoop installation.
We were including the second one (the one to be used with a user-provided Hadoop), which does not include a specific list of libraries.
Ideally, all of these libraries should be provided with Hadoop, however, some of these (like the ones required for Pyspark or Spark's Thrift server) are not, as their source is included within Spark itself, and are disabled at sources build time.

Therefore, in order to fix the issues for both threads, please follow these steps:

Download Spark 2.3 w/ Hadoop libs here: https://www.apache.org/dyn/closer.lua/spark/spark-2.3.0/spark-2.3.0-bin-hadoop2.7.tgz
```
wget http://apache.mirrors.ovh.net/ftp.apache.org/dist/spark/spark-2.3.0/spark-2.3.0-bin-hadoop2.7.tgz
```
Extract the tarball:
```
tar xzf spark-2.3.0-bin-hadoop2.7.tgz
```
Move all libraries inside the jars/ directory to your Bitnami Spark installation's jars directory:
```
sudo mv spark-2.3.0-bin-hadoop2.7/jars/* /opt/bitnami/hadoop/spark/jars
```

Set appropriate ownership for the files:

sudo chown -R hadoop:root /opt/bitnami/hadoop/spark/jars

Once you have done this, you should be properly able to start the Thrift server with the command you used above.

As we also mentioned above, this should fix the issue mentioned in "(Py)spark doesn’t work on Hadoop Stack".

EDIT: Ensure that /opt/bitnami/hadoop/spark/jars should now contain all files that were included in spark-2.3.0-bin-hadoop2.7/jars.

Unfortunately we don't have an ETA for when we will fix this in Bitnami stacks, but hopefully this should be enough for you to be able to continue doing your work on the Bitnami Hadoop stack.

Please let us know if you have any questions.

Best regards,
Marcos

marcos 2018-04-24 12:07:20 UTC #12

Hi,

Apparently following the instructions above make Spark not work properly with the Bitnami Hadoop, for instance, it is not able to read files in HDFS.

We just wanted to let you know that we're looking into it. Unfortunately we're unable to provide an ETA for when it will be fixed.

Best regards,
Marcos

dsnospam1 2018-04-24 13:47:55 UTC #13

Hi Marcos,

Thank you for your reply. I knew that Bitnami shipped without the hive jars, so I did the very thing you mentioned prior to starting the thrift server. After examining look at the logs, this issue appears to be with the DERBY metastore that is included in the Bitnami stack..

To start the thrift server is actually a 3 step process:

Step 1:Start master (This works)
$ sudo /opt/bitnami/hadoop/spark/sbin/start-master.sh

Step 2: Start slave (This also works)
$ sudo /opt/bitnami/hadoop/spark/sbin/start-slave.sh spark://hostname:7077

Step 3 - Start Thrift server (This does NOT work)
$ sudo /opt/bitnami/hadoop/spark/sbin/start-thriftserver.sh --hiveconf hive.server2.thrift.port=10001 hive.server2.transport.mode=binary

Logs are generated for each step per the terminal in

sudo /opt/bitnami/hadoop/spark/logs

The Thrift server log is the one that shows the error related to the Hive metastore:

WARN metastore.HiveMetaStore: Retrying creating default database after error: Error creating transactional connection factory
javax.jdo.JDOFatalInternalException: Error creating transactional connection factory
at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:587)
at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:788)
.....

Derby as a metastore is very limited (only one connection at a time). Some Hadoop users have "switched" from using
Derby to MySQL or Postgres for the Hive metastore.

Has the team looked into fixing the Derby metastore issue, or possibly testing MySQL or PostgresQL as a workaround?

Thanks!

dsnospam1 2018-04-24 20:37:07 UTC #14

Hi @marcos,

Thanks again for your reply. Since ThriftServer is not working and there isn't an ETA, I was wondering if the Apache HiveODBC connector might be an option as a workaround?

I might be willing to try this - but I don't know how to configure it with the Bitnami stack.

Thanks!

dsnospam1 2018-04-24 20:38:12 UTC #15

Hi @marcos,

To start the thrift server is actually a 3 step process:

Step 1:Start master (This works)
$ sudo /opt/bitnami/hadoop/spark/sbin/start-master.sh

Step 2: Start slave (This also works)
$ sudo /opt/bitnami/hadoop/spark/sbin/start-slave.sh spark://hostname:7077

Step 3 - Start Thrift server (This does NOT work)
$ sudo /opt/bitnami/hadoop/spark/sbin/start-thriftserver.sh --hiveconf hive.server2.thrift.port=10001 hive.server2.transport.mode=binary

Logs are generated for each step per the terminal in

sudo /opt/bitnami/hadoop/spark/logs

The Thrift server log is the one that shows the error related to the Hive metastore:

Derby as a metastore is very limited (only one connection at a time). Some Hadoop users have "switched" from using
Derby to MySQL or Postgres for the Hive metastore.

Has the team looked into fixing the Derby metastore issue, or possibly testing MySQL or PostgresQL as a workaround?

Thanks!

marcos 2018-04-25 09:23:42 UTC #16

Hi @dsnospam1,

That is happening because Spark is misconfigured.

Please follow these steps inside /opt/bitnami/hadoop/spark:

Remove derby-10.12.1.1.jar:
```
$ rm jars/derby-10.12.1.1.jar
```

Edit conf/spark-env.sh. Remove everything after the line ". /opt/bitnami/scripts/setenv.sh" (but not the line itself) and add the following:

SPARK_DIST_CLASSPATH=/opt/bitnami/hadoop/hive/derby/lib/*
export SPARK_DIST_CLASSPATH
HADOOP_CONF_DIR=/opt/bitnami/hadoop/conf
export HADOOP_CONF_DIR

Create conf/spark-defaults.conf:

spark.hadoop.javax.jdo.option.ConnectionURL jdbc:derby://localhost:1527//opt/bitnami/hadoop/hive/var/lib/metastore_db;create=true
spark.hadoop.javax.jdo.option.ConnectionDriverName org.apache.derby.jdbc.ClientDriver
hive.metastore.warehouse.dir /user/hive/warehouse

After all that, you should be able to properly run Spark's Thrift service.

dsnospam1 2018-04-25 20:55:02 UTC #17

Hi @marcos,

Thank you very much for your solution to this issue. I have made the configuration corrections per the steps you outlined, and the Thrift Server now starts successfully.

Will these configuration corrections be implemented in future Hadoop releases?

Thanks again!

marcos 2018-04-26 06:58:03 UTC #18

Hi @dsnospam1,

Yes, in fact, we have already released a new version of the Bitnami Hadoop stack which implement these fixes.

Best regards,
Marcos

dsnospam1 2018-04-28 15:28:41 UTC #19

Hi @marcos

After installing 3.1.0-3 on a separate machine, unfortunately, the Hive metastore will not start after the first reboot. For example, everything ran fine on the install and the services started, but after rebooting, the Hive metastore is the only service that will not start.

I believe this is because the hive.merastore.uri server and port is not being defined in the /opt/bitnami/hadoop/spark/conf/hive-site.xml file. Please see Hive Documentation:
https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin

It indicates that the hive.metastore.uris for the Thrift metastore server should be defined as:
thrift: host_name port

I believe the reason the manual fix worked in my 3.0,0-0 installation is because I had already created a hive-site.xml file in /opt/bitnami/hadoop/spark/conf with the following syntax:

configuration
property
name hive.metastore.uris /name
value thrift://localhost:9083 /value
/property
/configuration

So, in short:
1) My old installation with the 3.0 manual fix,for the thrift server, and which includes a /opt/bitnami/hadoop/spark/conf/hive-site.xml file defining the hive.metastore.uris, works.

2) The 3.1.0-3 installation Hive metastore does not work after reboot.

Thanks.

dsnospam1 2018-04-28 15:38:00 UTC #20

Hi @marcos,

One correction - this is the second URL I meant to reference above:

Thanks.

dsnospam1 2018-04-28 21:00:12 UTC #21

Hi @marcos,

On 3.1.0-3, I tested a new install by adding a hive-site-xml file to the /opt/bitnami/hadoop/spark/conf directory after the install, but before the first reboot:

/opt/bitnami/hadoop/spark/conf/hive-site.xml

<img src="/uploads/default/original/3X/3/3/330cd2a0adf2a096ec7f396b0c5a61d830d62dcd.png" width="623" height="171">

This fixed the "Hive metastore not starting" issue upon reboot that is a current configuration issue in 3.1.0-3. Pyspark works fine, and the Thrift Server works fine.

Please consider adding this fix to address the issue in 3.1.0-3 mentioned previously regarding hive metastore not starting upon reboot.

Thanks.

Bitnami provides free all-in-one installers, virtual machines and cloud images for popular open source applications. Get them now!