Update of Apache Spark and Python 3.6 compatibility

Now the Apache-Spark has released two stable version, 2.1.1 and 2.2.0. The compatibility problem has been solved, you don’t need to try pyenv anymore. Just install apache-spark from home-brew then you may use it.

But if you are going to use python3.6 and your default python in terminal is python2.7, you would find that the REPL would still be the python2.7 version.

In order to fix that, you should input

sudo PYSPARK_PYTHON=ptipython3 pyspark

As you can see, I am using ptipython3 as my Python 3 REPL. You can use the default interpreter by changing it to python3 and that will be good for you to use.

Ptpython is a better python interpreter, here is the link:

[github.com/jonathanslenders/ptpython