Fix compatibility of Apache-Spark 2.1.0 with Python 3.6

If you are using MacOs Sierra and Homebrew like me, and you want to build comething cool with Apache-Spark and Python3, you would find the compatibility problem while using the pyspark framework. So this article is to tell you how to solve it.

The only way for now is to install Python 3.5 by using Pyenv on Homebrew since the compatibility will be fixed on Apache-Spark 2.2.0 and it hasn’t been released.

  1. Install Pyenv

    1. Supposed that you have install Apache-Spark and Python 3.6 correctly by using the homebrew

    2. brew install pyenv

    3. Change your shell-rc file

      1. add the following command at the begining of configuation

        eval “$(jenv init -)”
        
    4. Install Python 3.5 (for now the latest version is 3.5.2)

      pyenv install python 3.5.2
      
    5. Change global Python to 3.5.2 instead of 3.6.x

      pyenv global 3.5.2
      
  2. Install pyspark framework by using pip3

    1. So for now your pip would be directed to python 3.5.2

    2. Go to your $SPARK_HOME/libexec/python

    3. pip3 install -e .
      pip2 install -e.
      
    4. Then you would find that you cannot get to the $SPARK_HOME

      brew link –over-write apache-spark
      
    5. Then happy hacking with pyspark