[Tip] Spark crontab 배치 script 에서 실행 안되는 문제 해결

https://stackoverflow.com/questions/28700624/spark-submit-scheduling-in-cron

crontab -e 를 통해서 script 를 정기적으로 돌릴때,

#!/bin/sh

spark submit >> 동작안함

#!/bin/bash 

/경로/spark-submit >> 동작함

아래와 같이 bash 로 수정하고 실행시키면 동작한다.

bash is the most common shell used as a default shell for users of Linux systems. It is a spiritual descendent of other shells used throughout Unix history. Its name, bash is an abbreviation of Bourne-Again Shell, an homage to the Bourne shell it was designed to replace, though it also incorporates features from the C Shell and the Korn Shell.

/bin/sh is an executable representing the system shell. Actually, it is usually implemented as a symbolic link pointing to the executable for whichever shell is the system shell. The system shell is kind of the default shell that system scripts should use. In Linux distributions, for a long time this was usually a symbolic link to bash, so much so that it has become somewhat of a convention to always link /bin/sh to bash or a bash-compatible shell. However, in the last couple of years Debian (and Ubuntu) decided to switch the system shell from bash to dash - a similar shell - breaking with a long tradition in Linux (well, GNU) of using bash for /bin/sh. Dash is seen as a lighter, and much faster, shell which can be beneficial to boot speed (and other things that require a lot of shell scripts, like package installation scripts).

If your script requires features only supported by bash, use #!/bin/bash.

But if at all possible, it would be good to make sure your script is POSIX-compatible, and use #!/bin/sh, which should always, quite reliably, point to the preferred POSIX-compatible system shell in any installation.