![]() ![]() The full breakdown can be found on the configuration page. The common SSL settings are at spark.ssl namespace in Spark configuration, while Akka SSL configuration is at and HTTP for broadcast and file server SSL configuration is at. This way the user can easily provide the common settings for all the protocols without disabling the ability to configure each one individually. The user can configure the default SSL settings which will be used for all the supported communication protocols unless they are overwritten by protocol-specific settings. SSL ConfigurationĬonfiguration for SSL is organized hierarchically. To configure your cluster manager to store application data on encrypted disks. If encrypting this data is desired, a workaround is Encryption is not yet supported for the WebUI.Įncryption is not yet supported for data stored by Spark in temporary local storage, such as shuffleįiles, cached data, and other application files. ![]() Supported for the block transfer service. Spark supports SSL for Akka and HTTP (for broadcast and file server) protocols. The event log files will be created by Spark with permissions such that only the user and group have read and write access. This will allow all users to write to the directory but will prevent unprivileged users from removing or renaming a file unless they own the file or directory. The owner of the directory should be the super user who is running the history server and the group permissions should be restricted to super user group. If you want those log files secured, the permissions should be set to drwxrwxrwxt for that directory. If your applications are using event logging, the directory where the event logs go ( ) should be manually created and have the proper permissions set on it. This is useful on a shared cluster where you might have administrators or support staff who help users debug applications. Spark allows for a set of administrators to be specified in the acls who always have view and modify permissions to all the applications. On YARN, the modify acls are passed in and control who has modify access via YARN interfaces. Note that if you are authenticating the web UI, in order to use the kill button on the web UI it might be necessary to add the users in the modify acls to the view acls also. This includes things like killing the application or a task. Spark also supports modify ACLs to control who has access to modify a running Spark application. ![]() On YARN, the Spark UI uses the standard YARN web application proxy mechanism and will authenticate via any installed Hadoop filters. Note that the user who started the application always has view access to the UI. The configs and spark.ui.view.acls control the behavior of the ACLs. The javax servlet filter specified by the user can authenticate the user and then once the user is logged in, Spark can compare that user versus the view ACLs to make sure they are authorized to view the UI. A user may want to secure the UI if it has data that other users should not be allowed to see. The Spark UI can also be secured by using javax servlet filters via the spark.ui.filters setting. This secret will be used by all the Master/Workers and applications.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |