Connection properties

In order for the Ray Snowflake to work, it needs basic connection settings either placed in the environment, in a configuration file, or passed to the SnowflakeConnector.create class method.

File connection properties

Snowflake configuration file properties have the exact same names as the properties specified in Snowflake connector API. The Ray Snowflake connector will look for a connection configuration files according to a search path. The default search path is:

1. File path passed to the engine create method (see below).

2. ./.snowflake/connect.yml

3. ./.snowflake.yml

4. ~/.snowflake/connect.yml

5. ~/.snowflake.yml

6. /mnt/user_storage/.snowflake/connect.yml

7. /mnt/user_storage/.snowflake.yml

Example connect.yml

cat example_connect.yml

account: my_locator
user: me
password: my_password
warehouse: my_warehouse
private_key_file: my_api_file
database: mydb
schema: myschema

Environment connection properties

In addition to connect files, connect properties can be placed in the environment according to a naming convention. Database environment variables have the same names as the properties specified in the Snowflake connector API,except they are upper case and prefixed with SNOWFLAKE_. Any properties loaded in the connection file will be overridden by these environment variables.

Example script connect.sh

cat example_connect.sh

SNOWFLAKE_ACCOUNT=my_locator
SNOWFLAKE_USER=me
SNOWFLAKE_PASSWORD=my_password
SNOWFLAKE_WAREHOUSE=my_warehouse
SNOWFLAKE_PRIVATE_KEY_FILE=my_api_file
SNOWFLAKE_DATABSE=mydb
SNOWFLAKE_SCHEMA=myschema

Connection properties in Python

Connection properties can also be defined within code when creating a Ray Snowflake connector. Any properties loaded from files or from the environment will be overridden by parameters passed in the SnowflakeConnector.create class method.

WARNING: It is not the recommended to keep connect settings in Python code for security reasons. Put these settings in a yaml file, or place them in environment variables. If placed in config files, be sure to add the file name to .gitignore so credentials are not added into source control.

from ray_db.provider.snowflake import SnowflakeConnector

connector = SnowflakeConnector.create(
    database='my_db',
    schema='m_schema',
    user='me',
    password='my_password',
    warehouse='my_warehouse',
    private_key_file='my_api_file'
)

print(connector)

SnowflakeConnector(connect_props={'account': 'OWB55786', 'user': 'me', 'password': 'my_password', 'warehouse': 'my_warehouse', 'database': 'my_db', 'schema': 'm_schema', 'private_key_file': 'my_api_file'}, connect_fn=<function Connect at 0x7f28ce95cc10>, connection=None, _ref_count=0)

Get properties from AWS Parameter Store

In addition to local properties, AWS Parameter store properties can also be obtained by prefixing ssm: to the parameter store key name.

import boto3

#boto3.client('ssm').put_parameter(Name='user', Value='my_user')
#boto3.client('ssm').put_parameter(Name='password', Value='my_password')

connector = SnowflakeConnector.create(
    database='my_db',
    schema='my_schema',
#    user='ssm:user',
#    password='ssm:password'
)

print(connector)

SnowflakeConnector(connect_props={'account': 'OWB55786', 'user': 'egr', 'password': 'C0lumbia!', 'warehouse': 'XSMALL', 'database': 'my_db', 'schema': 'my_schema'}, connect_fn=<function Connect at 0x7f28ce95cc10>, connection=None, _ref_count=0)

Create your own connect file

The code below allows you to create your own Snowflake connection properties. Modify the constants to match your Snowflake environment. By default it saves the file in the tmp director. You can also modify this to place it in your home directory, a local directory or in a mounted share folder.

import yaml

props = dict(
    database='my_db',
    schema='my_schema',
    user='me',
    password='my_password',
    warehouse='my_warehouse',
    private_key_file='my_api_file'
)
with open('/tmp/.snowflake.yml', 'w') as f:
    yaml.dump(props, f)