Getting Started With Analytics 2.0

Initial Configuration

• Run the configuration utility:
  python user_exp.py

This will help you set up the following:

• Delta Share source configuration
• MS SQL Server target configuration
• Scheduling preferences

Data Synchronization

After configuration, you have two options to run the data sync:

1. Direct Execution python
   delta_share_to_azure_panda.py
   OR
2. Scheduled Execution
   If configured during setup, the job will run automatically according to your cron schedule.

Delta Share Configuration

1. Create a new file named config.share with your Delta Share credentials in JSON format.

2. Get required fields:
   Note: These details can be obtained from the Procore Analytics web application.
   • ShareCredentialsVersion: Version number (currently 1).
   • BearerToken: Your Delta Share access token.
   • Endpoint: Your Delta Share endpoint URL.
   • Save the file in a secure location.
3. When configuring the data source, you'll be asked to provide:
   • List of tables (comma-separated).
   • Leave blank to sync all tables.
   • Example: `table1, t able2,  table3`.
   • Path to your `config.share` file.

MS SQL Server Configuration

You'll need to provide the following MS SQL Server details:

• database
• host
• password
• schema
• username

SSIS Configuration

1. Using  the command line, navigate to the folder by entering 'cd <path to the folder>'.
2. Install required packages using 'pip install -r requirements.txt' or 'python -m pip install -r requirements.txt'.
3. Open SSIS and create a new project.
4. From SSIS Toolbox, drag and drop 'Execute Process Task' activity.
   
   
5. Double-click on 'Execute Process Task' and navigate to Process tab.
6. In 'Executable', enter the path to python.exe in python installation folder.
7. In 'WorkingDirectory' enter a path to the folder containing  the script you want to execute (without script file name).
8. In 'Arguments' enter the name of the script 'delta_share_to_azure_panda.py' you want to execute with the .py extension and save.
   
   
    
9. Click on 'Start' button in upper pane:
   
10. During the execution of the task, output of the Python console is displayed in the external console window.
11. Once the task is done it will display a green tick:
    
    

Prepare the Package

1. Create a new file named config.share with your Delta Share credentials in JSON format.

2. Get required fields.
   Note: These details can be obtained from the Procore Analytics web application.
   • ShareCredentialsVersion: Version number (currently 1).
   • BearerToken: Your Delta Share access token.
   • Endpoint: Your Delta Share endpoint URL.
3. Download and extract the package.
4. Unzip the package to a directory of your choice.
5. Copy the *.share Delta Sharing profile file into the package directory for easy access.
   

Install Dependencies

1. Open a terminal in the package directory.
2. Run the following command to install the dependencies:
3. pip install -r requirements.txt
   

Generate Configuration

1. Generate the config.yaml file by running python user_exp.py:
   This script helps to generate the config.yaml file that contains necessary credentials and settings.
2. When configuring the data source, you'll be asked to provide:
   • List of tables (comma-separated).
   • Leave blank to sync all tables.
     Example: `table1, table2, table3`.
   • Path to your `config.share` file.
3. For the first time, you will provide your credentials like Delta Share source configuration location, tables, database, host and etc.
   Note: Afterwards, you may reuse or update the config manually or by the running python user_exp.py again.

Configure Cron Jobs and Immediate Execution (Optional)

1. Decide whether to set up a cron job for automatic execution.
2. Provide a cron schedule:
   • Format: * * * * * (minute, hour, day-of-month, month, day-of-week).
   • Example for daily execution at 2 AM: 0 2 * * *
   • To check scheduling logs, the file 'procore_scheduling.log' will be created as soon as scheduling is set up.

You can also check scheduling by running in terminal command:

For Linux and MacOs:    

To edit/delete - edit scheduling cron by using:

• After running the command above, you should see something similar to:
• 2 * * * * /Users/your_user/snowflake/venv/bin/python /Users/your_user/snowflake/sql_server_python/connection_config.py 2>&1 | while read line; do echo "$(date) - $line"; done >> /Users/your_user/snowflake/sql_server_python/procore_scheduling.log # procore-data-import
• You also can adjust schedule cron or delete the whole line to stop it running by schedule.

For Windows:

• Check the schedule task is created:
  ```powershell
  schtasks /query /tn "ProcoreDeltaShareScheduling" /fo LIST /v
  ```
• To edit/delete - scheduling task:
  Open the Task Scheduler:
  • Press Win + R, type taskschd.msc, and press Enter.
  • Navigate to the scheduled tasks.
  • In the left pane, expand the Task Scheduler Library.
  • Look for the folder where your task is saved (e.g., Task Scheduler Library or a custom folder).
• Find your task:
  • Look for the task name ProcoreDeltaShareScheduling.
  • Click on it to view its details in the bottom pane.
• Verify its schedule:
  • Check the Triggers tab to see when the task is set to run.
  • Check the History tab to confirm recent runs.
• To delete task:
  • Delete task from the GUI.

Immediate Execution question:

• Option to run script for copying data immediately after configuration.
• After generating the config.yaml, the CLI is ready to be run anytime independently, by running script for copying data, depending on your package. See examples below:
  python delta_share_to_azure_panda.py
  OR
  python delta_share_to_sql_spark.py
  OR
  python delta_share_to_azure_dfs_spark.py

Execution and Maintenance

Common Issues and Solutions

1. Cron Job Setup:
   • Ensure system permissions are correctly configured.
   • Check system logs if the job fails to run.
   •  Verify the script delta_share_to_azure_panda.py has execute permissions.
2. Configuration File:
   • Ensure the file config.yaml is in the same directory as the script.
   • Backup the file before making changes.

Support

For additional help:

1. Review script logs for detailed error messages.
2. Double-check the config.yaml file for misconfigurations.
3. Contact your system administrator for permission-related issues.
4. Reach out to Procore support for issues related to Delta Share access.
5. Review log for failed tables: failed_tables.log.

Notes

1. Always backup your configuration files before making changes.
2. Test new configurations in a non-production environment to prevent disruptions.

Prepare the Package

1. Create a new file named config.share with your Delta Share credentials in JSON format.

2. Get required fields.
   Note: These details can be obtained from the Procore Analytics web application.
   • ShareCredentialsVersion: Version number (currently 1).
   • BearerToken: Your Delta Share access token.
   • Endpoint: Your Delta Share endpoint URL.
3. Download and extract the package.
4. Unzip the package to a directory of your choice.
5. Copy the *.share Delta Sharing profile file into the package directory for easy access.
   

Install Dependencies

1. Open a terminal in the package directory.
2. Run the following command to install the dependencies:
3. pip install -r requirements.txt
   

Generate Configuration

1. Generate the config.yaml file by running python user_exp.py:
   This script helps to generate the config.yaml file that contains necessary credentials and settings.
2. When configuring the data source, you'll be asked to provide:
   • List of tables (comma-separated).
   • Leave blank to sync all tables.
     Example: `table1, table2, table3`.
   • Path to your `config.share` file.
3. For the first time, you will provide your credentials like Delta Share source configuration location, tables, database, host and etc.
   Note: Afterwards, you may reuse or update the config manually or by the running python user_exp.py again.

Configure Cron Jobs and Immediate Execution (Optional)

1. Decide whether to set up a cron job for automatic execution.
2. Provide a cron schedule:
   • Format: * * * * * (minute, hour, day-of-month, month, day-of-week).
   • Example for daily execution at 2 AM: 0 2 * * *
   • To check scheduling logs, the file 'procore_scheduling.log' will be created as soon as scheduling is set up.

You can also check scheduling by running in terminal command:

For Linux and MacOs:    

To edit/delete - edit scheduling cron by using:

• After running the command above, you should see something similar to:
• 2 * * * * /Users/your_user/snowflake/venv/bin/python /Users/your_user/snowflake/sql_server_python/connection_config.py 2>&1 | while read line; do echo "$(date) - $line"; done >> /Users/your_user/snowflake/sql_server_python/procore_scheduling.log # procore-data-import
• You also can adjust schedule cron or delete the whole line to stop it running by schedule.

For Windows:

• Check the schedule task is created:
  ```powershell
  schtasks /query /tn "ProcoreDeltaShareScheduling" /fo LIST /v
  ```
• To edit/delete - scheduling task:
  Open the Task Scheduler:
  • Press Win + R, type taskschd.msc, and press Enter.
  • Navigate to the scheduled tasks.
  • In the left pane, expand the Task Scheduler Library.
  • Look for the folder where your task is saved (e.g., Task Scheduler Library or a custom folder).
• Find your task:
  • Look for the task name ProcoreDeltaShareScheduling.
  • Click on it to view its details in the bottom pane.
• Verify its schedule:
  • Check the Triggers tab to see when the task is set to run.
  • Check the History tab to confirm recent runs.
• To delete task:
  • Delete task from the GUI.

Immediate Execution question:

• Option to run script for copying data immediately after configuration.
• After generating the config.yaml, the CLI is ready to be run anytime independently, by running script for copying data, depending on your package. See examples below:
  python delta_share_to_azure_panda.py
  OR
  python delta_share_to_sql_spark.py
  OR
  python delta_share_to_azure_dfs_spark.py

Execution and Maintenance

Common Issues and Solutions

1. Cron Job Setup:
   • Ensure system permissions are correctly configured.
   • Check system logs if the job fails to run.
   •  Verify the script delta_share_to_azure_panda.py has execute permissions.
2. Configuration File:
   • Ensure the file config.yaml is in the same directory as the script.
   • Backup the file before making changes.

Support

For additional help:

1. Review script logs for detailed error messages.
2. Double-check the config.yaml file for misconfigurations.
3. Contact your system administrator for permission-related issues.
4. Reach out to Procore support for issues related to Delta Share access.
5. Review log for failed tables: failed_tables.log.

Notes

1. Always backup your configuration files before making changes.
2. Test new configurations in a non-production environment to prevent disruptions.

Switch to the Data Factory Experience

1. Navigate to your Microsoft Fabric Workspace.
   
    
2. Select New, then choose Dataflow Gen2.
   

Configure the Dataflow

1. Go to the dataflow editor.
2. Click Get Data and select More.
3. Under New source, select Delta Sharing Other as the data source.
   
4. Enter the following details:
   • URL: From your Delta Sharing configuration file.
   • Bearer Token: Found in your config.share file.
     
5. Click Next and select the desired tables.
6. Click Create to complete the setup.

Perform Data Transformations

After configuring the dataflow, you can now apply transformations to the shared Delta data. Choose your Delta Sharing Data option from the list below:

• Add Data Destination
• Create/Open Lakehouse

Add Data Destination

1. Go to Data Factory.
2. Click Add Data Destination.
3. Select Lakehouse as the target and click Next.
4. Choose your destination target and confirm by clicking Next.

Create-Open Lakehouse

1. Create/open yourLakehouse and click Get data.
2. Select New Dataflow Gen2.
3. Click Get Data, then More and find Delta Sharing.
4. Enter the URL bearer token from your config.share file and then select Next.
5. Choose your data/table/s to download and click Next.
6. After these manipulations, you should have all the selected data in your Fabric Lakehouse.

Validation and Monitoring

Test your data pipelines and flows to ensure smooth execution.Use monitoring tools within Data
Factory to track progress and logs for each activity.

Set Up Configuration

1. Create the config.yaml file and define the configuration in the following structure
   source_config:
       config_path: path/to/your/delta-sharing-credentials-file.share
   tables: # Optional - Leave empty to process all tables
       - table_name1
       - table_name2
   target_config:
       lakehouse_path: path/to/your/fabric/lakehouse/Tables/ # Path to the Fabric Lakehouse

Set Up Your Lakehouse

1. Open your Microsoft Fabric workspace.
2. Navigate to your Lakehouse and click Open Notebook, then New Notebook.
3. If you don’t know the value in config.yaml#lakehouse_path, you can copy it from the screen.
4. Click the the ellipsis on Files, and select Copy ABFS path:
   
   
   

3. Copy code of ds_to_lakehouse.py and paste into notebook window (Pyspark Python):

The next step is to upload your own config.yaml and config.share into the Resources folder of the Lakehouse. You can create your own directory or use a builtin directory (already created for resources by Lakehouse):

The example below shows a standard builtin directory for a config.yaml file .
Note: Make sure you upload both files on the same level and for the property config_path:

4. Check the code of the notebook, lines 170-175.
The example below shows the necessary line changes:

config_path = "./env/config.yaml"

to 

config_path = "./builtin/config.yaml"

Since the files are in a builtin folder and not in a custom env, make sure to monitor your own structure of the files. You can upload them into different folders, but in such cases, update the code of the notebook to find config.yaml file properly.
 

5. Click Run cell:

Validation

• Once the job completes, verify the data has been copied successfully to your Lakehouse.
• Check the specified tables and ensure the data matches the shared Delta tables.
• Wait until the job is finished, it should copy all the data.

Prepare the Package

1. Download the required package (adls_azure_function or sql_server_azure_function).
2. Extract the package files to a local directory.
3. Place Delta Sharing file:
   • Copy your *.share Delta Sharing profile file into the extracted directory.

 Install Dependencies

1. Open a terminal in the package directory.
2. Run the following command to install the required Python dependencies:
   • pip install -r requirements.txt

Generate Configuration

1. Generate the config.yaml file by running:
   • python user_exp.py
2. The script will prompt you to enter credentials such as:
   • Tables
   • Database name
   • Host
   • Additional credentials..
3. The configuration can be reused or updated manually or by re-running python user_exp.py.

 Azure CLI Setup

1. Log in to Azure.
2. Run the following command to log in:
   az login
3. Verify Azure Account:
   • az account show
   • If the az command is not available, install the Azure CLI by following the instructions found here: Microsoft Learn.

Install Azure Functions Core Tools

Go to Microsoft Learn to for instructions on installing Azure Functions Core Tools.

Prepare the Azure Function

1. Use the Azure Portal guide to create the following:
   • A function app
   • A resource group
   • Consumption plan
   • Storage account
2. Set Custom Cron schedule (Optional).
   • Open function_app.py in an editor.
   • Locate the line: @app.timer_trigger(schedule="0 0 */8 * * *",
3. Replace the schedule with your custom Cron expression and save the file.

Deployment

1. Open a terminal in the package directory (adls_azure_function).
2. Run the following deployment command:
   • func azure functionapp publish <FunctionAppName> --build remote --python --clean
3. Replace <FunctionAppName> with the name of your Azure function app in your Azure subscription.

Validation

• Ensure the deployment is successful by checking the Azure Portal for your function app status.
• Monitor logs to verify that the function is executing as expected.

Access Data Flow Gen2

1. Log in to your Microsoft Fabric account.
2. Navigate to the workspace.
3. Select 'Data Flow Gen2' from the available options.

Configure Data Source

1. Click on 'Data from another source' to begin the configuration.
2. From the Get Data screen do the following:
   • Locate the search bar labeled 'Choose data source'.
   • Type 'delta sharing' in the search field.
   • Select Delta Sharing from the results.
     
     

Set Up Delta Share Connection

1. Enter your Delta Share credentials when prompted.
   • Ensure all required fields are completed accurately.
   • Validate the connection if possible.
2. Click 'Next' to proceed.
3. Review the list of available tables:
   • All tables you have access to will be displayed.
   • Select the desired tables for transfer.
     

Configure Data Destination

1. Click 'Add Data Destination'.
2. Select 'SQL warehouse' as your destination.
   
3. Enter SQL credentials:
   • Server details.
   • Authentication information.
   • Database specifications.
   • Verify the connection settings.

Finalize and Deploy

1. Review all configurations.
2. Click 'Publish' to deploy the data flow.
3. Wait for the confirmation message.

Verification

1. Access your SQL warehouse.
2. Verify that the data is available and properly structured.
3. Run test queries to ensure data integrity.

Troubleshooting

Common issues and solutions:

• Connection failures: Verify credentials and network connectivity.
• Missing tables: Check Delta Share permissions.
• Performance issues: Review resource allocation and optimization settings.

Initial Configuration

Run the configuration utility using python user_exp.py.

Data Synchronization

After configuration, you have two options to run the data sync: 

• Direct Execution:
  • python ds_to_snowflake.py
• Scheduled Execution
  • If configured during setup, the job will run automatically according to your Cron schedule.
  • To check scheduling logs, the file `procore_scheduling.log` will be created as soon as scheduling will set up.
  • Also, you can check scheduling by running in terminal command:

For Linux and MacOS:

• To edit/delete - edit scheduling cron by using:

  ```bash
  
  EDITOR=nano crontab -e
  
  ```

• After running the command above, you should see something similar to:

  2 * * * *
  /Users/your_user/snowflake/venv/bin/python
  
   /Users/your_user/snowflake/sql_server_python/connection_config.py
  2>&1 | while read line; do echo "$(date) - $line"; done>>
  
   /Users/your_user/snowflake/sql_server_python/procore_scheduling.log # procore-data-import 

   
• You also can adjust schedule cron or delete the whole line to stop it running by schedule.

For Windows:

• Check that the schedule task is created: 

  ```
      powershell 
  
      schtasks /query /tn "ProcoreDeltaShareScheduling" /fo LIST /v 
  
  ``` 

• To edit/delete scheduling task, open the Task Scheduler.
• Press Win + R, type taskschd.msc, and press enter. 
• Navigate to the scheduled tasks. 
• In the left pane, expand the Task Scheduler Library. 
• Look for the folder where your task is saved:
  Example: Task Scheduler Library or a custom folder.
• Find your task.
• Look for the task name: ProcoreDeltaShareScheduling. 
• Click on it to view the details in the bottom pane. 
• Verify its schedule: 
  • Check the Triggers tab to see when the task is set to run. 
  • Check the History tab to confirm recent runs. 
• To delete task: 
  • Delete task from the GUI.

Delta Share Configuration

• Creating config.share file 
• Before running the configuration utility, you need to create a config.share file with your Delta Share credentials. The file should be in JSON format: 

  { 

  "shareCredentialsVersion": 1, 
  "bearerToken": "xxxxxxxxxxxxx", 
  "endpoint": "https://nvirginia.cloud.databricks.c...astores/xxxxxx"
  
  } 

   

• Required fields: 
  • ShareCredentialsVersion: Version number (currently 1). 
  • BearerToken: Your Delta Share access token.
  • Endpoint: Your Delta Share endpoint URL.
  • These details can be obtained from the Procore web UI.
• Steps to create config.share: 
  • Create a new file named config.share.
  • Copy the above JSON template. 
  • Replace the placeholder values with your actual credentials. 
  • Save the file in a secure location. 
  • You'll need to provide the path to this file during configuration. When configuring the data source, you'll be asked to provide:
    • List of tables (comma-separated).
    • Leave blank to sync all tables.
    • Example: table1, table2, table3.
• Path to your config.share file.

Snowflake Configuration

You'll need to provide the following Snowflake details:

• Authentication (choose one): 
  • User Authentication 
    • Username 
    • Password (entered securely) 
• Key Pair Authentication
  • Username 
  • Private key file path
  • Private key file password 
• Connection Details: 
  • Account identifier 
  • Warehouse name 
  • Database name 
  • Schema name 
  • Number of concurrent threads 

Scheduling Options

The tool offers the ability to schedule automatic data synchronization.

• Cron Job Configuration 
  • Choose whether to set up a daily job 
  • If yes, provide a cron schedule 
  • Format: * * * * * (minute hour day-of-month month day-of-week)
  • Example for daily at 2 AM: 0 2 * * * 
• Immediate Execution 
  • Option to run the ds_to_snowflake.py immediately after configuration  
• File Structure

  Unset 
  ├── requirements.txt                    # Dependencies 
  ├── user_exp.py                                # Configuration utility
  ├── ds_to_snowflake.py                # Data sync script
  ├── config.yaml                                  # Generated configuration
  ├── config.share                               # Delta Share config file
  ├── procore_scheduling.log       # Log of scheduling runs 

  
  Example Usage 
  • Step 1: Install dependencies 
    $ pip install -r requirements.txt
  • Step 2: Run configuration utility 
    $ python user_exp.py 
  • Procore Analytics Cloud Connect Access 
    • This CLI will help you choose your source and destination store to access/write Procore data into Snowflake. 
    • Press Enter to Continue.
    • Enter list of tables (comma-separated), leave it blank for all tables: projects,users,tasks.
    • Enter path to config.share: /path/to/config.share.
    • Enter user name: snowflake_user.
    • What authentication type do you want to use? (user/key_pair): Enter.
    • 1 for user,
    • 2 for key-pair:
    • 1 
    • Enter password: ******** 
    • Enter Account: my_account 
    • Enter warehouse: my_warehouse 
    • Enter database name: procore_db
    • Enter schema name: procore_schema 
    • Enter number of threads: 4 
    • Do you want to configure this as a daily job on cron? (Yes/No): Yes 
    • Enter the schedule in cron format (e.g., * * * * * ): 0 2 * * * 
    • Do you want to execute the job now? (Yes/No): Yes 
  • Step 3: Manual execution (if needed)
    $ python ds_to_snowflake.py 
• Configuration Reuse
  The tool saves your configuration in the config.yaml file and offers to reuse previously stored settings: 
  • Source configuration can be reused.
  • Target (Snowflake) configuration can be reused. 
  • You can choose to update either configuration independently.

Best Practices 

• Authentication
  • Use key pair authentication when possible.
  • Regularly rotate credentials. 
  • Use minimal required permissions.
• Performance 
  • Adjust thread count based on your system capabilities. 
  • Start with a smaller subset of tables for testing.

Troubleshooting 

• Common issues and solutions: 
  • Invalid Authentication Type 
    • Ensure to select either '1' (user) or '2' (key_pair) when prompted.
• Cron Job Setup  
  • Verify you have appropriate system permissions.
  • Check system logs if the job fails to run.
  • Ensure the ds_to_snowflake.py has correct permissions. 
  • Verify the cron job setup by checking system logs:
    See `procore_scheduling.log` file.
• Configuration File
  • Located in the same directory as the script,
  • Named config.yaml.
  • Backup before making any changes.
• Support
  • Check the script's logging output. 
  • Review your config.yaml file. 
  • Contact your system administrator for permission-related issues.
  • Reach out to Procore support for Delta Share access issues.

Note: Remember to always backup your configuration before making changes and test new configurations in a non-production environment first.
 

Initial Configuration

Run the configuration utility using python user_exp.py.

This will help you set up the following:

• Delta Share Configuration
• S3 Target Configuration
• Scheduling Preferences 

Delta Share Configuration

• Creating config.share file 
• Before running the configuration utility, you need to create a config.share file with your Delta Share credentials. The file should be in JSON format: 

  { 

  "shareCredentialsVersion": 1, 
  "bearerToken": "xxxxxxxxxxxxx", 
  "endpoint": "xxxxxx"
  
  } 

• Required fields: 
  • ShareCredentialsVersion: Version number (currently 1). 
  • BearerToken: Your Delta Share access token.
  • Endpoint: Your Delta Share endpoint URL.
  • These details can be obtained from the Procore web UI.
• Steps to create config.share: 
  • Create a new file named config.share.
  • Copy the above JSON template. 
  • Replace the placeholder values with your actual credentials. 
  • Save the file in a secure location. 
  • You'll need to provide the path to this file during configuration. When configuring the data source, you'll be asked to provide:
    • List of tables (comma-separated).
    • Leave blank to sync all tables.
    • Example: table1, table2, table3.
• Path to your config.share file.

S3 Configuration

You'll need to provide the following S3 details:

• Authentication:
  • Access key
  • Secret Key
  • Bucket name without s3://
  • key - directory 

Scheduling Options

The tool offers the ability to schedule automatic data synchronization.

• Cron Job Configuration 
  • Choose whether to set up a daily job. 
  • If yes, provide a cron schedule. 
  • Format: * * * * * (minute hour day-of-month month day-of-week).
  • Example for daily at 2 AM: 0 2 * * * 
  • To check scheduling logs, the file 'procore_scheduling.log' will be created as soon as scheduling is set up.

You can also check scheduling by running in terminal command

For Linux and MacOs:    

To edit/delete - edit scheduling cron by using:

• After running the command above, you should see something similar to:
• 2 * * * * /Users/your_user/snowflake/venv/bin/python /Users/your_user/snowflake/sql_server_python/connection_config.py 2>&1 | while read line; do echo "$(date) - $line"; done >> /Users/your_user/snowflake/sql_server_python/procore_scheduling.log # procore-data-import
• You also can adjust schedule cron or delete the whole line to stop it running by schedule.

For Windows:

• Check the schedule task is created:
  ```powershell
  schtasks /query /tn "ProcoreDeltaShareScheduling" /fo LIST /v
  ```
• To edit/delete - scheduling task:
  Open the Task Scheduler:
  • Press Win + R, type taskschd.msc, and press Enter.
  • Navigate to the scheduled tasks.
  • In the left pane, expand the Task Scheduler Library.
  • Look for the folder where your task is saved (e.g., Task Scheduler Library or a custom folder).
• Find your task:
  • Look for the task name ProcoreDeltaShareScheduling.
  • Click on it to view its details in the bottom pane.
• Verify its schedule:
  • Check the Triggers tab to see when the task is set to run.
  • Check the History tab to confirm recent runs.
• To delete task:
  • Delete task from the GUI.

• Immediate Execution 
  • Option to run the delta_share_to_s3_.py File Structure

• Unset 
  
  ├── requirements.txt                    # Dependencies 
  ├── user_exp.py                                # Configuration utility
  ├── delta_share_to_s3.py            # Data sync script
  ├── config.yaml                                  # Generated configuration
  ├── config.share                               # Delta Share config file
  ├── procore_scheduling.log       # Log of scheduling runs 

  
  Example Usage 
  • Step 1: Install dependencies 
    $ pip install -r requirements.txt
  • Step 2: Run configuration utility 
    $ python user_exp.py 
  • Procore Analytics Cloud Connect Access 
    • This CLI will help you choose your source and destination store to access/write Procore data into S3. 
    • Press Enter to Continue.
    • Enter list of tables (comma-separated), leave it blank for all tables: projects,users,tasks.
    • Enter path to config.share: /path/to/config.share.
    • Enter access key: s3 key.
    • Enter secret: secret.
    • Enter bucket: bucket name.
    • Do you want to configure this as a daily job on cron? (Yes/No): Yes 
    • Enter the schedule in cron format (e.g., * * * * * ): 0 2 * * * 
    • Do you want to execute the job now? (Yes/No): Yes 
  • Step 3: Manual execution (if needed)
    $ python delta_share_to_s3.py 
• Configuration Reuse
  The tool saves your configuration in the config.yaml file and offers to reuse previously stored settings: 
  • Source configuration can be reused.
  • Target (S3) configuration can be reused. 
  • You can choose to update either configuration independently.

Troubleshooting 

Common issues and solutions: 

• Cron Job Setup  
  • Ensure system permissions are correctly configured.
  • Check system logs if the job fails to run.
  • Verify the script (delta_share_to_s3.py) execute permissions.
• Configuration File
  • Confirm the file config.yaml is in the same directory as the script.
  • Backup before making any changes.
• Support
  • Review script logs for detailed error messages.
  • Review your config.yaml file for misconfigurations.
  • Contact your system administrator for permission-related issues.
  • Reach out to Procore support for Delta Share access issues.
  • Verify cron job setup by checking system logs: See 'procore_scheduling_log' file.

Notes:

1. Remember to always backup your configuration before making changes.
2. Test new configurations in a non-production environment first.

Python Connector

The Delta Sharing Python Connector is a Python library that implements the Delta Sharing Protocol to read tables from a Delta Sharing server. You can load shared tables as a pandas DataFrame, or as an Apache Spark DataFrame if running in PySpark with the Apache Spark Connector installed.

Apache Spark Connector

The Apache Spark Connector implements the Delta Sharing Protocol to read shared tables from a Delta Sharing Server. It can be used in SQL, Python, Java, Scala and R.

Configuring Apache Spark

You can set up Apache Spark to load the Delta Sharing connector in the following two
ways:

• Run interactively: Start the Spark shell (Scala or Python) with the Delta Sharing connector and run the code snippets interactively in the shell.
• Run as a project: Set up a Maven or SBT project (Scala or Java) with the Delta Sharing connector, copy the code snippets into a source file, and run the project.

If you are using Databricks Runtime, you can skip this section and follow Databricks Libraries doc to install the connector on your clusters.

Set up an interactive shell

To use Delta Sharing connector interactively within the Spark’s Scala/Python shell, you can launch the shells as follows.

PySpark Shell

Unset
pyspark --packages io.delta:delta-sharing-spark_2.12:3.1.0

Scala Shell

Unset
bin/spark-shell --packages
io.delta:delta-sharing-spark_2.12:3.1.0

Set up a standalone project

If you want to build a Java/Scala project using Delta Sharing connector from Maven Central Repository, you can use the following Maven coordinates.