⛄Iceberg

Instructions for installation and configuration of the Iceberg Agent

Note: We currently only support Iceberg for our AWS Customers

Instructions

We recommend that you follow the Installation instructions from the UI. This can be found on app.lariatdata.com by clicking on the Integrations tab and going to "Add new integration" as outlined in the Installation & Configuration page.

To describe what the install entails, we outline how you could do this outside of the UI. If you are interested in the code that powers the install, please take a look here https://github.com/lariat-data/install-aws-iceberg-agent

What do you need for the install?

docker
Your Cloud Account ID and region
Your own Cloud account keys

Note: If you are running the install outside of the UI, you will need your Lariat API Key and Lariat Application Key and Lariat generated Cloud access keys (e.g. AWS key and secret key)

Installation command

If using the UI, copy the installation command and fill in the unpopulated fields.

Here is what the command looks like:

docker run -it --pull=always --mount type=bind,source=/local/path/to/config/iceberg.yaml,target=/workspace/iceberg_agent.yaml,readonly -e AWS_REGION={YOUR_AWS_REGION} -e AWS_ACCOUNT_ID={YOUR_AWS_ACCOUNT_ID} -e AWS_ACCESS_KEY_ID=$(aws configure get aws_access_key_id) -e AWS_SECRET_ACCESS_KEY=$(aws configure get aws_secret_access_key) -e AWS_SESSION_TOKEN=$(aws configure get aws_session_token) -e AWS_ACCOUNT_ID={YOUR_AWS_ACCOUNT_ID} -e LARIAT_TMP_AWS_ACCESS_KEY_ID={PREFILLED_BY_UI} -e LARIAT_TMP_AWS_SECRET_ACCESS_KEY={PREFILLED_BY_UI} -e LARIAT_API_KEY={PREFILLED_BY_UI} -e LARIAT_APPLICATION_KEY={PREFILLED_BY_UI} lariatdata/install-aws-iceberg-agent:latest install

If you do not have access to the UI and need LARIAT_TMP_AWS_ACCESS_KEY_ID and LARIAT_TMP_AWS_SECRET_ACCESS_KEY you will have to reach out to [email protected].

The config defines catalogs, databases and tables that will be tracked by the Iceberg agent. The agent installer sets up the correct permissions to access Glue Catalogs and ensures that the agent is running and collecting the relevant health metrics from the Iceberg agent.

catalog:
  glue:
    databases:
      DATABASE_NAME:
        - TABLE_NAME:
            uri: "s3://location_of_data/prefix/to/location/"
        - TABLE_NAME_2:
            uri: "s3://location_of_data/prefix/to/location/"
  postgres:
    user: $POSTGRES_USER
    password: $POSTGRES_PASSWORD
    host: $POSTGRES_HOST_NAME
    port: $POSTGRES_PORT
    databases:
      DATABASE_NAME:
        - TABLE_NAME:
            uri: "s3://location_of_data/prefix/to/location/"
source_id: unique_source_id

The catalogs available are glue , postgres , rest , hive and others defined in https://py.iceberg.apache.org/configuration/#glue-catalog

Post-Install Setup

In order to configure tracked tables outlined in the configuration, go to the datasets section in the homepage and click on the ellipses next to the newly tracked Iceberg Datasets. Select "Configure" and move through the menu to rapidly set up indicators. You can also click on "Indicators" from the sidebar and select "Create an Indicator" to create fully customized indicators. That is covered in Working with Datasets and Indicators - This page summarizes the details of the Rapid Set Up path. The rapid path to setting up indicators should take no longer than 5 minutes. If you need to batch set up indicators for 100s of datasets, we have a beta feature that does. Just reach out to [email protected] and we will activate it for you.

Select a timestamp field that either has an ISO formatted date or is a unix timestamp. You can also use duckdb sql to define a computation that creates a unix timestamp from existing fields. We encourage you to use partition fields as the timestamp field.

You can also select dimensions, these are columns that you can later break metrics out by as seen below. You can also use these dimensions to filter health metrics for a more granular view.

You can then select the frequency with which Metrics should be collected.

Finally, you can rapidly select the columns and type of health metrics prior to complete metric definition. You will start seeing indicators populate based on the backfill schedule or general schedule you select. Note: Only select summary stats for numeric fields.

Reach out to [email protected] if you have any further questions!

PreviousS3 Object Storage NextAWS Athena

Last updated 11 months ago