Skip to content

Connecting to AWS S3 from Python

This page explains how to authenticate to AWS S3 locally, so Simulation can fetch KySim simulation files from the cloud when testing a model on your own machine.

The bucket name is not a secret — AWS itself has no concept of a "default bucket" (every S3 request takes the bucket name as a plain argument, the same way a filename is an argument to open()), so it needs no special protection. It's just set as a plain environment variable right before running a model, the same way the MATLAB side of KyosLib already does it (getenv/setenv('KYOS_S3_BUCKET', ...), no file, nothing persisted).

Credentials, on the other hand, use AWS SSO rather than static keys — no access key, secret key, or session token is ever typed into kyoslib_py or saved anywhere by it. boto3 (which kyoslib_py uses internally) already knows how to use an AWS SSO profile automatically — no code in this repo needs to know anything about SSO at all.

1. Install the AWS CLI (one-time)

Download and run the official installer: https://awscli.amazonaws.com/AWSCLIV2.msi. Confirm it installed correctly, and that it's version 2 (SSO login needs v2):

aws --version

2. Set up an SSO profile for each AWS account you need

aws configure sso

This asks for the SSO start URL — https://kyos.awsapps.com/start/#/ — then walks you through picking your account and role in a browser. At the end, it asks you to name the profile — give it a clear name describing the account, e.g. kyos-storage.

One profile is not always enough. Simulation data doesn't all live in the same AWS account: most clients' buckets are in the Storage account, but Japanese clients' buckets are in a separate Compute account, and Kubernetes-based clients (currently only ISEM) are in EKS. If you aren't sure which account a particular client's bucket lives in, ask the platform team. Set up one named SSO profile per account you actually need, once each.

3. Log in and select the right profile whenever your session expires

aws sso login --profile kyos-storage
$env:AWS_PROFILE = "kyos-storage"

On macOS/Linux, use export AWS_PROFILE=kyos-storage instead of the $env: line.

aws sso login --profile <name> opens a browser, you log in, done — SSO sessions expire after some hours, so you'll run this again from time to time. AWS_PROFILE tells boto3 which of your configured profiles to actually use, so set it to match whichever profile covers the client you're working with right now.

4. Find the S3 bucket name for the server/client you're working with

  1. Log in to the same portal, https://kyos.awsapps.com/start/#/.
  2. Click Storage → ClientStorageAccess.
  3. Search for S3 in the top search bar.
  4. In the list of buckets, search for the relevant KYOS server name (e.g. a client server name like shell, or staging-quants).

5. Set the bucket and run your model

$env:KYOS_S3_BUCKET = "<bucket-name>"

(macOS/Linux: export KYOS_S3_BUCKET=<bucket-name>.)

If you're setting this on a pod that also runs MATLAB jobs, KYOS_S3_BUCKET=s3://<bucket-name> (the form MATLAB expects) works too — kyoslib_py strips the s3:// prefix itself.

Make sure the job's settings XML has <FetchFromS3>1</FetchFromS3> under <KySimInfo>, then run the model as usual. No other setup is needed — files that are missing locally are fetched from S3 automatically the first time they're needed.

See S3 File Loader for the other environment variables you can set (KYOS_S3_LOCAL_ROOT) and the IAM permissions a bucket must grant.