Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
115 commits
Select commit Hold shift + click to select a range
1651740
DOC: Use new URL for docs
cortadocodes Sep 23, 2025
969946e
ENH: Update docs URL in CLI help text
cortadocodes Sep 23, 2025
e92baf8
REF: Factor out template setting as a class
cortadocodes Sep 23, 2025
6ae0a46
DOC: Update CLI help text
cortadocodes Sep 23, 2025
fecabb8
FEA: Add `octue twined question ask example` command
cortadocodes Sep 23, 2025
e0cdaf6
DOC: Simplify installation instructions
cortadocodes Sep 23, 2025
e3206f4
ENH: Add dummy output values to example command
cortadocodes Sep 23, 2025
ec1cb35
FIX: Clean up after template class
cortadocodes Sep 23, 2025
7cc9ad8
DOC: Add getting started guide to index page of docs
cortadocodes Sep 23, 2025
87a251c
DOC: Add command for real data service to getting started
cortadocodes Sep 24, 2025
4492e0d
DOC: Fix nested bulleted list
cortadocodes Sep 24, 2025
764ba28
DOC: Adjust wording
cortadocodes Sep 24, 2025
679c47b
DOC: Explain question/answer terminology
cortadocodes Sep 24, 2025
134b375
DOC: Add shell redirection tip and add JSON link
cortadocodes Sep 24, 2025
a02fb8c
DOC: Simplify installation instructions
cortadocodes Sep 24, 2025
8ae7fb5
REF: Remove `--logger-uri` option from CLI
cortadocodes Sep 24, 2025
ff3f054
CHO: Remove `check-yaml` pre-commit check
cortadocodes Sep 24, 2025
b5aae2e
DOC: Enable tabbed docs
cortadocodes Sep 24, 2025
7709f7d
DOC: Add CLI tab to getting started guide
cortadocodes Sep 24, 2025
a651472
DOC: Add empty python tab to getting started guide
cortadocodes Sep 24, 2025
a7ad869
DOC: Add python code to getting started guide
cortadocodes Sep 24, 2025
9efdc80
DOC: Enable code snippet copy buttons
cortadocodes Sep 24, 2025
8f75681
DOC: Adjust wording and fix typo
cortadocodes Sep 26, 2025
4eb926c
DOC: Add "next steps" to end of getting started guide
cortadocodes Sep 26, 2025
3f3877e
DOC: Adjust wording and style on licence and version history pages
cortadocodes Sep 26, 2025
7e8a3bb
FEA: Add `ExampleChild` and use in CLI
cortadocodes Sep 26, 2025
3a7c97c
DOC: Use `ExampleChild` in getting started guide
cortadocodes Sep 26, 2025
5a3c107
DOC: Add infrastructure page to docs
cortadocodes Sep 26, 2025
6f74b5a
DOC: Add support page
cortadocodes Sep 26, 2025
a45e7a6
DOC: Add links to getting started guide
cortadocodes Sep 26, 2025
eadb77c
DOC: Add link to github issues
cortadocodes Sep 26, 2025
b1b5972
ENH: Merge `example` CLI subcommand into `remote` subcommand
cortadocodes Oct 1, 2025
2d56154
FIX: Return correct values from `ExampleChild.ask`
cortadocodes Oct 1, 2025
2843956
REF: Remove unnecessary context manager from `ExampleChild`
cortadocodes Oct 1, 2025
a8d675b
ENH: Give `ExampleChild` same signature as `Child`
cortadocodes Oct 1, 2025
dcaff9b
REF: Use analysis ID in example child and simplify template variable
cortadocodes Oct 1, 2025
fc87371
REF: Simplify `ExampleChild.__init__`
cortadocodes Oct 1, 2025
3c5626a
ENH: Raise error if "example" is used as a service namespace
cortadocodes Oct 1, 2025
965535a
ENH: Replace `ExampleChild` with `Child` with "example" namespace
cortadocodes Oct 1, 2025
cafbd45
DOC: Move admonition and clarify it
cortadocodes Oct 1, 2025
6bb9d4b
REF: Un-nest CLI commands for asking questions
cortadocodes Oct 1, 2025
f49c81c
DOC: Remove old documentation from navigation
cortadocodes Oct 2, 2025
60d797f
DOC: Remove old documentation
cortadocodes Oct 2, 2025
d1fc983
DOC: Add glossary
cortadocodes Oct 2, 2025
243e84c
DOC: Add outline "creating services" doc
cortadocodes Oct 3, 2025
6b704d6
DOC: Add overview page and move getting started guide
cortadocodes Oct 3, 2025
b86c7f5
DOC: Add more info to overview
cortadocodes Oct 3, 2025
ab93f24
DOC: Split docs into three main sections
cortadocodes Oct 3, 2025
6313469
DOC: Add placeholders for authentication docs
cortadocodes Oct 6, 2025
6774474
DOC: Add intro lines to getting started guides
cortadocodes Oct 6, 2025
d57df99
DOC: Add links to getting started guides to index page
cortadocodes Oct 6, 2025
9756d51
DOC: Add content to using services authentication page
cortadocodes Oct 6, 2025
ebdef0f
DOC: Number headings in index
cortadocodes Oct 8, 2025
898faa3
DOC: Add getting started guide for creating services
cortadocodes Oct 8, 2025
3f68256
DOC: Add remaining content to creating services getting started guide
cortadocodes Oct 8, 2025
2d14135
DOC: Remove redundant authentication page
cortadocodes Oct 8, 2025
62e84e8
DOC: Remove incorrect backticks from shell command
cortadocodes Oct 8, 2025
9157d4d
DOC: Simplify constraint on creating services quickstart guide
cortadocodes Oct 23, 2025
1a826f7
DOC: Improve creating services quickstart
cortadocodes Oct 23, 2025
93659b9
DOC: Add data containers docs
cortadocodes Oct 23, 2025
1e565bd
DOC: Add more core concepts
cortadocodes Oct 23, 2025
4b7f73b
DEP: Move `strincase` from dev to production dependencies
cortadocodes Oct 24, 2025
444e841
REV: Revert "DEP: Move `strincase` from dev to production dependencies"
cortadocodes Oct 24, 2025
f126bb7
DOC: Improve getting started guides
cortadocodes Oct 27, 2025
c955592
DOC: Move getting started guides into one section
cortadocodes Oct 27, 2025
adfa1a2
DOC: Improve authentication instructions
cortadocodes Oct 27, 2025
dfeda11
DOC: Add file tree to creating services guide
cortadocodes Oct 27, 2025
6adea0c
DOC: Add links to twine file and octue.yaml file docs
cortadocodes Oct 27, 2025
197bb5e
DOC: Improve authentication instructions
cortadocodes Oct 27, 2025
ec95f12
DOC: Add screenshot of repo actions tab
cortadocodes Oct 27, 2025
ed066ec
DOC: Update wording
cortadocodes Oct 27, 2025
ea73da2
ENH: Replace example app with much simpler Fibonacci sequence one
cortadocodes Oct 28, 2025
699daa0
TST: Update test docstrings
cortadocodes Oct 28, 2025
802d50b
FIX: Raise error at correct point
cortadocodes Oct 28, 2025
ffe8650
DOC: Update getting started docs with new example service
cortadocodes Oct 28, 2025
510e67c
DOC: Add remaining docs to core concepts section
cortadocodes Oct 28, 2025
932c091
DOC: Add JSON schema link and fix next steps link
cortadocodes Oct 31, 2025
1617c73
DOC: Add anatomy of twine file main page
cortadocodes Oct 31, 2025
aa2c45e
DOC: Add twine file quickstart
cortadocodes Oct 31, 2025
245f3e1
DOC: Remove pandoc fragment
cortadocodes Oct 31, 2025
1e5dcc8
FIX: Update log message advising use of `octue twined question` command
cortadocodes Nov 3, 2025
58649d9
DOC: Fix `app.py` in creating services getting started guide
cortadocodes Nov 3, 2025
d25d08d
DOC: Add tip about cloud location error to getting started guide
cortadocodes Nov 3, 2025
0c64731
DOC: Remove unused import from example `app.py`
cortadocodes Nov 3, 2025
ed86975
DOC: Add comments in app code
cortadocodes Nov 3, 2025
8b03dc5
FIX: Give input to example child
cortadocodes Nov 3, 2025
80abbd2
FIX: Fix getting `n` for example app in CLI
cortadocodes Nov 3, 2025
72fdcea
REF: Rename example Fibonacci analysis function
cortadocodes Nov 3, 2025
6c3cc6b
REF: Just output sequence from Fibonacci example function
cortadocodes Nov 3, 2025
f41d349
DOC: Use Fibonacci example in getting started guide
cortadocodes Nov 3, 2025
efd017d
DOC: Update description of example app
cortadocodes Nov 3, 2025
243f918
DOC: Add success admonitions to other getting started guides
cortadocodes Nov 3, 2025
c6b92eb
DOC: Fix typo
cortadocodes Nov 4, 2025
b28163d
DOC: Add full managing infrastructure getting started guide
cortadocodes Nov 4, 2025
009b09c
DOC: Standardise on "Twined service network" wording
cortadocodes Nov 4, 2025
bb791d2
DOC: Improve wording in guide
cortadocodes Nov 4, 2025
dffeab7
DOC: Improve instructions in guide
cortadocodes Nov 5, 2025
4b25a45
DOC: Fix example in guide
cortadocodes Nov 5, 2025
d5d4baf
DOC: Add troubleshooting section to creating services guide
cortadocodes Nov 5, 2025
c71adfd
DOC: Add `kubectl` troubleshooting info
cortadocodes Nov 5, 2025
9bd48af
DOC: Link to example service repo in guide
cortadocodes Nov 5, 2025
5135344
TST: Update test docstrings
cortadocodes Nov 6, 2025
d4ac9cc
DOC: Fix Fibonacci docstrings
cortadocodes Nov 6, 2025
9635dcb
DOC: Add missing filters docs
cortadocodes Nov 6, 2025
914289c
CHO: Add version compatibility metadata
cortadocodes Nov 6, 2025
466e489
DOC: Add extra info to auth docs
cortadocodes Nov 7, 2025
c61ca5f
DOC: Add link to support page
cortadocodes Nov 7, 2025
00508c7
DOC: Fix typo
cortadocodes Nov 7, 2025
356854c
DOC: Update infrastructure getting started guide
cortadocodes Nov 7, 2025
ca4b1cd
DOC: Clarify which GitHub account handle to use creating services
cortadocodes Nov 7, 2025
9711bf8
DOC: Update cluster module version in infra getting started
cortadocodes Nov 7, 2025
0053b7e
DOC: Add missing creating apps doc
cortadocodes Nov 7, 2025
3a773c1
DOC: Add missing downloading datafiles doc
cortadocodes Nov 7, 2025
7b267ea
DOC: Use more canonical links in docs
cortadocodes Nov 7, 2025
699f620
DOC: Fix links to images
cortadocodes Nov 7, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@ repos:
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
- id: check-yaml
- id: check-added-large-files

- repo: https://github.com/astral-sh/ruff-pre-commit
Expand Down
3 changes: 1 addition & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
The python SDK for running [Octue](https://octue.com) data services, digital twins, and applications - get faster data
groundwork so you have more time for the science!

Read the docs [here.](https://octue-python-sdk.readthedocs.io/en/latest/)
Read the docs [here.](https://twined.octue.com/)

Uses our [twined](https://twined.readthedocs.io/en/latest/) library for data validation.

Expand Down Expand Up @@ -41,7 +41,6 @@ Usage: octue [OPTIONS] COMMAND [ARGS]...
Read more in the docs: https://octue-python-sdk.readthedocs.io/en/latest/

Options:
--logger-uri TEXT Stream logs to a websocket at the given URI.
--log-level [debug|info|warning|error]
Log level used for the analysis. [default:
info]
Expand Down
27 changes: 12 additions & 15 deletions docs/authentication.md
Original file line number Diff line number Diff line change
@@ -1,37 +1,35 @@
# Authentication
You need authentication while using Twined to:

You need authentication while using `octue` to:

- Access data from Google Cloud Storage
- Use, run, or deploy Twined services
- Use or run services
- Access input and output data from analyses run on services

Authentication is provided by a GCP service account.

## Creating a service account

By setting up your Twined service network with the
[Twined Terraform modules](/deploying_services), a set of maintainer service accounts have already been
created with the required permissions.
[Twined Terraform modules](core_concepts/deploying_services.md), a set of maintainer service accounts have already been
created with the required permissions. These will have names starting with `maintainer-`.

## Using a service account

### Locally

1. Access your service accounts
[here](https://console.cloud.google.com/iam-admin/serviceaccounts),
making sure the correct project is selected
making sure the correct project is selected, or ask your Twined service network administrator
2. Click on the relevant service account, go to the "Keys" tab, and
create (download) a JSON key for it - it will be called
`<project-name>-XXXXX.json`.

!!! danger

It's best not to store this in your project to prevent accidentally
It's best not to store this in your repository to prevent accidentally
committing it or building it into a docker image layer. Instead, keep it
somewhere else on your local system with any other service account keys
you already have.

If you must keep within your project, it's good practice to name the
If you must keep within your repository, it's good practice to name the
file `gcp-credentials.json` and make sure that `gcp-cred*` is in your
`.gitignore` and `.dockerignore` files.

Expand All @@ -46,8 +44,7 @@ created with the required permissions.

### On GCP infrastructure / deployed services

- Credentials are automatically provided when running code or services
on GCP infrastructure, including the Kubernetes cluster
- `octue` uses these when when running on these platforms, so there's
no need to upload a service account key or include one in service
docker images
- Credentials are automatically provided when running code or services on GCP infrastructure, including the Kubernetes
cluster
- Twined uses these when running on these platforms, so there's no need to upload a service account key or include one
in service docker images
110 changes: 0 additions & 110 deletions docs/available_filters.md

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ to a child for processing/analysis. Questions can be:
or custom logic in your own webserver.

Questions are always asked to a _revision_ of a service. You can ask a
service a question if you have its [SRUID](/services/#service-names), project ID, and the necessary permissions.
service a question if you have its [SRUID](services.md/#service-names), project ID, and the necessary permissions.

## Asking a question

Expand Down Expand Up @@ -49,7 +49,7 @@ answer["output_manifest"]["my_dataset"].files

If you're using an environment other than the `main` environment, then before asking any questions to your Twined
services, set the `TWINED_SERVICES_TOPIC_NAME` environment variable to the name of the Twined services Pub/Sub topic
(this is set when [deploying a service network](/deploying_services/#deploying-services-advanced-developers-guide).
(this is set when [deploying a service network](deploying_services.md).
It will be in the form `<environment-name>.octue.twined.services`

!!! note
Expand All @@ -64,7 +64,7 @@ You can also set the following options when you call `Child.ask`:

- `children` - If the child has children of its own (i.e. grandchildren of the parent), this optional argument can be
used to override the child's "default" children. This allows you to specify particular versions of grandchildren to
use (see [this subsection below](#overriding-a-childs-children).
use (see [this subsection below](#overriding-a-childs-children)).
- `subscribe_to_logs` - if true, the child will forward its logs to you
- `allow_local_files` - if true, local files/datasets are allowed in any input manifest you supply
- `handle_monitor_message` - if provided a function, it will be called on any monitor messages from the child
Expand Down Expand Up @@ -242,10 +242,10 @@ at once instead of one after another.
## Asking a question within a service

If you have
[created your own Twined service](/creating_services) and want to ask children questions, you can do this more
[created your own Twined service](creating_services.md) and want to ask children questions, you can do this more
easily than above. Children are accessible from the `analysis` object by
the keys you give them in the
[service configuration](/creating_services/#octueyaml) file. For example, you can ask an `elevation` service a
[service configuration](creating_services.md/#octueyaml) file. For example, you can ask an `elevation` service a
question like this:

```python
Expand Down Expand Up @@ -314,7 +314,7 @@ you want it to use (dynamic children) to the
children will instead go to the dynamic children. Note that:

- You must provide the children in the same format as they're provided
in the [service configuration](/creating_services/#octueyaml)
in the [service configuration](../core_concepts/creating_services.md/#octueyaml)
- If you override one static child, you must override others, too
- The dynamic children must have the same keys as the static children
(so the child knows which service to ask which questions)
Expand Down
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -12,19 +12,16 @@ run locally on any machine or be deployed to the cloud. Currently:

## Anatomy of a Twined service

A Twined service is defined by the following files (located in the
repository root by default).
A Twined service is defined by the following files (located in the repository root by default).

### app.py

This is the entrypoint into your code - read more [here](/creating_apps).
This is the entrypoint into your code - read more [here](creating_apps.md).

### twine.json

This file defines the schema for the service's configuration, input,
and output data. Read more
[here](https://twined.readthedocs.io/en/latest/) and see an example
[here](https://twined.readthedocs.io/en/latest/quick_start_create_your_first_twine.html).
This file defines the schema for the service's configuration, input, and output data. Read more [here](twines.md) and
see an example [here](twine_file_quickstart.md).

### Dependencies file

Expand All @@ -45,26 +42,26 @@ are supported.

``` yaml
services:
- namespace: my-organisation
- namespace: my-organisation
name: my-app
```
It may also need the following key-value pairs:

- `app_source_path: <path>` - if your `app.py` file is not in the repository root

All paths should be relative to the repository root. Other valid
entries can be found in the `ServiceConfiguration` constructor.

!!! warning

Currently, only one service can be defined per repository, but it must
still appear as a list item of the "services" key. At some point, it
will be possible to define multiple services in one repository.

If a service's app needs any configuration, asks questions to any
other Twined services, or produces output datafiles/datasets, you will
need to provide some or all of the following values for that service:

- `configuration_values`
- `configuration_manifest`
- `children`
Expand Down Expand Up @@ -95,36 +92,35 @@ are supported.
- [An OpenFAST service](https://github.com/octue/openfast-service/blob/main/Dockerfile)

If you do provide one, you must provide its path relative to your
repository to the [build-twined-services] GitHub Actions [workflow](https://github.com/octue/workflows/blob/main/.github/workflows/build-twined-service.yml).
repository to the `build-twined-services` GitHub Actions [workflow](https://github.com/octue/workflows/blob/main/.github/workflows/build-twined-service.yml).

As always, if you need help with this, feel free to drop us a message or raise an issue!


### Where to specify the namespace, name, and revision tag

See [here](/services/#service-names) for service naming requirements.
See [here](services.md/#service-names) for service naming requirements.

**Namespace**

- Required: yes
- Set in:
- `octue.yaml`
- `OCTUE_SERVICE_NAMESPACE` environment variable (takes priority)
- `octue.yaml`
- `OCTUE_SERVICE_NAMESPACE` environment variable (takes priority)

**Name**

- Required: yes
- Set in:
- `octue.yaml`
- `OCTUE_SERVICE_NAME` environment variable (takes priority)
- `octue.yaml`
- `OCTUE_SERVICE_NAME` environment variable (takes priority)

**Revision tag**

- Required: no
- Default: a random "coolname" (e.g. `hungry-hippo`)
- Set in:
- `OCTUE_SERVICE_REVISION_TAG` environment variable
- If using `octue twined service start` command, the `--revision-tag` option (takes priority)
- `OCTUE_SERVICE_REVISION_TAG` environment variable
- If using `octue twined service start` command, the `--revision-tag` option (takes priority)

## Template apps

Expand Down Expand Up @@ -153,4 +149,4 @@ Automated deployment with Octue means:
other services are sent to it, meaning there are minimal costs to
having it deployed but not in use.

If you'd like help deploying services, contact us. To do it yourself, see [here](/deploying_services).
If you'd like help deploying services, [contact us](../support.md). To do it yourself, see [here](deploying_services.md).
Loading