Experiment tracking

Note

We only integrated Comet ML as the dashboard for tracking experiment and logging media.

Supported dashboard

While some experiment tracking tools have different APIs and UI looks, they still the same in somewhat. We are having plan to support some of very dashboard, such as:

Dashboard	Supported
Comet ML	✓
Weights & Biases	✗
MLFlow	✗

Note

We are working on supporting more dashboard in the near future. Stay tuned!

Intergrating UETAI in your script

We would like to introduce our very simple API to logging experiment and managing artifact which has been customized from original framework’s API and carefully tested.

Initialize experiment

Initializing every dashboard’s experiment and using all its API through object build in uetai.logger. While you call a new run at the top of your script, uetai will automatically initialize a Experiment object which will create a local directory to saved all log, files and streamed it to dashboard server (if the tools is hosted online).

from uetai.logger import CometLogger

# Initialize experiment
logger = CometLogger(
    project_name="your-project-name",
    api_key="your-api-key",
    workspace="your-workspace-name",
)

Tip

To get your own API_KEY, you need to sign up a Comet ML account, go to Setting> Developer Information> Generate API key and copy the key.

Logging your experiment

In order to log your metrics, metadata like loss, accuracy in training loop, use log(). You might find that different because we’ve re-designed your dashboard logging methods to be more synchronous and easy to use.

log() can be used to logged mupliple thing. See supported types in below table.

# Logging your metrics
logger.log({'loss'=loss, 'accuracy'=accuracy,}, step=step)

# Logging your image
logger.log({'image': image_array}, step=step)

# Multiple logging
logger.log({
    'loss': loss,
    'accuracy': accuracy,
    'image': image_array,
}, step=step)

Supported data types

We’re still in developing progress, some data types might not be supported yet. However, you can still log it through dashboard’s original API, example:

# Get experiment object
exp = logger.get_experiment()

# Using original API
exp.log_confusion_matrix(labels=["one", "two", "three"],
    matrix=[[10, 0, 0],
    [ 0, 9, 1],
    [ 1, 1, 8]]
)

Currently supported data types:

Dashboard	Float metrics	Image	Text	Audio	Graph
Comet ML	✓	✓	✓	✗	✗