"It is important to maintain a conda dependency file and/or MLstudio environment. \r\n",
"\r\n",
"Every user of the workspace will use their own compute instance, with conda files and environments it is easy to install dependencies on these different compute instances."
"Azure ML environments are an encapsulation of the environment where your machine learning training happens. They define Python packages, environment variables, Docker settings and other attributes in declarative fashion. Environments are versioned: you can update them and retrieve old versions to revisit and review your work.\r\n",
"\r\n",
"Environments allow you to:\r\n",
"\r\n",
"- Encapsulate dependencies of your training process, such as Python packages and their versions.\r\n",
"- Reproduce the Python environment on your local computer in a remote run on VM or ML Compute cluster\r\n",
"- Reproduce your experimentation environment in production setting.\r\n",
"- Revisit and audit the environment in which an existing model was trained.\r\n",
"- Environment, compute target and training script together form run configuration: the full specification of training run.\r\n",
"\r\n",
"With the following code we can list current available environments."
],
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
}
},
{
"cell_type": "code",
"source": [
"from azureml.core import Environment\r\n",
"\r\n",
"envs = Environment.list(workspace=ws)\r\n",
"\r\n",
"for env in envs:\r\n",
" print(\"Name\",env)\r\n",
" if envs[env].python.conda_dependencies is not None:\r\n",
"You can view and manage datastores in Azure Machine Learning Studio, or you can use the Azure Machine Learning SDK. For example, the following code lists the names of each datastore in the workspace."
],
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
}
},
{
"cell_type": "code",
"source": [
"from azureml.core import Datastore\r\n",
"\r\n",
"for ds_name in ws.datastores:\r\n",
" print(ds_name)"
],
"outputs": [],
"execution_count": null,
"metadata": {
"jupyter": {
"source_hidden": false,
"outputs_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
},
"gather": {
"logged": 1646659467722
}
}
},
{
"cell_type": "markdown",
"source": [
"You can get a reference to any datastore by using the Datastore.get() method as shown here:\r\n",
"The workspace always includes a default datastore (initially, this is the built-in workspaceblobstore datastore), which you can retrieve by using the get_default_datastore() method of a Workspace object, like this:"
],
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
}
},
{
"cell_type": "code",
"source": [
"default_store = ws.get_default_datastore()"
],
"outputs": [],
"execution_count": null,
"metadata": {
"jupyter": {
"source_hidden": false,
"outputs_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
},
"gather": {
"logged": 1646659609540
}
}
},
{
"cell_type": "markdown",
"source": [
"When planning for datastores, consider the following guidelines:\r\n",
"\r\n",
"- When using Azure blob storage, premium level storage may provide improved I/O performance for large datasets. However, this option will increase cost and may limit replication options for data redundancy.\r\n",
"- When working with data files, although CSV format is very common, Parquet format generally results in better performance."
],
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
}
},
{
"cell_type": "markdown",
"source": [
"# Datasets\r\n",
"\r\n",
"Datasets are versioned packaged data objects that can be easily consumed in experiments and pipelines. Datasets are the recommended way to work with data, and are the primary mechanism for advanced Azure Machine Learning capabilities like data labeling and data drift monitoring.\r\n",
"\r\n",
"Datasets are typically based on files in a datastore, though they can also be based on URLs and other sources. You can create the following types of dataset:\r\n",
"\r\n",
"- **Tabular:** The data is read from the dataset as a table. You should use this type of dataset when your data is consistently structured and you want to work with it in common tabular data structures, such as Pandas dataframes.\r\n",
"- **File:** The dataset presents a list of file paths that can be read as though from the file system. Use this type of dataset when your data is unstructured, or when you need to process the data at the file level (for example, to train a convolutional neural network from a set of image files).\r\n",
"\r\n",
"Our data consists of two datasets:\r\n",
"- **labels_short.csv**: csv with labels of images. We will create a Tabular dataset from this csv.\r\n",
"- **images**: a folder with images of plants with diseases. We will create a File dataset from this folder."
],
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
}
},
{
"cell_type": "markdown",
"source": [
"To create a tabular dataset using the SDK, use the from_delimited_files method of the Dataset.Tabular class"
"Datasets can be versioned, enabling you to track historical versions of datasets that were used in experiments, and reproduce those experiments with data in the same state.\r\n",
"\r\n",
"You can create a new version of a dataset by registering it with the same name as a previously registered dataset and specifying the create_new_version property:"
"You can read data directly from a tabular dataset by converting it into a Pandas or Spark dataframe:\r\n",
"\r\n"
],
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
}
},
{
"cell_type": "code",
"source": [
"df = tab_ds.to_pandas_dataframe()\r\n",
"# code to work with dataframe goes here, for example:\r\n",
"print(df.head())"
],
"outputs": [],
"execution_count": null,
"metadata": {
"jupyter": {
"source_hidden": false,
"outputs_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
},
"gather": {
"logged": 1646660275291
}
}
},
{
"cell_type": "markdown",
"source": [
"When working with a file dataset, you can use the to_path() method to return a list of the file paths encapsulated by the dataset:\r\n",
"\r\n"
],
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
}
},
{
"cell_type": "code",
"source": [
"for file_path in img_ds.to_path():\r\n",
" print(file_path)"
],
"outputs": [],
"execution_count": null,
"metadata": {
"jupyter": {
"source_hidden": false,
"outputs_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
},
"gather": {
"logged": 1646660293091
}
}
}
],
"metadata": {
"kernelspec": {
"name": "workshop_env",
"language": "python",
"display_name": "workshop_env"
},
"language_info": {
"name": "python",
"version": "3.6.9",
"mimetype": "text/x-python",
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"pygments_lexer": "ipython3",
"nbconvert_exporter": "python",
"file_extension": ".py"
},
"kernel_info": {
"name": "workshop_env"
},
"nteract": {
"version": "nteract-front-end@1.0.0"
},
"microsoft": {
"host": {
"AzureML": {
"notebookHasBeenCompleted": true
}
}
}
},
"nbformat": 4,
"nbformat_minor": 2
}
\ No newline at end of file
%% Cell type:markdown id: tags:
# Setup
It is important to maintain a conda dependency file and/or MLstudio environment.
Every user of the workspace will use their own compute instance, with conda files and environments it is easy to install dependencies on these different compute instances.
Azure ML environments are an encapsulation of the environment where your machine learning training happens. They define Python packages, environment variables, Docker settings and other attributes in declarative fashion. Environments are versioned: you can update them and retrieve old versions to revisit and review your work.
Environments allow you to:
- Encapsulate dependencies of your training process, such as Python packages and their versions.
- Reproduce the Python environment on your local computer in a remote run on VM or ML Compute cluster
- Reproduce your experimentation environment in production setting.
- Revisit and audit the environment in which an existing model was trained.
- Environment, compute target and training script together form run configuration: the full specification of training run.
With the following code we can list current available environments.
# We can directly build the environment - this will create a new Docker
# image in Azure Container Registry (ACR), and directly 'bake in' our dependencies
# from the conda definition. When we later use the Environment, all AML will need to
# do is pull the image for environment, thus saving the time for potentially a
# long-running conda environment creation.
build=env.build(workspace=ws)
build.wait_for_completion(show_output=True)
else:
# load existing environment
env=Environment.get(workspace=ws,name=env_name)
```
%% Cell type:markdown id: tags:
# Datastores
You can view and manage datastores in Azure Machine Learning Studio, or you can use the Azure Machine Learning SDK. For example, the following code lists the names of each datastore in the workspace.
%% Cell type:code id: tags:
``` python
fromazureml.coreimportDatastore
fords_nameinws.datastores:
print(ds_name)
```
%% Cell type:markdown id: tags:
You can get a reference to any datastore by using the Datastore.get() method as shown here:
The workspace always includes a default datastore (initially, this is the built-in workspaceblobstore datastore), which you can retrieve by using the get_default_datastore() method of a Workspace object, like this:
%% Cell type:code id: tags:
``` python
default_store=ws.get_default_datastore()
```
%% Cell type:markdown id: tags:
When planning for datastores, consider the following guidelines:
- When using Azure blob storage, premium level storage may provide improved I/O performance for large datasets. However, this option will increase cost and may limit replication options for data redundancy.
- When working with data files, although CSV format is very common, Parquet format generally results in better performance.
%% Cell type:markdown id: tags:
# Datasets
Datasets are versioned packaged data objects that can be easily consumed in experiments and pipelines. Datasets are the recommended way to work with data, and are the primary mechanism for advanced Azure Machine Learning capabilities like data labeling and data drift monitoring.
Datasets are typically based on files in a datastore, though they can also be based on URLs and other sources. You can create the following types of dataset:
-**Tabular:** The data is read from the dataset as a table. You should use this type of dataset when your data is consistently structured and you want to work with it in common tabular data structures, such as Pandas dataframes.
-**File:** The dataset presents a list of file paths that can be read as though from the file system. Use this type of dataset when your data is unstructured, or when you need to process the data at the file level (for example, to train a convolutional neural network from a set of image files).
Our data consists of two datasets:
-**labels_short.csv**: csv with labels of images. We will create a Tabular dataset from this csv.
-**images**: a folder with images of plants with diseases. We will create a File dataset from this folder.
%% Cell type:markdown id: tags:
To create a tabular dataset using the SDK, use the from_delimited_files method of the Dataset.Tabular class
Datasets can be versioned, enabling you to track historical versions of datasets that were used in experiments, and reproduce those experiments with data in the same state.
You can create a new version of a dataset by registering it with the same name as a previously registered dataset and specifying the create_new_version property:
file_ds_filtered=file_ds_filtered.register(workspace=ws,name='img_files_name',description='images of plant disease dataset filtered on Train name',create_new_version=True)
```
%% Cell type:markdown id: tags:
We have selected only images with **Train_** in the name since we only know the labels of these images.
%% Cell type:markdown id: tags:
You can retrieve a specific version of a dataset by specifying the version parameter in the get_by_name method of the Dataset class.
