Introduction

Developers work on applications that are supposed to be deployed on some server in order to allow anyone to use those. Typically in the machine where these apps live, developers set up environment variables that allow the app to run. These variables can be API keys of external services, URL of your database and much more.

For local development though, it is really inconvenient to declare these variables on the machine because it is a slow and messy process. So I’d like to share in this short tutorial how to use Pydantic to handle environment variables in a secure way.

.env file

What you commonly do in a Python project is to store all your environment variables in a file named .env. This is a text file containing all the variables in a key : value format. You can use also the value of one of the variables to declare one of the other variables by leveraging the {} syntax.

The following is an example:

#.env file

OPENAI_API_KEY=”sk-your private key”
OPENAI_MODEL_ID=”gpt-4o-mini”

# Development settings
DOMAIN=example.org
ADMIN_EMAIL=admin@${DOMAIN}

WANDB_API_KEY=”your-private-key”
WANDB_PROJECT=”myproject”
WANDB_ENTITY=”my-entity”

SERPAPI_KEY= “your-api-key”
PERPLEXITY_TOKEN = “your-api-token”

Be aware the .env file should remain private, so it is important that this file is mentioned in your .gitignore file, to be sure that you never push it on GitHub, otherwise, other developers could steal your keys and use the tools you’ve paid for.

env.example file

To ease the life of developers who will clone your repository, you could include an env.example file in your project. This is a file containing only the keys of what is supposed to go into the .env file. In this way, other people know what APIs, tokens, or secrets in general they need to set to make the scripts work.

#env.example

OPENAI_API_KEY=””
OPENAI_MODEL_ID=””

DOMAIN=””
ADMIN_EMAIL=””

WANDB_API_KEY=””
WANDB_PROJECT=””
WANDB_ENTITY=””

SERPAPI_KEY= “”
PERPLEXITY_TOKEN = “”

python-dotenv

python-dotenv is the library you use to load the variables declared into the .env file. To install this library:

pip install python-dotenv

Now you can use the load_dotenv to load the variables. Then get a reference to these variables with the os module.

import os
from dotenv import load_dotenv

load_dotenv()

OPENAI_API_KEY = os.getenv(‘OPENAI_API_KEY’)
OPENAI_MODEL_ID = os.getenv(‘OPENAI_MODEL_ID’)

This method will first look into your .env file to load the variables you’ve declared there. If this file doesn’t exist, the variable will be taken from the host machine. This means that you can use the .env file for your local development but then when the code is deployed to a host environment like a virtual machine or Docker container we are going to directly use the environment variables defined in the host environment.

Pydantic

Pydantic is one of the most used libraries in Python for data validation. It is also used for serializing and deserializing classes into JSON and back. It automatically generates JSON schema, reducing the need for manual schema management. It also provides built-in data validation, ensuring that the serialized data adheres to the expected format. Lastly, it easily integrates with popular web frameworks like FastAPI.

pydantic-settings is a Pydantic feature needed to load and validate settings or config classes from environment variables.

!pip install pydantic-settings

We are going to create a class named Settings. This class will inherit BaseSettings. This makes the default behaviours of determining the values of any fields to be read from the .env file. If no var is found in the .env file it will be used the default value if provided.

from pydantic_settings import BaseSettings, SettingsConfigDict

from pydantic import (
AliasChoices,
Field,
RedisDsn,
)

class Settings(BaseSettings):
auth_key: str = Field(validation_alias=’my_auth_key’)
api_key: str = Field(alias=’my_api_key’)

redis_dsn: RedisDsn = Field(
‘redis://user:pass@localhost:6379/1’, #default value
validation_alias=AliasChoices(‘service_redis_dsn’, ‘redis_url’),
)

model_config = SettingsConfigDict(env_prefix=’my_prefix_’)

In the Settings class above we have defined several fields. The Field class is used to provide extra information about an attribute.

In our case, we setup a validation_alias. So the variable name to look for in the .env file is overridden. In the case reported above, the environment variable my_auth_key will be read instead of auth_key.

You can also have multiple aliases to look for in the .env file that you can specify by leveraging AliasChoises(choise1, choise2).

The last attribute model_config , contains all the variables regarding a particular topic (e.g connection to a db). And this variable will store all .env var that start with the prefix env_prefix.

Instantiate and use settings

The next step would be to actually instantiate and use these settings in your Python project.

from pydantic_settings import BaseSettings, SettingsConfigDict

from pydantic import (
AliasChoices,
Field,
RedisDsn,
)

class Settings(BaseSettings):
auth_key: str = Field(validation_alias=’my_auth_key’)
api_key: str = Field(alias=’my_api_key’)

redis_dsn: RedisDsn = Field(
‘redis://user:pass@localhost:6379/1’, #default value
validation_alias=AliasChoices(‘service_redis_dsn’, ‘redis_url’),
)

model_config = SettingsConfigDict(env_prefix=’my_prefix_’)

# create immediately a settings object
settings = Settings()

Now what use the settings in other parts of our codebase.

from Settings import settings

print(settings.auth_key)

You finally have an easy access to your settings, and Pydantic helps you validate that the secrets have the correct format. For more advanced validation tips refer to the Pydantic documentation: https://docs.pydantic.dev/latest/

Final thoughts

Managing the configuration of a project is a boring but important part of software development. Secrets like API keys, db connections are what usually power your application. Naively you can hardcode these variables in your code and it will still work, but for obvious reasons, this could not be a good practice. In this article, I showed you an introduction on how to use pydantic settings to have a structured and safe way to handle your configurations.

 Linkedin |  X (Twitter) |  Website

The post Manage Environment Variables with Pydantic appeared first on Towards Data Science.