Environment class
An environment for running orchestration tasks.
Environments run Apache Airflow software on Google infrastructure.
To get more information about Environments, see:
- Managed Service for Apache Airflow documentation
- Managed Airflow API documentation
- How-to Guides for Managed Airflow (Gen 3)
- Creating environments
- Scaling environments
- Change environment networking type (Private or Public IP)
- Connect an environment to a VPC network
- How-to Guides for Managed Airflow (Gen 2)
- Creating environments
- Scaling environments
- Configuring Shared VPC
- Apache Airflow Documentation
> Note Managed Airflow (Legacy Gen 1) is in the post-maintenance mode. Google doesn't release any further updates to Managed Service for Managed Airflow (Legacy Gen 1), including new versions of Airflow, bugfixes, and security updates. We recommend using Managed Airflow (Gen 3) or Managed Airflow (Gen 2) instead.
Several special considerations apply to using Terraform with Managed Service for Apache Airflow:
- The Environment resource is based on several layers of Google Cloud infrastructure. Terraform doesn't manage these underlying resources. For example, in Managed Airflow (Gen 2), this includes a Google Kubernetes Engine cluster, Cloud Storage, and Compute networking resources.
- Creating or updating an environment usually takes around 25 minutes.
- In some cases, errors in the configuration are detected and reported only during the process of environment creation. If you encounter such errors, please verify that your configuration is valid for the Managed Airflow environment you are creating before filing bugs for the Terraform provider.
- Environments have Google Cloud Storage buckets that are not automatically deleted with the environment. See Delete environments for more information.
- See Troubleshooting pages if you encounter problems.
Example Usage
Basic usage in Managed Airflow (Gen 3)
import * as pulumi from "@pulumi/pulumi";
import * as gcp from "@pulumi/gcp";
const test = new gcp.composer.Environment("test", {
name: "example-composer-env",
region: "us-central1",
config: {
softwareConfig: {
imageVersion: "composer-3-airflow-2",
},
},
});
import pulumi
import pulumi_gcp as gcp
test = gcp.composer.Environment("test",
name="example-composer-env",
region="us-central1",
config={
"software_config": {
"image_version": "composer-3-airflow-2",
},
})
using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Gcp = Pulumi.Gcp;
return await Deployment.RunAsync(() =>
{
var test = new Gcp.Composer.Environment("test", new()
{
Name = "example-composer-env",
Region = "us-central1",
Config = new Gcp.Composer.Inputs.EnvironmentConfigArgs
{
SoftwareConfig = new Gcp.Composer.Inputs.EnvironmentConfigSoftwareConfigArgs
{
ImageVersion = "composer-3-airflow-2",
},
},
});
});
package main
import (
"github.com/pulumi/pulumi-gcp/sdk/v9/go/gcp/composer"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)
func main() {
pulumi.Run(func(ctx *pulumi.Context) error {
_, err := composer.NewEnvironment(ctx, "test", &composer.EnvironmentArgs{
Name: pulumi.String("example-composer-env"),
Region: pulumi.String("us-central1"),
Config: &composer.EnvironmentConfigArgs{
SoftwareConfig: &composer.EnvironmentConfigSoftwareConfigArgs{
ImageVersion: pulumi.String("composer-3-airflow-2"),
},
},
})
if err != nil {
return err
}
return nil
})
}
pulumi {
required_providers {
gcp = {
source = "pulumi/gcp"
}
}
}
resource "gcp_composer_environment" "test" {
name = "example-composer-env"
region = "us-central1"
config = {
software_config = {
image_version = "composer-3-airflow-2"
}
}
}
package generated_program;
import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.gcp.composer.Environment;
import com.pulumi.gcp.composer.EnvironmentArgs;
import com.pulumi.gcp.composer.inputs.EnvironmentConfigArgs;
import com.pulumi.gcp.composer.inputs.EnvironmentConfigSoftwareConfigArgs;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;
public class App {
public static void main(String[] args) {
Pulumi.run(App::stack);
}
public static void stack(Context ctx) {
var test = new Environment("test", EnvironmentArgs.builder()
.name("example-composer-env")
.region("us-central1")
.config(EnvironmentConfigArgs.builder()
.softwareConfig(EnvironmentConfigSoftwareConfigArgs.builder()
.imageVersion("composer-3-airflow-2")
.build())
.build())
.build());
}
}
resources:
test:
type: gcp:composer:Environment
properties:
name: example-composer-env
region: us-central1
config:
softwareConfig:
imageVersion: composer-3-airflow-2
Basic usage in Managed Airflow (Gen 2)
import * as pulumi from "@pulumi/pulumi";
import * as gcp from "@pulumi/gcp";
const test = new gcp.composer.Environment("test", {
name: "example-composer-env",
region: "us-central1",
config: {
softwareConfig: {
imageVersion: "composer-2-airflow-2",
},
},
});
import pulumi
import pulumi_gcp as gcp
test = gcp.composer.Environment("test",
name="example-composer-env",
region="us-central1",
config={
"software_config": {
"image_version": "composer-2-airflow-2",
},
})
using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Gcp = Pulumi.Gcp;
return await Deployment.RunAsync(() =>
{
var test = new Gcp.Composer.Environment("test", new()
{
Name = "example-composer-env",
Region = "us-central1",
Config = new Gcp.Composer.Inputs.EnvironmentConfigArgs
{
SoftwareConfig = new Gcp.Composer.Inputs.EnvironmentConfigSoftwareConfigArgs
{
ImageVersion = "composer-2-airflow-2",
},
},
});
});
package main
import (
"github.com/pulumi/pulumi-gcp/sdk/v9/go/gcp/composer"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)
func main() {
pulumi.Run(func(ctx *pulumi.Context) error {
_, err := composer.NewEnvironment(ctx, "test", &composer.EnvironmentArgs{
Name: pulumi.String("example-composer-env"),
Region: pulumi.String("us-central1"),
Config: &composer.EnvironmentConfigArgs{
SoftwareConfig: &composer.EnvironmentConfigSoftwareConfigArgs{
ImageVersion: pulumi.String("composer-2-airflow-2"),
},
},
})
if err != nil {
return err
}
return nil
})
}
pulumi {
required_providers {
gcp = {
source = "pulumi/gcp"
}
}
}
resource "gcp_composer_environment" "test" {
name = "example-composer-env"
region = "us-central1"
config = {
software_config = {
image_version = "composer-2-airflow-2"
}
}
}
package generated_program;
import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.gcp.composer.Environment;
import com.pulumi.gcp.composer.EnvironmentArgs;
import com.pulumi.gcp.composer.inputs.EnvironmentConfigArgs;
import com.pulumi.gcp.composer.inputs.EnvironmentConfigSoftwareConfigArgs;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;
public class App {
public static void main(String[] args) {
Pulumi.run(App::stack);
}
public static void stack(Context ctx) {
var test = new Environment("test", EnvironmentArgs.builder()
.name("example-composer-env")
.region("us-central1")
.config(EnvironmentConfigArgs.builder()
.softwareConfig(EnvironmentConfigSoftwareConfigArgs.builder()
.imageVersion("composer-2-airflow-2")
.build())
.build())
.build());
}
}
resources:
test:
type: gcp:composer:Environment
properties:
name: example-composer-env
region: us-central1
config:
softwareConfig:
imageVersion: composer-2-airflow-2
Basic Usage in Managed Airflow (Legacy Gen 1)
import * as pulumi from "@pulumi/pulumi";
import * as gcp from "@pulumi/gcp";
const test = new gcp.composer.Environment("test", {
name: "example-composer-env",
region: "us-central1",
config: {
softwareConfig: {
imageVersion: "composer-1-airflow-2",
},
},
});
import pulumi
import pulumi_gcp as gcp
test = gcp.composer.Environment("test",
name="example-composer-env",
region="us-central1",
config={
"software_config": {
"image_version": "composer-1-airflow-2",
},
})
using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Gcp = Pulumi.Gcp;
return await Deployment.RunAsync(() =>
{
var test = new Gcp.Composer.Environment("test", new()
{
Name = "example-composer-env",
Region = "us-central1",
Config = new Gcp.Composer.Inputs.EnvironmentConfigArgs
{
SoftwareConfig = new Gcp.Composer.Inputs.EnvironmentConfigSoftwareConfigArgs
{
ImageVersion = "composer-1-airflow-2",
},
},
});
});
package main
import (
"github.com/pulumi/pulumi-gcp/sdk/v9/go/gcp/composer"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)
func main() {
pulumi.Run(func(ctx *pulumi.Context) error {
_, err := composer.NewEnvironment(ctx, "test", &composer.EnvironmentArgs{
Name: pulumi.String("example-composer-env"),
Region: pulumi.String("us-central1"),
Config: &composer.EnvironmentConfigArgs{
SoftwareConfig: &composer.EnvironmentConfigSoftwareConfigArgs{
ImageVersion: pulumi.String("composer-1-airflow-2"),
},
},
})
if err != nil {
return err
}
return nil
})
}
pulumi {
required_providers {
gcp = {
source = "pulumi/gcp"
}
}
}
resource "gcp_composer_environment" "test" {
name = "example-composer-env"
region = "us-central1"
config = {
software_config = {
image_version = "composer-1-airflow-2"
}
}
}
package generated_program;
import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.gcp.composer.Environment;
import com.pulumi.gcp.composer.EnvironmentArgs;
import com.pulumi.gcp.composer.inputs.EnvironmentConfigArgs;
import com.pulumi.gcp.composer.inputs.EnvironmentConfigSoftwareConfigArgs;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;
public class App {
public static void main(String[] args) {
Pulumi.run(App::stack);
}
public static void stack(Context ctx) {
var test = new Environment("test", EnvironmentArgs.builder()
.name("example-composer-env")
.region("us-central1")
.config(EnvironmentConfigArgs.builder()
.softwareConfig(EnvironmentConfigSoftwareConfigArgs.builder()
.imageVersion("composer-1-airflow-2")
.build())
.build())
.build());
}
}
resources:
test:
type: gcp:composer:Environment
properties:
name: example-composer-env
region: us-central1
config:
softwareConfig:
imageVersion: composer-1-airflow-2
With environment resources configuration
> Note
To use custom service accounts, you must give at least the
role/composer.worker role to the service account of the Managed Airflow
environment. For more information, see the
Access Control
page in the Managed Airflow documentation.
You might need to assign additional roles depending on specific workflows
that the Airflow DAGs will be running.
Environment resources configuration in Managed Airflow (Gen 3)
import * as pulumi from "@pulumi/pulumi";
import * as gcp from "@pulumi/gcp";
const testAccount = new gcp.serviceaccount.Account("test", {
accountId: "composer-env-account",
displayName: "Test Service Account for Managed Airflow Environment",
});
const test = new gcp.composer.Environment("test", {
name: "example-composer-env-tf-c3",
region: "us-central1",
config: {
softwareConfig: {
imageVersion: "composer-3-airflow-2",
},
workloadsConfig: {
scheduler: {
cpu: 0.5,
memoryGb: 2,
storageGb: 1,
count: 1,
},
triggerer: {
cpu: 0.5,
memoryGb: 1,
count: 1,
},
dagProcessor: {
cpu: 1,
memoryGb: 2,
storageGb: 1,
count: 1,
},
webServer: {
cpu: 0.5,
memoryGb: 2,
storageGb: 1,
},
worker: {
cpu: 0.5,
memoryGb: 2,
storageGb: 1,
minCount: 1,
maxCount: 3,
},
},
environmentSize: "ENVIRONMENT_SIZE_SMALL",
nodeConfig: {
serviceAccount: testAccount.name,
},
},
});
const composer_worker = new gcp.projects.IAMMember("composer-worker", {
project: "your-project-id",
role: "roles/composer.worker",
member: pulumi.interpolate`serviceAccount:${testAccount.email}`,
});
import pulumi
import pulumi_gcp as gcp
test_account = gcp.serviceaccount.Account("test",
account_id="composer-env-account",
display_name="Test Service Account for Managed Airflow Environment")
test = gcp.composer.Environment("test",
name="example-composer-env-tf-c3",
region="us-central1",
config={
"software_config": {
"image_version": "composer-3-airflow-2",
},
"workloads_config": {
"scheduler": {
"cpu": 0.5,
"memory_gb": float(2),
"storage_gb": float(1),
"count": 1,
},
"triggerer": {
"cpu": 0.5,
"memory_gb": float(1),
"count": 1,
},
"dag_processor": {
"cpu": float(1),
"memory_gb": float(2),
"storage_gb": float(1),
"count": 1,
},
"web_server": {
"cpu": 0.5,
"memory_gb": float(2),
"storage_gb": float(1),
},
"worker": {
"cpu": 0.5,
"memory_gb": float(2),
"storage_gb": float(1),
"min_count": 1,
"max_count": 3,
},
},
"environment_size": "ENVIRONMENT_SIZE_SMALL",
"node_config": {
"service_account": test_account.name,
},
})
composer_worker = gcp.projects.IAMMember("composer-worker",
project="your-project-id",
role="roles/composer.worker",
member=test_account.email.apply(lambda email: f"serviceAccount:{email}"))
using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Gcp = Pulumi.Gcp;
return await Deployment.RunAsync(() =>
{
var testAccount = new Gcp.ServiceAccount.Account("test", new()
{
AccountId = "composer-env-account",
DisplayName = "Test Service Account for Managed Airflow Environment",
});
var test = new Gcp.Composer.Environment("test", new()
{
Name = "example-composer-env-tf-c3",
Region = "us-central1",
Config = new Gcp.Composer.Inputs.EnvironmentConfigArgs
{
SoftwareConfig = new Gcp.Composer.Inputs.EnvironmentConfigSoftwareConfigArgs
{
ImageVersion = "composer-3-airflow-2",
},
WorkloadsConfig = new Gcp.Composer.Inputs.EnvironmentConfigWorkloadsConfigArgs
{
Scheduler = new Gcp.Composer.Inputs.EnvironmentConfigWorkloadsConfigSchedulerArgs
{
Cpu = 0.5,
MemoryGb = 2.0,
StorageGb = 1.0,
Count = 1,
},
Triggerer = new Gcp.Composer.Inputs.EnvironmentConfigWorkloadsConfigTriggererArgs
{
Cpu = 0.5,
MemoryGb = 1.0,
Count = 1,
},
DagProcessor = new Gcp.Composer.Inputs.EnvironmentConfigWorkloadsConfigDagProcessorArgs
{
Cpu = 1.0,
MemoryGb = 2.0,
StorageGb = 1.0,
Count = 1,
},
WebServer = new Gcp.Composer.Inputs.EnvironmentConfigWorkloadsConfigWebServerArgs
{
Cpu = 0.5,
MemoryGb = 2.0,
StorageGb = 1.0,
},
Worker = new Gcp.Composer.Inputs.EnvironmentConfigWorkloadsConfigWorkerArgs
{
Cpu = 0.5,
MemoryGb = 2.0,
StorageGb = 1.0,
MinCount = 1,
MaxCount = 3,
},
},
EnvironmentSize = "ENVIRONMENT_SIZE_SMALL",
NodeConfig = new Gcp.Composer.Inputs.EnvironmentConfigNodeConfigArgs
{
ServiceAccount = testAccount.Name,
},
},
});
var composer_worker = new Gcp.Projects.IAMMember("composer-worker", new()
{
Project = "your-project-id",
Role = "roles/composer.worker",
Member = testAccount.Email.Apply(email => $"serviceAccount:{email}"),
});
});
package main
import (
"fmt"
"github.com/pulumi/pulumi-gcp/sdk/v9/go/gcp/composer"
"github.com/pulumi/pulumi-gcp/sdk/v9/go/gcp/projects"
"github.com/pulumi/pulumi-gcp/sdk/v9/go/gcp/serviceaccount"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)
func main() {
pulumi.Run(func(ctx *pulumi.Context) error {
testAccount, err := serviceaccount.NewAccount(ctx, "test", &serviceaccount.AccountArgs{
AccountId: pulumi.String("composer-env-account"),
DisplayName: pulumi.String("Test Service Account for Managed Airflow Environment"),
})
if err != nil {
return err
}
_, err = composer.NewEnvironment(ctx, "test", &composer.EnvironmentArgs{
Name: pulumi.String("example-composer-env-tf-c3"),
Region: pulumi.String("us-central1"),
Config: &composer.EnvironmentConfigArgs{
SoftwareConfig: &composer.EnvironmentConfigSoftwareConfigArgs{
ImageVersion: pulumi.String("composer-3-airflow-2"),
},
WorkloadsConfig: &composer.EnvironmentConfigWorkloadsConfigArgs{
Scheduler: &composer.EnvironmentConfigWorkloadsConfigSchedulerArgs{
Cpu: pulumi.Float64(0.5),
MemoryGb: pulumi.Float64(2),
StorageGb: pulumi.Float64(1),
Count: pulumi.Int(1),
},
Triggerer: &composer.EnvironmentConfigWorkloadsConfigTriggererArgs{
Cpu: pulumi.Float64(0.5),
MemoryGb: pulumi.Float64(1),
Count: pulumi.Int(1),
},
DagProcessor: &composer.EnvironmentConfigWorkloadsConfigDagProcessorArgs{
Cpu: pulumi.Float64(1),
MemoryGb: pulumi.Float64(2),
StorageGb: pulumi.Float64(1),
Count: pulumi.Int(1),
},
WebServer: &composer.EnvironmentConfigWorkloadsConfigWebServerArgs{
Cpu: pulumi.Float64(0.5),
MemoryGb: pulumi.Float64(2),
StorageGb: pulumi.Float64(1),
},
Worker: &composer.EnvironmentConfigWorkloadsConfigWorkerArgs{
Cpu: pulumi.Float64(0.5),
MemoryGb: pulumi.Float64(2),
StorageGb: pulumi.Float64(1),
MinCount: pulumi.Int(1),
MaxCount: pulumi.Int(3),
},
},
EnvironmentSize: pulumi.String("ENVIRONMENT_SIZE_SMALL"),
NodeConfig: &composer.EnvironmentConfigNodeConfigArgs{
ServiceAccount: testAccount.Name,
},
},
})
if err != nil {
return err
}
_, err = projects.NewIAMMember(ctx, "composer-worker", &projects.IAMMemberArgs{
Project: pulumi.String("your-project-id"),
Role: pulumi.String("roles/composer.worker"),
Member: testAccount.Email.ApplyT(func(email string) (string, error) {
return fmt.Sprintf("serviceAccount:%v", email), nil
}).(pulumi.StringOutput),
})
if err != nil {
return err
}
return nil
})
}
pulumi {
required_providers {
gcp = {
source = "pulumi/gcp"
}
}
}
resource "gcp_composer_environment" "test" {
name = "example-composer-env-tf-c3"
region = "us-central1"
config = {
software_config = {
image_version = "composer-3-airflow-2"
}
workloads_config = {
scheduler = {
cpu = 0.5
memory_gb = 2
storage_gb = 1
count = 1
}
triggerer = {
cpu = 0.5
memory_gb = 1
count = 1
}
dag_processor = {
cpu = 1
memory_gb = 2
storage_gb = 1
count = 1
}
web_server = {
cpu = 0.5
memory_gb = 2
storage_gb = 1
}
worker = {
cpu = 0.5
memory_gb = 2
storage_gb = 1
min_count = 1
max_count = 3
}
}
environment_size = "ENVIRONMENT_SIZE_SMALL"
node_config = {
service_account = gcp_serviceaccount_account.test.name
}
}
}
resource "gcp_serviceaccount_account" "test" {
account_id = "composer-env-account"
display_name = "Test Service Account for Managed Airflow Environment"
}
resource "gcp_projects_iammember" "composer-worker" {
project = "your-project-id"
role = "roles/composer.worker"
member ="serviceAccount:${gcp_serviceaccount_account.test.email}"
}
package generated_program;
import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.gcp.serviceaccount.Account;
import com.pulumi.gcp.serviceaccount.AccountArgs;
import com.pulumi.gcp.composer.Environment;
import com.pulumi.gcp.composer.EnvironmentArgs;
import com.pulumi.gcp.composer.inputs.EnvironmentConfigArgs;
import com.pulumi.gcp.composer.inputs.EnvironmentConfigSoftwareConfigArgs;
import com.pulumi.gcp.composer.inputs.EnvironmentConfigWorkloadsConfigArgs;
import com.pulumi.gcp.composer.inputs.EnvironmentConfigWorkloadsConfigSchedulerArgs;
import com.pulumi.gcp.composer.inputs.EnvironmentConfigWorkloadsConfigTriggererArgs;
import com.pulumi.gcp.composer.inputs.EnvironmentConfigWorkloadsConfigDagProcessorArgs;
import com.pulumi.gcp.composer.inputs.EnvironmentConfigWorkloadsConfigWebServerArgs;
import com.pulumi.gcp.composer.inputs.EnvironmentConfigWorkloadsConfigWorkerArgs;
import com.pulumi.gcp.composer.inputs.EnvironmentConfigNodeConfigArgs;
import com.pulumi.gcp.projects.IAMMember;
import com.pulumi.gcp.projects.IAMMemberArgs;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;
public class App {
public static void main(String[] args) {
Pulumi.run(App::stack);
}
public static void stack(Context ctx) {
var testAccount = new Account("testAccount", AccountArgs.builder()
.accountId("composer-env-account")
.displayName("Test Service Account for Managed Airflow Environment")
.build());
var test = new Environment("test", EnvironmentArgs.builder()
.name("example-composer-env-tf-c3")
.region("us-central1")
.config(EnvironmentConfigArgs.builder()
.softwareConfig(EnvironmentConfigSoftwareConfigArgs.builder()
.imageVersion("composer-3-airflow-2")
.build())
.workloadsConfig(EnvironmentConfigWorkloadsConfigArgs.builder()
.scheduler(EnvironmentConfigWorkloadsConfigSchedulerArgs.builder()
.cpu(0.5)
.memoryGb(2.0)
.storageGb(1.0)
.count(1)
.build())
.triggerer(EnvironmentConfigWorkloadsConfigTriggererArgs.builder()
.cpu(0.5)
.memoryGb(1.0)
.count(1)
.build())
.dagProcessor(EnvironmentConfigWorkloadsConfigDagProcessorArgs.builder()
.cpu(1.0)
.memoryGb(2.0)
.storageGb(1.0)
.count(1)
.build())
.webServer(EnvironmentConfigWorkloadsConfigWebServerArgs.builder()
.cpu(0.5)
.memoryGb(2.0)
.storageGb(1.0)
.build())
.worker(EnvironmentConfigWorkloadsConfigWorkerArgs.builder()
.cpu(0.5)
.memoryGb(2.0)
.storageGb(1.0)
.minCount(1)
.maxCount(3)
.build())
.build())
.environmentSize("ENVIRONMENT_SIZE_SMALL")
.nodeConfig(EnvironmentConfigNodeConfigArgs.builder()
.serviceAccount(testAccount.name())
.build())
.build())
.build());
var composer_worker = new IAMMember("composer-worker", IAMMemberArgs.builder()
.project("your-project-id")
.role("roles/composer.worker")
.member(testAccount.email().applyValue(_email -> String.format("serviceAccount:%s", _email)))
.build());
}
}
resources:
test:
type: gcp:composer:Environment
properties:
name: example-composer-env-tf-c3
region: us-central1
config:
softwareConfig:
imageVersion: composer-3-airflow-2
workloadsConfig:
scheduler:
cpu: 0.5
memoryGb: 2
storageGb: 1
count: 1
triggerer:
cpu: 0.5
memoryGb: 1
count: 1
dagProcessor:
cpu: 1
memoryGb: 2
storageGb: 1
count: 1
webServer:
cpu: 0.5
memoryGb: 2
storageGb: 1
worker:
cpu: 0.5
memoryGb: 2
storageGb: 1
minCount: 1
maxCount: 3
environmentSize: ENVIRONMENT_SIZE_SMALL
nodeConfig:
serviceAccount: ${testAccount.name}
testAccount:
type: gcp:serviceaccount:Account
name: test
properties:
accountId: composer-env-account
displayName: Test Service Account for Managed Airflow Environment
composer-worker:
type: gcp:projects:IAMMember
properties:
project: your-project-id
role: roles/composer.worker
member: serviceAccount:${testAccount.email}
Environment resources configuration in Managed Airflow (Gen 2)
import * as pulumi from "@pulumi/pulumi";
import * as gcp from "@pulumi/gcp";
const testNetwork = new gcp.compute.Network("test", {
name: "composer-test-network3",
autoCreateSubnetworks: false,
});
const testSubnetwork = new gcp.compute.Subnetwork("test", {
name: "composer-test-subnetwork",
ipCidrRange: "10.2.0.0/16",
region: "us-central1",
network: testNetwork.id,
});
const testAccount = new gcp.serviceaccount.Account("test", {
accountId: "composer-env-account",
displayName: "Test Service Account for Managed Airflow",
});
const test = new gcp.composer.Environment("test", {
name: "example-composer-env-tf-c2",
region: "us-central1",
config: {
softwareConfig: {
imageVersion: "composer-2-airflow-2",
},
workloadsConfig: {
scheduler: {
cpu: 0.5,
memoryGb: 1.875,
storageGb: 1,
count: 1,
},
webServer: {
cpu: 0.5,
memoryGb: 1.875,
storageGb: 1,
},
worker: {
cpu: 0.5,
memoryGb: 1.875,
storageGb: 1,
minCount: 1,
maxCount: 3,
},
},
environmentSize: "ENVIRONMENT_SIZE_SMALL",
nodeConfig: {
network: testNetwork.id,
subnetwork: testSubnetwork.id,
serviceAccount: testAccount.name,
},
},
});
const composer_worker = new gcp.projects.IAMMember("composer-worker", {
project: "your-project-id",
role: "roles/composer.worker",
member: pulumi.interpolate`serviceAccount:${testAccount.email}`,
});
import pulumi
import pulumi_gcp as gcp
test_network = gcp.compute.Network("test",
name="composer-test-network3",
auto_create_subnetworks=False)
test_subnetwork = gcp.compute.Subnetwork("test",
name="composer-test-subnetwork",
ip_cidr_range="10.2.0.0/16",
region="us-central1",
network=test_network.id)
test_account = gcp.serviceaccount.Account("test",
account_id="composer-env-account",
display_name="Test Service Account for Managed Airflow")
test = gcp.composer.Environment("test",
name="example-composer-env-tf-c2",
region="us-central1",
config={
"software_config": {
"image_version": "composer-2-airflow-2",
},
"workloads_config": {
"scheduler": {
"cpu": 0.5,
"memory_gb": 1.875,
"storage_gb": float(1),
"count": 1,
},
"web_server": {
"cpu": 0.5,
"memory_gb": 1.875,
"storage_gb": float(1),
},
"worker": {
"cpu": 0.5,
"memory_gb": 1.875,
"storage_gb": float(1),
"min_count": 1,
"max_count": 3,
},
},
"environment_size": "ENVIRONMENT_SIZE_SMALL",
"node_config": {
"network": test_network.id,
"subnetwork": test_subnetwork.id,
"service_account": test_account.name,
},
})
composer_worker = gcp.projects.IAMMember("composer-worker",
project="your-project-id",
role="roles/composer.worker",
member=test_account.email.apply(lambda email: f"serviceAccount:{email}"))
using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Gcp = Pulumi.Gcp;
return await Deployment.RunAsync(() =>
{
var testNetwork = new Gcp.Compute.Network("test", new()
{
Name = "composer-test-network3",
AutoCreateSubnetworks = false,
});
var testSubnetwork = new Gcp.Compute.Subnetwork("test", new()
{
Name = "composer-test-subnetwork",
IpCidrRange = "10.2.0.0/16",
Region = "us-central1",
Network = testNetwork.Id,
});
var testAccount = new Gcp.ServiceAccount.Account("test", new()
{
AccountId = "composer-env-account",
DisplayName = "Test Service Account for Managed Airflow",
});
var test = new Gcp.Composer.Environment("test", new()
{
Name = "example-composer-env-tf-c2",
Region = "us-central1",
Config = new Gcp.Composer.Inputs.EnvironmentConfigArgs
{
SoftwareConfig = new Gcp.Composer.Inputs.EnvironmentConfigSoftwareConfigArgs
{
ImageVersion = "composer-2-airflow-2",
},
WorkloadsConfig = new Gcp.Composer.Inputs.EnvironmentConfigWorkloadsConfigArgs
{
Scheduler = new Gcp.Composer.Inputs.EnvironmentConfigWorkloadsConfigSchedulerArgs
{
Cpu = 0.5,
MemoryGb = 1.875,
StorageGb = 1.0,
Count = 1,
},
WebServer = new Gcp.Composer.Inputs.EnvironmentConfigWorkloadsConfigWebServerArgs
{
Cpu = 0.5,
MemoryGb = 1.875,
StorageGb = 1.0,
},
Worker = new Gcp.Composer.Inputs.EnvironmentConfigWorkloadsConfigWorkerArgs
{
Cpu = 0.5,
MemoryGb = 1.875,
StorageGb = 1.0,
MinCount = 1,
MaxCount = 3,
},
},
EnvironmentSize = "ENVIRONMENT_SIZE_SMALL",
NodeConfig = new Gcp.Composer.Inputs.EnvironmentConfigNodeConfigArgs
{
Network = testNetwork.Id,
Subnetwork = testSubnetwork.Id,
ServiceAccount = testAccount.Name,
},
},
});
var composer_worker = new Gcp.Projects.IAMMember("composer-worker", new()
{
Project = "your-project-id",
Role = "roles/composer.worker",
Member = testAccount.Email.Apply(email => $"serviceAccount:{email}"),
});
});
package main
import (
"fmt"
"github.com/pulumi/pulumi-gcp/sdk/v9/go/gcp/composer"
"github.com/pulumi/pulumi-gcp/sdk/v9/go/gcp/compute"
"github.com/pulumi/pulumi-gcp/sdk/v9/go/gcp/projects"
"github.com/pulumi/pulumi-gcp/sdk/v9/go/gcp/serviceaccount"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)
func main() {
pulumi.Run(func(ctx *pulumi.Context) error {
testNetwork, err := compute.NewNetwork(ctx, "test", &compute.NetworkArgs{
Name: pulumi.String("composer-test-network3"),
AutoCreateSubnetworks: pulumi.Bool(false),
})
if err != nil {
return err
}
testSubnetwork, err := compute.NewSubnetwork(ctx, "test", &compute.SubnetworkArgs{
Name: pulumi.String("composer-test-subnetwork"),
IpCidrRange: pulumi.String("10.2.0.0/16"),
Region: pulumi.String("us-central1"),
Network: testNetwork.ID().ToIDOutput().ToStringOutput(),
})
if err != nil {
return err
}
testAccount, err := serviceaccount.NewAccount(ctx, "test", &serviceaccount.AccountArgs{
AccountId: pulumi.String("composer-env-account"),
DisplayName: pulumi.String("Test Service Account for Managed Airflow"),
})
if err != nil {
return err
}
_, err = composer.NewEnvironment(ctx, "test", &composer.EnvironmentArgs{
Name: pulumi.String("example-composer-env-tf-c2"),
Region: pulumi.String("us-central1"),
Config: &composer.EnvironmentConfigArgs{
SoftwareConfig: &composer.EnvironmentConfigSoftwareConfigArgs{
ImageVersion: pulumi.String("composer-2-airflow-2"),
},
WorkloadsConfig: &composer.EnvironmentConfigWorkloadsConfigArgs{
Scheduler: &composer.EnvironmentConfigWorkloadsConfigSchedulerArgs{
Cpu: pulumi.Float64(0.5),
MemoryGb: pulumi.Float64(1.875),
StorageGb: pulumi.Float64(1),
Count: pulumi.Int(1),
},
WebServer: &composer.EnvironmentConfigWorkloadsConfigWebServerArgs{
Cpu: pulumi.Float64(0.5),
MemoryGb: pulumi.Float64(1.875),
StorageGb: pulumi.Float64(1),
},
Worker: &composer.EnvironmentConfigWorkloadsConfigWorkerArgs{
Cpu: pulumi.Float64(0.5),
MemoryGb: pulumi.Float64(1.875),
StorageGb: pulumi.Float64(1),
MinCount: pulumi.Int(1),
MaxCount: pulumi.Int(3),
},
},
EnvironmentSize: pulumi.String("ENVIRONMENT_SIZE_SMALL"),
NodeConfig: &composer.EnvironmentConfigNodeConfigArgs{
Network: testNetwork.ID().ToIDOutput().ToStringOutput(),
Subnetwork: testSubnetwork.ID().ToIDOutput().ToStringOutput(),
ServiceAccount: testAccount.Name,
},
},
})
if err != nil {
return err
}
_, err = projects.NewIAMMember(ctx, "composer-worker", &projects.IAMMemberArgs{
Project: pulumi.String("your-project-id"),
Role: pulumi.String("roles/composer.worker"),
Member: testAccount.Email.ApplyT(func(email string) (string, error) {
return fmt.Sprintf("serviceAccount:%v", email), nil
}).(pulumi.StringOutput),
})
if err != nil {
return err
}
return nil
})
}
pulumi {
required_providers {
gcp = {
source = "pulumi/gcp"
}
}
}
resource "gcp_composer_environment" "test" {
name = "example-composer-env-tf-c2"
region = "us-central1"
config = {
software_config = {
image_version = "composer-2-airflow-2"
}
workloads_config = {
scheduler = {
cpu = 0.5
memory_gb = 1.875
storage_gb = 1
count = 1
}
web_server = {
cpu = 0.5
memory_gb = 1.875
storage_gb = 1
}
worker = {
cpu = 0.5
memory_gb = 1.875
storage_gb = 1
min_count = 1
max_count = 3
}
}
environment_size = "ENVIRONMENT_SIZE_SMALL"
node_config = {
network = gcp_compute_network.test.id
subnetwork = gcp_compute_subnetwork.test.id
service_account = gcp_serviceaccount_account.test.name
}
}
}
resource "gcp_compute_network" "test" {
name = "composer-test-network3"
auto_create_subnetworks = false
}
resource "gcp_compute_subnetwork" "test" {
name = "composer-test-subnetwork"
ip_cidr_range = "10.2.0.0/16"
region = "us-central1"
network = gcp_compute_network.test.id
}
resource "gcp_serviceaccount_account" "test" {
account_id = "composer-env-account"
display_name = "Test Service Account for Managed Airflow"
}
resource "gcp_projects_iammember" "composer-worker" {
project = "your-project-id"
role = "roles/composer.worker"
member ="serviceAccount:${gcp_serviceaccount_account.test.email}"
}
package generated_program;
import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.gcp.compute.Network;
import com.pulumi.gcp.compute.NetworkArgs;
import com.pulumi.gcp.compute.Subnetwork;
import com.pulumi.gcp.compute.SubnetworkArgs;
import com.pulumi.gcp.serviceaccount.Account;
import com.pulumi.gcp.serviceaccount.AccountArgs;
import com.pulumi.gcp.composer.Environment;
import com.pulumi.gcp.composer.EnvironmentArgs;
import com.pulumi.gcp.composer.inputs.EnvironmentConfigArgs;
import com.pulumi.gcp.composer.inputs.EnvironmentConfigSoftwareConfigArgs;
import com.pulumi.gcp.composer.inputs.EnvironmentConfigWorkloadsConfigArgs;
import com.pulumi.gcp.composer.inputs.EnvironmentConfigWorkloadsConfigSchedulerArgs;
import com.pulumi.gcp.composer.inputs.EnvironmentConfigWorkloadsConfigWebServerArgs;
import com.pulumi.gcp.composer.inputs.EnvironmentConfigWorkloadsConfigWorkerArgs;
import com.pulumi.gcp.composer.inputs.EnvironmentConfigNodeConfigArgs;
import com.pulumi.gcp.projects.IAMMember;
import com.pulumi.gcp.projects.IAMMemberArgs;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;
public class App {
public static void main(String[] args) {
Pulumi.run(App::stack);
}
public static void stack(Context ctx) {
var testNetwork = new Network("testNetwork", NetworkArgs.builder()
.name("composer-test-network3")
.autoCreateSubnetworks(false)
.build());
var testSubnetwork = new Subnetwork("testSubnetwork", SubnetworkArgs.builder()
.name("composer-test-subnetwork")
.ipCidrRange("10.2.0.0/16")
.region("us-central1")
.network(testNetwork.id())
.build());
var testAccount = new Account("testAccount", AccountArgs.builder()
.accountId("composer-env-account")
.displayName("Test Service Account for Managed Airflow")
.build());
var test = new Environment("test", EnvironmentArgs.builder()
.name("example-composer-env-tf-c2")
.region("us-central1")
.config(EnvironmentConfigArgs.builder()
.softwareConfig(EnvironmentConfigSoftwareConfigArgs.builder()
.imageVersion("composer-2-airflow-2")
.build())
.workloadsConfig(EnvironmentConfigWorkloadsConfigArgs.builder()
.scheduler(EnvironmentConfigWorkloadsConfigSchedulerArgs.builder()
.cpu(0.5)
.memoryGb(1.875)
.storageGb(1.0)
.count(1)
.build())
.webServer(EnvironmentConfigWorkloadsConfigWebServerArgs.builder()
.cpu(0.5)
.memoryGb(1.875)
.storageGb(1.0)
.build())
.worker(EnvironmentConfigWorkloadsConfigWorkerArgs.builder()
.cpu(0.5)
.memoryGb(1.875)
.storageGb(1.0)
.minCount(1)
.maxCount(3)
.build())
.build())
.environmentSize("ENVIRONMENT_SIZE_SMALL")
.nodeConfig(EnvironmentConfigNodeConfigArgs.builder()
.network(testNetwork.id())
.subnetwork(testSubnetwork.id())
.serviceAccount(testAccount.name())
.build())
.build())
.build());
var composer_worker = new IAMMember("composer-worker", IAMMemberArgs.builder()
.project("your-project-id")
.role("roles/composer.worker")
.member(testAccount.email().applyValue(_email -> String.format("serviceAccount:%s", _email)))
.build());
}
}
resources:
test:
type: gcp:composer:Environment
properties:
name: example-composer-env-tf-c2
region: us-central1
config:
softwareConfig:
imageVersion: composer-2-airflow-2
workloadsConfig:
scheduler:
cpu: 0.5
memoryGb: 1.875
storageGb: 1
count: 1
webServer:
cpu: 0.5
memoryGb: 1.875
storageGb: 1
worker:
cpu: 0.5
memoryGb: 1.875
storageGb: 1
minCount: 1
maxCount: 3
environmentSize: ENVIRONMENT_SIZE_SMALL
nodeConfig:
network: ${testNetwork.id}
subnetwork: ${testSubnetwork.id}
serviceAccount: ${testAccount.name}
testNetwork:
type: gcp:compute:Network
name: test
properties:
name: composer-test-network3
autoCreateSubnetworks: false
testSubnetwork:
type: gcp:compute:Subnetwork
name: test
properties:
name: composer-test-subnetwork
ipCidrRange: 10.2.0.0/16
region: us-central1
network: ${testNetwork.id}
testAccount:
type: gcp:serviceaccount:Account
name: test
properties:
accountId: composer-env-account
displayName: Test Service Account for Managed Airflow
composer-worker:
type: gcp:projects:IAMMember
properties:
project: your-project-id
role: roles/composer.worker
member: serviceAccount:${testAccount.email}
Environment resources configuration in Managed Airflow (Legacy Gen 1)
import * as pulumi from "@pulumi/pulumi";
import * as gcp from "@pulumi/gcp";
const testNetwork = new gcp.compute.Network("test", {
name: "composer-test-network",
autoCreateSubnetworks: false,
});
const testSubnetwork = new gcp.compute.Subnetwork("test", {
name: "composer-test-subnetwork",
ipCidrRange: "10.2.0.0/16",
region: "us-central1",
network: testNetwork.id,
});
const testAccount = new gcp.serviceaccount.Account("test", {
accountId: "composer-env-account",
displayName: "Test Service Account for Managed Airflow",
});
const test = new gcp.composer.Environment("test", {
name: "example-composer-env",
region: "us-central1",
config: {
softwareConfig: {
imageVersion: "composer-1-airflow-2",
},
nodeCount: 4,
nodeConfig: {
zone: "us-central1-a",
machineType: "n1-standard-1",
network: testNetwork.id,
subnetwork: testSubnetwork.id,
serviceAccount: testAccount.name,
},
databaseConfig: {
machineType: "db-n1-standard-2",
},
webServerConfig: {
machineType: "composer-n1-webserver-2",
},
},
});
const composer_worker = new gcp.projects.IAMMember("composer-worker", {
role: "roles/composer.worker",
member: pulumi.interpolate`serviceAccount:${testAccount.email}`,
});
import pulumi
import pulumi_gcp as gcp
test_network = gcp.compute.Network("test",
name="composer-test-network",
auto_create_subnetworks=False)
test_subnetwork = gcp.compute.Subnetwork("test",
name="composer-test-subnetwork",
ip_cidr_range="10.2.0.0/16",
region="us-central1",
network=test_network.id)
test_account = gcp.serviceaccount.Account("test",
account_id="composer-env-account",
display_name="Test Service Account for Managed Airflow")
test = gcp.composer.Environment("test",
name="example-composer-env",
region="us-central1",
config={
"software_config": {
"image_version": "composer-1-airflow-2",
},
"node_count": 4,
"node_config": {
"zone": "us-central1-a",
"machine_type": "n1-standard-1",
"network": test_network.id,
"subnetwork": test_subnetwork.id,
"service_account": test_account.name,
},
"database_config": {
"machine_type": "db-n1-standard-2",
},
"web_server_config": {
"machine_type": "composer-n1-webserver-2",
},
})
composer_worker = gcp.projects.IAMMember("composer-worker",
role="roles/composer.worker",
member=test_account.email.apply(lambda email: f"serviceAccount:{email}"))
using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Gcp = Pulumi.Gcp;
return await Deployment.RunAsync(() =>
{
var testNetwork = new Gcp.Compute.Network("test", new()
{
Name = "composer-test-network",
AutoCreateSubnetworks = false,
});
var testSubnetwork = new Gcp.Compute.Subnetwork("test", new()
{
Name = "composer-test-subnetwork",
IpCidrRange = "10.2.0.0/16",
Region = "us-central1",
Network = testNetwork.Id,
});
var testAccount = new Gcp.ServiceAccount.Account("test", new()
{
AccountId = "composer-env-account",
DisplayName = "Test Service Account for Managed Airflow",
});
var test = new Gcp.Composer.Environment("test", new()
{
Name = "example-composer-env",
Region = "us-central1",
Config = new Gcp.Composer.Inputs.EnvironmentConfigArgs
{
SoftwareConfig = new Gcp.Composer.Inputs.EnvironmentConfigSoftwareConfigArgs
{
ImageVersion = "composer-1-airflow-2",
},
NodeCount = 4,
NodeConfig = new Gcp.Composer.Inputs.EnvironmentConfigNodeConfigArgs
{
Zone = "us-central1-a",
MachineType = "n1-standard-1",
Network = testNetwork.Id,
Subnetwork = testSubnetwork.Id,
ServiceAccount = testAccount.Name,
},
DatabaseConfig = new Gcp.Composer.Inputs.EnvironmentConfigDatabaseConfigArgs
{
MachineType = "db-n1-standard-2",
},
WebServerConfig = new Gcp.Composer.Inputs.EnvironmentConfigWebServerConfigArgs
{
MachineType = "composer-n1-webserver-2",
},
},
});
var composer_worker = new Gcp.Projects.IAMMember("composer-worker", new()
{
Role = "roles/composer.worker",
Member = testAccount.Email.Apply(email => $"serviceAccount:{email}"),
});
});
package main
import (
"fmt"
"github.com/pulumi/pulumi-gcp/sdk/v9/go/gcp/composer"
"github.com/pulumi/pulumi-gcp/sdk/v9/go/gcp/compute"
"github.com/pulumi/pulumi-gcp/sdk/v9/go/gcp/projects"
"github.com/pulumi/pulumi-gcp/sdk/v9/go/gcp/serviceaccount"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)
func main() {
pulumi.Run(func(ctx *pulumi.Context) error {
testNetwork, err := compute.NewNetwork(ctx, "test", &compute.NetworkArgs{
Name: pulumi.String("composer-test-network"),
AutoCreateSubnetworks: pulumi.Bool(false),
})
if err != nil {
return err
}
testSubnetwork, err := compute.NewSubnetwork(ctx, "test", &compute.SubnetworkArgs{
Name: pulumi.String("composer-test-subnetwork"),
IpCidrRange: pulumi.String("10.2.0.0/16"),
Region: pulumi.String("us-central1"),
Network: testNetwork.ID().ToIDOutput().ToStringOutput(),
})
if err != nil {
return err
}
testAccount, err := serviceaccount.NewAccount(ctx, "test", &serviceaccount.AccountArgs{
AccountId: pulumi.String("composer-env-account"),
DisplayName: pulumi.String("Test Service Account for Managed Airflow"),
})
if err != nil {
return err
}
_, err = composer.NewEnvironment(ctx, "test", &composer.EnvironmentArgs{
Name: pulumi.String("example-composer-env"),
Region: pulumi.String("us-central1"),
Config: &composer.EnvironmentConfigArgs{
SoftwareConfig: &composer.EnvironmentConfigSoftwareConfigArgs{
ImageVersion: pulumi.String("composer-1-airflow-2"),
},
NodeCount: pulumi.Int(4),
NodeConfig: &composer.EnvironmentConfigNodeConfigArgs{
Zone: pulumi.String("us-central1-a"),
MachineType: pulumi.String("n1-standard-1"),
Network: testNetwork.ID().ToIDOutput().ToStringOutput(),
Subnetwork: testSubnetwork.ID().ToIDOutput().ToStringOutput(),
ServiceAccount: testAccount.Name,
},
DatabaseConfig: &composer.EnvironmentConfigDatabaseConfigArgs{
MachineType: pulumi.String("db-n1-standard-2"),
},
WebServerConfig: &composer.EnvironmentConfigWebServerConfigArgs{
MachineType: pulumi.String("composer-n1-webserver-2"),
},
},
})
if err != nil {
return err
}
_, err = projects.NewIAMMember(ctx, "composer-worker", &projects.IAMMemberArgs{
Role: pulumi.String("roles/composer.worker"),
Member: testAccount.Email.ApplyT(func(email string) (string, error) {
return fmt.Sprintf("serviceAccount:%v", email), nil
}).(pulumi.StringOutput),
})
if err != nil {
return err
}
return nil
})
}
pulumi {
required_providers {
gcp = {
source = "pulumi/gcp"
}
}
}
resource "gcp_composer_environment" "test" {
name = "example-composer-env"
region = "us-central1"
config = {
software_config = {
image_version = "composer-1-airflow-2"
}
node_count = 4
node_config = {
zone = "us-central1-a"
machine_type = "n1-standard-1"
network = gcp_compute_network.test.id
subnetwork = gcp_compute_subnetwork.test.id
service_account = gcp_serviceaccount_account.test.name
}
database_config = {
machine_type = "db-n1-standard-2"
}
web_server_config = {
machine_type = "composer-n1-webserver-2"
}
}
}
resource "gcp_compute_network" "test" {
name = "composer-test-network"
auto_create_subnetworks = false
}
resource "gcp_compute_subnetwork" "test" {
name = "composer-test-subnetwork"
ip_cidr_range = "10.2.0.0/16"
region = "us-central1"
network = gcp_compute_network.test.id
}
resource "gcp_serviceaccount_account" "test" {
account_id = "composer-env-account"
display_name = "Test Service Account for Managed Airflow"
}
resource "gcp_projects_iammember" "composer-worker" {
role = "roles/composer.worker"
member ="serviceAccount:${gcp_serviceaccount_account.test.email}"
}
package generated_program;
import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.gcp.compute.Network;
import com.pulumi.gcp.compute.NetworkArgs;
import com.pulumi.gcp.compute.Subnetwork;
import com.pulumi.gcp.compute.SubnetworkArgs;
import com.pulumi.gcp.serviceaccount.Account;
import com.pulumi.gcp.serviceaccount.AccountArgs;
import com.pulumi.gcp.composer.Environment;
import com.pulumi.gcp.composer.EnvironmentArgs;
import com.pulumi.gcp.composer.inputs.EnvironmentConfigArgs;
import com.pulumi.gcp.composer.inputs.EnvironmentConfigSoftwareConfigArgs;
import com.pulumi.gcp.composer.inputs.EnvironmentConfigNodeConfigArgs;
import com.pulumi.gcp.composer.inputs.EnvironmentConfigDatabaseConfigArgs;
import com.pulumi.gcp.composer.inputs.EnvironmentConfigWebServerConfigArgs;
import com.pulumi.gcp.projects.IAMMember;
import com.pulumi.gcp.projects.IAMMemberArgs;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;
public class App {
public static void main(String[] args) {
Pulumi.run(App::stack);
}
public static void stack(Context ctx) {
var testNetwork = new Network("testNetwork", NetworkArgs.builder()
.name("composer-test-network")
.autoCreateSubnetworks(false)
.build());
var testSubnetwork = new Subnetwork("testSubnetwork", SubnetworkArgs.builder()
.name("composer-test-subnetwork")
.ipCidrRange("10.2.0.0/16")
.region("us-central1")
.network(testNetwork.id())
.build());
var testAccount = new Account("testAccount", AccountArgs.builder()
.accountId("composer-env-account")
.displayName("Test Service Account for Managed Airflow")
.build());
var test = new Environment("test", EnvironmentArgs.builder()
.name("example-composer-env")
.region("us-central1")
.config(EnvironmentConfigArgs.builder()
.softwareConfig(EnvironmentConfigSoftwareConfigArgs.builder()
.imageVersion("composer-1-airflow-2")
.build())
.nodeCount(4)
.nodeConfig(EnvironmentConfigNodeConfigArgs.builder()
.zone("us-central1-a")
.machineType("n1-standard-1")
.network(testNetwork.id())
.subnetwork(testSubnetwork.id())
.serviceAccount(testAccount.name())
.build())
.databaseConfig(EnvironmentConfigDatabaseConfigArgs.builder()
.machineType("db-n1-standard-2")
.build())
.webServerConfig(EnvironmentConfigWebServerConfigArgs.builder()
.machineType("composer-n1-webserver-2")
.build())
.build())
.build());
var composer_worker = new IAMMember("composer-worker", IAMMemberArgs.builder()
.role("roles/composer.worker")
.member(testAccount.email().applyValue(_email -> String.format("serviceAccount:%s", _email)))
.build());
}
}
resources:
test:
type: gcp:composer:Environment
properties:
name: example-composer-env
region: us-central1
config:
softwareConfig:
imageVersion: composer-1-airflow-2
nodeCount: 4
nodeConfig:
zone: us-central1-a
machineType: n1-standard-1
network: ${testNetwork.id}
subnetwork: ${testSubnetwork.id}
serviceAccount: ${testAccount.name}
databaseConfig:
machineType: db-n1-standard-2
webServerConfig:
machineType: composer-n1-webserver-2
testNetwork:
type: gcp:compute:Network
name: test
properties:
name: composer-test-network
autoCreateSubnetworks: false
testSubnetwork:
type: gcp:compute:Subnetwork
name: test
properties:
name: composer-test-subnetwork
ipCidrRange: 10.2.0.0/16
region: us-central1
network: ${testNetwork.id}
testAccount:
type: gcp:serviceaccount:Account
name: test
properties:
accountId: composer-env-account
displayName: Test Service Account for Managed Airflow
composer-worker:
type: gcp:projects:IAMMember
properties:
role: roles/composer.worker
member: serviceAccount:${testAccount.email}
Networking configuration in Managed Airflow (Gen 3)
In Managed Airflow (Gen 3), networking configuration is simplified compared to previous versions. You don't need to specify network ranges, and can attach custom VPC networks to your environment.
> Note It's not possible to detach a VPC network using Terraform. Instead, you can attach a different VPC network in its place, or detach the network using other tools like Google Cloud CLI.
Use Private IP networking:
import * as pulumi from "@pulumi/pulumi";
import * as gcp from "@pulumi/gcp";
const example = new gcp.composer.Environment("example", {
name: "example-environment",
region: "us-central1",
config: {
enablePrivateEnvironment: true,
},
});
import pulumi
import pulumi_gcp as gcp
example = gcp.composer.Environment("example",
name="example-environment",
region="us-central1",
config={
"enable_private_environment": True,
})
using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Gcp = Pulumi.Gcp;
return await Deployment.RunAsync(() =>
{
var example = new Gcp.Composer.Environment("example", new()
{
Name = "example-environment",
Region = "us-central1",
Config = new Gcp.Composer.Inputs.EnvironmentConfigArgs
{
EnablePrivateEnvironment = true,
},
});
});
package main
import (
"github.com/pulumi/pulumi-gcp/sdk/v9/go/gcp/composer"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)
func main() {
pulumi.Run(func(ctx *pulumi.Context) error {
_, err := composer.NewEnvironment(ctx, "example", &composer.EnvironmentArgs{
Name: pulumi.String("example-environment"),
Region: pulumi.String("us-central1"),
Config: &composer.EnvironmentConfigArgs{
EnablePrivateEnvironment: pulumi.Bool(true),
},
})
if err != nil {
return err
}
return nil
})
}
pulumi {
required_providers {
gcp = {
source = "pulumi/gcp"
}
}
}
resource "gcp_composer_environment" "example" {
name = "example-environment"
region = "us-central1"
config = {
enable_private_environment = true
}
}
package generated_program;
import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.gcp.composer.Environment;
import com.pulumi.gcp.composer.EnvironmentArgs;
import com.pulumi.gcp.composer.inputs.EnvironmentConfigArgs;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;
public class App {
public static void main(String[] args) {
Pulumi.run(App::stack);
}
public static void stack(Context ctx) {
var example = new Environment("example", EnvironmentArgs.builder()
.name("example-environment")
.region("us-central1")
.config(EnvironmentConfigArgs.builder()
.enablePrivateEnvironment(true)
.build())
.build());
}
}
resources:
example:
type: gcp:composer:Environment
properties:
name: example-environment
region: us-central1
config:
enablePrivateEnvironment: true
Attach a custom VPC network (Managed Airflow creates a new network attachment):
import * as pulumi from "@pulumi/pulumi";
import * as gcp from "@pulumi/gcp";
const example = new gcp.composer.Environment("example", {
name: "example-environment",
region: "us-central1",
config: {
nodeConfig: {
network: "projects/example-project/global/networks/example-network",
subnetwork: "projects/example-project/regions/us-central1/subnetworks/example-subnetwork",
},
},
});
import pulumi
import pulumi_gcp as gcp
example = gcp.composer.Environment("example",
name="example-environment",
region="us-central1",
config={
"node_config": {
"network": "projects/example-project/global/networks/example-network",
"subnetwork": "projects/example-project/regions/us-central1/subnetworks/example-subnetwork",
},
})
using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Gcp = Pulumi.Gcp;
return await Deployment.RunAsync(() =>
{
var example = new Gcp.Composer.Environment("example", new()
{
Name = "example-environment",
Region = "us-central1",
Config = new Gcp.Composer.Inputs.EnvironmentConfigArgs
{
NodeConfig = new Gcp.Composer.Inputs.EnvironmentConfigNodeConfigArgs
{
Network = "projects/example-project/global/networks/example-network",
Subnetwork = "projects/example-project/regions/us-central1/subnetworks/example-subnetwork",
},
},
});
});
package main
import (
"github.com/pulumi/pulumi-gcp/sdk/v9/go/gcp/composer"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)
func main() {
pulumi.Run(func(ctx *pulumi.Context) error {
_, err := composer.NewEnvironment(ctx, "example", &composer.EnvironmentArgs{
Name: pulumi.String("example-environment"),
Region: pulumi.String("us-central1"),
Config: &composer.EnvironmentConfigArgs{
NodeConfig: &composer.EnvironmentConfigNodeConfigArgs{
Network: pulumi.String("projects/example-project/global/networks/example-network"),
Subnetwork: pulumi.String("projects/example-project/regions/us-central1/subnetworks/example-subnetwork"),
},
},
})
if err != nil {
return err
}
return nil
})
}
pulumi {
required_providers {
gcp = {
source = "pulumi/gcp"
}
}
}
resource "gcp_composer_environment" "example" {
name = "example-environment"
region = "us-central1"
config = {
node_config = {
network = "projects/example-project/global/networks/example-network"
subnetwork = "projects/example-project/regions/us-central1/subnetworks/example-subnetwork"
}
}
}
package generated_program;
import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.gcp.composer.Environment;
import com.pulumi.gcp.composer.EnvironmentArgs;
import com.pulumi.gcp.composer.inputs.EnvironmentConfigArgs;
import com.pulumi.gcp.composer.inputs.EnvironmentConfigNodeConfigArgs;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;
public class App {
public static void main(String[] args) {
Pulumi.run(App::stack);
}
public static void stack(Context ctx) {
var example = new Environment("example", EnvironmentArgs.builder()
.name("example-environment")
.region("us-central1")
.config(EnvironmentConfigArgs.builder()
.nodeConfig(EnvironmentConfigNodeConfigArgs.builder()
.network("projects/example-project/global/networks/example-network")
.subnetwork("projects/example-project/regions/us-central1/subnetworks/example-subnetwork")
.build())
.build())
.build());
}
}
resources:
example:
type: gcp:composer:Environment
properties:
name: example-environment
region: us-central1
config:
nodeConfig:
network: projects/example-project/global/networks/example-network
subnetwork: projects/example-project/regions/us-central1/subnetworks/example-subnetwork
Attach a custom VPC network (use an existing network attachment):
import * as pulumi from "@pulumi/pulumi";
import * as gcp from "@pulumi/gcp";
const example = new gcp.composer.Environment("example", {
name: "example-environment",
region: "us-central1",
config: {
nodeConfig: {
composerNetworkAttachment: String(Number(projects) / Number(example_project) / Number(regions) / Number(us_central1) / Number(networkAttachments) / Number(example_network_attachment)),
},
},
});
import pulumi
import pulumi_gcp as gcp
example = gcp.composer.Environment("example",
name="example-environment",
region="us-central1",
config={
"node_config": {
"composer_network_attachment": str(float(projects) / float(example_project) / float(regions) / float(us_central1) / float(network_attachments) / float(example_network_attachment)),
},
})
using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Gcp = Pulumi.Gcp;
return await Deployment.RunAsync(() =>
{
var example = new Gcp.Composer.Environment("example", new()
{
Name = "example-environment",
Region = "us-central1",
Config = new Gcp.Composer.Inputs.EnvironmentConfigArgs
{
NodeConfig = new Gcp.Composer.Inputs.EnvironmentConfigNodeConfigArgs
{
ComposerNetworkAttachment = (double.Parse(projects, System.Globalization.CultureInfo.InvariantCulture) / double.Parse(example_project, System.Globalization.CultureInfo.InvariantCulture) / double.Parse(regions, System.Globalization.CultureInfo.InvariantCulture) / double.Parse(us_central1, System.Globalization.CultureInfo.InvariantCulture) / double.Parse(networkAttachments, System.Globalization.CultureInfo.InvariantCulture) / double.Parse(example_network_attachment, System.Globalization.CultureInfo.InvariantCulture)).ToString(System.Globalization.CultureInfo.InvariantCulture),
},
},
});
});
package main
import (
"github.com/pulumi/pulumi-gcp/sdk/v9/go/gcp/composer"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)
func main() {
pulumi.Run(func(ctx *pulumi.Context) error {
_, err := composer.NewEnvironment(ctx, "example", &composer.EnvironmentArgs{
Name: pulumi.String("example-environment"),
Region: pulumi.String("us-central1"),
Config: &composer.EnvironmentConfigArgs{
NodeConfig: &composer.EnvironmentConfigNodeConfigArgs{
ComposerNetworkAttachment: pulumi.String(projects / example_project / regions / us_central1 / networkAttachments / example_network_attachment),
},
},
})
if err != nil {
return err
}
return nil
})
}
pulumi {
required_providers {
gcp = {
source = "pulumi/gcp"
}
}
}
resource "gcp_composer_environment" "example" {
name = "example-environment"
region = "us-central1"
config = {
node_config = {
composer_network_attachment = projects / example-project / regions / us-central1 / networkAttachments / example-network-attachment
}
}
}
package generated_program;
import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.gcp.composer.Environment;
import com.pulumi.gcp.composer.EnvironmentArgs;
import com.pulumi.gcp.composer.inputs.EnvironmentConfigArgs;
import com.pulumi.gcp.composer.inputs.EnvironmentConfigNodeConfigArgs;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;
public class App {
public static void main(String[] args) {
Pulumi.run(App::stack);
}
public static void stack(Context ctx) {
var example = new Environment("example", EnvironmentArgs.builder()
.name("example-environment")
.region("us-central1")
.config(EnvironmentConfigArgs.builder()
.nodeConfig(EnvironmentConfigNodeConfigArgs.builder()
.composerNetworkAttachment(Output.tuple(((Number) projects).doubleValue(), ((Number) example_project).doubleValue(), ((Number) regions).doubleValue(), ((Number) us_central1).doubleValue(), ((Number) networkAttachments).doubleValue(), ((Number) example_network_attachment).doubleValue()).applyValue(values -> {
var __convert = values.t1;
var __convert1 = values.t2;
var __convert2 = values.t3;
var __convert3 = values.t4;
var __convert4 = values.t5;
var __convert5 = values.t6;
return __convert / __convert1 / __convert2 / __convert3 / __convert4 / __convert5;
}).applyValue(_n -> _n))
.build())
.build())
.build());
}
}
If you specify an existing network attachment that you also manage in
Terraform, then Terraform will revert changes to the attachment that were done
by Managed Airflow when you apply configuration changes. As a result, the
environment will no longer use the attachment. To address this, make sure that
Terraform ignores changes to the producerAcceptLists parameter of the
attachment, as follows:
import * as pulumi from "@pulumi/pulumi";
import * as gcp from "@pulumi/gcp";
const example = new gcp.compute.NetworkAttachment("example", {});
const exampleEnvironment = new gcp.composer.Environment("example", {
name: "example-environment",
region: "us-central1",
config: {
nodeConfig: {
composerNetworkAttachment: example.id,
},
},
});
import pulumi
import pulumi_gcp as gcp
example = gcp.compute.NetworkAttachment("example")
example_environment = gcp.composer.Environment("example",
name="example-environment",
region="us-central1",
config={
"node_config": {
"composer_network_attachment": example.id,
},
})
using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Gcp = Pulumi.Gcp;
return await Deployment.RunAsync(() =>
{
var example = new Gcp.Compute.NetworkAttachment("example");
var exampleEnvironment = new Gcp.Composer.Environment("example", new()
{
Name = "example-environment",
Region = "us-central1",
Config = new Gcp.Composer.Inputs.EnvironmentConfigArgs
{
NodeConfig = new Gcp.Composer.Inputs.EnvironmentConfigNodeConfigArgs
{
ComposerNetworkAttachment = example.Id,
},
},
});
});
package main
import (
"github.com/pulumi/pulumi-gcp/sdk/v9/go/gcp/composer"
"github.com/pulumi/pulumi-gcp/sdk/v9/go/gcp/compute"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)
func main() {
pulumi.Run(func(ctx *pulumi.Context) error {
example, err := compute.NewNetworkAttachment(ctx, "example", nil)
if err != nil {
return err
}
_, err = composer.NewEnvironment(ctx, "example", &composer.EnvironmentArgs{
Name: pulumi.String("example-environment"),
Region: pulumi.String("us-central1"),
Config: &composer.EnvironmentConfigArgs{
NodeConfig: &composer.EnvironmentConfigNodeConfigArgs{
ComposerNetworkAttachment: example.ID().ToIDOutput().ToStringOutput(),
},
},
})
if err != nil {
return err
}
return nil
})
}
pulumi {
required_providers {
gcp = {
source = "pulumi/gcp"
}
}
}
resource "gcp_compute_networkattachment" "example" {
}
resource "gcp_composer_environment" "example" {
name = "example-environment"
region = "us-central1"
config = {
node_config = {
composer_network_attachment = gcp_compute_networkattachment.example.id
}
}
}
package generated_program;
import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.gcp.compute.NetworkAttachment;
import com.pulumi.gcp.composer.Environment;
import com.pulumi.gcp.composer.EnvironmentArgs;
import com.pulumi.gcp.composer.inputs.EnvironmentConfigArgs;
import com.pulumi.gcp.composer.inputs.EnvironmentConfigNodeConfigArgs;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;
public class App {
public static void main(String[] args) {
Pulumi.run(App::stack);
}
public static void stack(Context ctx) {
var example = new NetworkAttachment("example");
var exampleEnvironment = new Environment("exampleEnvironment", EnvironmentArgs.builder()
.name("example-environment")
.region("us-central1")
.config(EnvironmentConfigArgs.builder()
.nodeConfig(EnvironmentConfigNodeConfigArgs.builder()
.composerNetworkAttachment(example.id())
.build())
.build())
.build());
}
}
resources:
example:
type: gcp:compute:NetworkAttachment
exampleEnvironment:
type: gcp:composer:Environment
name: example
properties:
name: example-environment
region: us-central1
config:
nodeConfig:
composerNetworkAttachment: ${example.id}
Import
Environment can be imported using any of these accepted formats:
projects/{{project}}/locations/{{region}}/environments/{{name}}{{project}}/{{region}}/{{name}}{{name}}
When using the
pulumi import command,
Environment can be imported using one of the formats above. For example:
$ pulumi import gcp:composer/environment:Environment default projects/{{project}}/locations/{{region}}/environments/{{name}}
$ pulumi import gcp:composer/environment:Environment default {{project}}/{{region}}/{{name}}
$ pulumi import gcp:composer/environment:Environment default {{name}}
Constructors
- Environment(String name, {EnvironmentArgs? args, CustomResourceOptions? options})
-
Creates a new Environment.
nameThe Pulumi resource name.argsArguments used to configure this Environment. The set of arguments for Environment.optionsResource options controlling this resource's behavior. - Environment.reference(String urn)
- Creates a typed reference to an existing Environment resource.
Properties
-
childResources
→ Set<
Resource> -
finalinherited
-
completionSources
↔ Map<
String, IOutputCompletionSource> -
latefinalinherited
-
config
↔ Output<
EnvironmentConfig> -
Configuration parameters for this environment.
latefinal
-
deletionPolicy
↔ Output<
String> -
Whether Terraform will be prevented from destroying the instance. Defaults to "DELETE".
When a 'terraform destroy' or 'terraform apply' would delete the instance,
the command will fail if this field is set to "PREVENT" in Terraform state.
When set to "ABANDON", the command will remove the resource from Terraform
management without updating or deleting the resource in the API.
When set to "DELETE", deleting the resource is allowed.
latefinal
-
effectiveLabels
↔ Output<
Map< String, String> > -
All of labels (key/value pairs) present on the resource in GCP, including the labels configured through Terraform, other clients and services.
latefinal
- hashCode → int
-
The hash code for this object.
no setterinherited
-
id
↔ Output<
String> -
getter/setter pairinherited
- isCustom → bool
-
Returns whether this resource is provider-managed.
no setterinherited
- isProtected → bool
-
Returns whether this resource is protected from deletion.
no setterinherited
- isRemote → bool
-
Whether this resource is registered as remote.
no setterinherited
- isResourceReference → bool
-
Whether this instance represents a resource value returned over RPC.
finalinherited
-
labels
↔ Output<
Map< String, String> ?> -
User-defined labels for this environment. The labels map can contain no more than 64 entries. Entries of the labels map are UTF8 strings that comply with the following restrictions: Label keys must be between 1 and 63 characters long and must conform to the following regular expression: a-z?. Label values must be between 0 and 63 characters long and must conform to the regular expression (a-z?)?. No more than 64 labels can be associated with a given environment. Both keys and values must be <= 128 bytes in size.
latefinal
-
name
↔ Output<
String> -
Name of the environment.
latefinal
-
project
↔ Output<
String> -
The ID of the project in which the resource belongs. If it is not provided, the provider project is used.
latefinal
-
pulumiLabels
↔ Output<
Map< String, String> > -
The combination of labels configured directly on the resource and default labels configured on the provider.
latefinal
-
region
↔ Output<
String> -
The location or Compute Engine region for the environment.
latefinal
-
resourceTransforms
→ List<
ResourceTransform> -
Inherited/explicit async transforms.
no setterinherited
- runtimeType → Type
-
A representation of the runtime type of the object.
no setterinherited
-
storageConfig
↔ Output<
EnvironmentStorageConfig> -
Configuration options for storage used by Composer environment.
latefinal
-
transformations
→ List<
ResourceTransformation> -
Inherited/explicit legacy transformations.
no setterinherited
-
urn
↔ Output<
String> -
latefinalinherited
Methods
-
failId(
Object error) → void -
Completes this resource ID with an error when registration fails.
inherited
-
failOutputs(
Object error) → void -
Completes all output properties with
error.inherited -
failUrn(
Object error) → void -
Completes this resource URN with an error when registration fails.
inherited
-
getProvider(
String moduleMember) → ProviderResource? -
Returns provider for
moduleMember's package, if configured.inherited -
getResourceName(
) → String -
Returns this resource's logical name.
inherited
-
getResourceType(
) → String -
Returns this resource's Pulumi type token.
inherited
-
noSuchMethod(
Invocation invocation) → dynamic -
Invoked when a nonexistent method or property is accessed.
inherited
-
registerOutput<
T> (String propertyName, {Object? decoder(Object?)?, bool isSecret = false}) → Output< T> -
Registers a dynamic output property for this resource.
inherited
-
resolveId(
String? value, {required bool isKnown}) → void -
Resolves the provider-assigned ID for this resource.
inherited
-
resolveOutputs(
Struct outputs) → void -
Resolves all output properties from a monitor response payload.
inherited
-
resolveUrn(
String value) → void -
Resolves this resource's URN once assigned by the engine.
inherited
-
serializeProperties(
Map< String, dynamic> properties) → Future<Struct> -
Serializes resource properties for RPC transmission.
inherited
-
toString(
) → String -
A string representation of this object.
inherited
Operators
-
operator ==(
Object other) → bool -
The equality operator.
inherited
Static Methods
-
get(
String name, Input< String> id, {EnvironmentState? state, CustomResourceOptions? options}) → Environment -
Gets an existing Environment resource's state with the given
nameandid.