ExternalDataConfiguration class
Constructors
-
ExternalDataConfiguration({Input<bool>? autodetect, Input<AvroOptions>? avroOptions, Input<BigtableOptions>? bigtableOptions, Input<String>? compression, Input<String>? connectionId, Input<CsvOptions>? csvOptions, Input<List<String>>? decimalTargetTypes, Input<String>? fileSetSpecType, Input<GoogleSheetsOptions>? googleSheetsOptions, Input<HivePartitioningOptions>? hivePartitioningOptions, Input<bool>? ignoreUnknownValues, Input<JsonOptions>? jsonOptions, Input<int>? maxBadRecords, Input<String>? metadataCacheMode, Input<String>? objectMetadata, Input<ParquetOptions>? parquetOptions, Input<String>? referenceFileSchemaUri, Input<TableSchema>? schema, Input<String>? sourceFormat, Input<List<String>>? sourceUris})
-
Creates a new ExternalDataConfiguration.
autodetect Try to detect schema and format options automatically. Any option specified explicitly will be honored.
avroOptions Additional properties to set if sourceFormat is set to Avro.
bigtableOptions Optional Additional options if sourceFormat is set to BIGTABLE.
compression Optional The compression type of the data source. Possible values include GZIP and NONE. The default value is NONE. This setting is ignored for Google Cloud Bigtable, Google Cloud Datastore backups and Avro formats.
connectionId Optional, Trusted Tester Connection for external data source.
csvOptions Additional properties to set if sourceFormat is set to CSV.
decimalTargetTypes Optional Defines the list of possible SQL data types to which the source decimal values are converted. This list and the precision and the scale parameters of the decimal field determine the target type. In the order of NUMERIC, BIGNUMERIC, and STRING, a type is picked if it is in the specified list and if it supports the precision and the scale. STRING supports all precision and scale values. If none of the listed types supports the precision and the scale, the type supporting the widest range in the specified list is picked, and if a value exceeds the supported range when reading the data, an error will be thrown. Example: Suppose the value of this field is "NUMERIC", "BIGNUMERIC". If (precision,scale) is: (38,9) -> NUMERIC; (39,9) -> BIGNUMERIC (NUMERIC cannot hold 30 integer digits); (38,10) -> BIGNUMERIC (NUMERIC cannot hold 10 fractional digits); (76,38) -> BIGNUMERIC; (77,38) -> BIGNUMERIC (error if value exeeds supported range). This field cannot contain duplicate types. The order of the types in this field is ignored. For example, "BIGNUMERIC", "NUMERIC" is the same as "NUMERIC", "BIGNUMERIC" and NUMERIC always takes precedence over BIGNUMERIC. Defaults to "NUMERIC", "STRING" for ORC and "NUMERIC" for the other file formats.
fileSetSpecType Optional Specifies how source URIs are interpreted for constructing the file set to load. By default source URIs are expanded against the underlying storage. Other options include specifying manifest files. Only applicable to object storage systems.
googleSheetsOptions Optional Additional options if sourceFormat is set to GOOGLE_SHEETS.
hivePartitioningOptions Optional Options to configure hive partitioning support.
ignoreUnknownValues Optional Indicates if BigQuery should allow extra values that are not represented in the table schema. If true, the extra values are ignored. If false, records with extra columns are treated as bad records, and if there are too many bad records, an invalid error is returned in the job result. The default value is false. The sourceFormat property determines what BigQuery treats as an extra value: CSV: Trailing columns JSON: Named values that don't match any column names Google Cloud Bigtable: This setting is ignored. Google Cloud Datastore backups: This setting is ignored. Avro: This setting is ignored.
jsonOptions Additional properties to set if sourceFormat is set to NEWLINE_DELIMITED_JSON.
maxBadRecords Optional The maximum number of bad records that BigQuery can ignore when reading data. If the number of bad records exceeds this value, an invalid error is returned in the job result. This is only valid for CSV, JSON, and Google Sheets. The default value is 0, which requires that all records are valid. This setting is ignored for Google Cloud Bigtable, Google Cloud Datastore backups and Avro formats.
metadataCacheMode Optional Metadata Cache Mode for the table. Set this to enable caching of metadata from external data source.
objectMetadata ObjectMetadata is used to create Object Tables. Object Tables contain a listing of objects (with their metadata) found at the source_uris. If ObjectMetadata is set, source_format should be omitted. Currently SIMPLE is the only supported Object Metadata type.
parquetOptions Additional properties to set if sourceFormat is set to Parquet.
referenceFileSchemaUri Optional Provide a referencing file with the expected table schema. Enabled for the format: AVRO, PARQUET, ORC.
schema Optional The schema for the data. Schema is required for CSV and JSON formats. Schema is disallowed for Google Cloud Bigtable, Cloud Datastore backups, and Avro formats.
sourceFormat Required The data format. For CSV files, specify "CSV". For Google sheets, specify "GOOGLE_SHEETS". For newline-delimited JSON, specify "NEWLINE_DELIMITED_JSON". For Avro files, specify "AVRO". For Google Cloud Datastore backups, specify "DATASTORE_BACKUP". Beta For Google Cloud Bigtable, specify "BIGTABLE".
sourceUris Required The fully-qualified URIs that point to your data in Google Cloud. For Google Cloud Storage URIs: Each URI can contain one '' wildcard character and it must come after the 'bucket' name. Size limits related to load jobs apply to external data sources. For Google Cloud Bigtable URIs: Exactly one URI can be specified and it has be a fully specified and valid HTTPS URL for a Google Cloud Bigtable table. For Google Cloud Datastore backups, exactly one URI can be specified. Also, the '' wildcard character is not allowed.
const
-
ExternalDataConfiguration.fromMap(Map<String, dynamic> map)
-
factory
Properties
-
autodetect
→ Input<bool>?
-
Try to detect schema and format options automatically. Any option specified explicitly will be honored.
final
-
avroOptions
→ Input<AvroOptions>?
-
Additional properties to set if sourceFormat is set to Avro.
final
-
bigtableOptions
→ Input<BigtableOptions>?
-
Optional Additional options if sourceFormat is set to BIGTABLE.
final
-
compression
→ Input<String>?
-
Optional The compression type of the data source. Possible values include GZIP and NONE. The default value is NONE. This setting is ignored for Google Cloud Bigtable, Google Cloud Datastore backups and Avro formats.
final
-
connectionId
→ Input<String>?
-
Optional, Trusted Tester Connection for external data source.
final
-
csvOptions
→ Input<CsvOptions>?
-
Additional properties to set if sourceFormat is set to CSV.
final
-
decimalTargetTypes
→ Input<List<String>>?
-
Optional Defines the list of possible SQL data types to which the source decimal values are converted. This list and the precision and the scale parameters of the decimal field determine the target type. In the order of NUMERIC, BIGNUMERIC, and STRING, a type is picked if it is in the specified list and if it supports the precision and the scale. STRING supports all precision and scale values. If none of the listed types supports the precision and the scale, the type supporting the widest range in the specified list is picked, and if a value exceeds the supported range when reading the data, an error will be thrown. Example: Suppose the value of this field is "NUMERIC", "BIGNUMERIC". If (precision,scale) is: (38,9) -> NUMERIC; (39,9) -> BIGNUMERIC (NUMERIC cannot hold 30 integer digits); (38,10) -> BIGNUMERIC (NUMERIC cannot hold 10 fractional digits); (76,38) -> BIGNUMERIC; (77,38) -> BIGNUMERIC (error if value exeeds supported range). This field cannot contain duplicate types. The order of the types in this field is ignored. For example, "BIGNUMERIC", "NUMERIC" is the same as "NUMERIC", "BIGNUMERIC" and NUMERIC always takes precedence over BIGNUMERIC. Defaults to "NUMERIC", "STRING" for ORC and "NUMERIC" for the other file formats.
final
-
fileSetSpecType
→ Input<String>?
-
Optional Specifies how source URIs are interpreted for constructing the file set to load. By default source URIs are expanded against the underlying storage. Other options include specifying manifest files. Only applicable to object storage systems.
final
-
googleSheetsOptions
→ Input<GoogleSheetsOptions>?
-
Optional Additional options if sourceFormat is set to GOOGLE_SHEETS.
final
-
hashCode
→ int
-
The hash code for this object.
no setterinherited
-
hivePartitioningOptions
→ Input<HivePartitioningOptions>?
-
Optional Options to configure hive partitioning support.
final
-
ignoreUnknownValues
→ Input<bool>?
-
Optional Indicates if BigQuery should allow extra values that are not represented in the table schema. If true, the extra values are ignored. If false, records with extra columns are treated as bad records, and if there are too many bad records, an invalid error is returned in the job result. The default value is false. The sourceFormat property determines what BigQuery treats as an extra value: CSV: Trailing columns JSON: Named values that don't match any column names Google Cloud Bigtable: This setting is ignored. Google Cloud Datastore backups: This setting is ignored. Avro: This setting is ignored.
final
-
jsonOptions
→ Input<JsonOptions>?
-
Additional properties to set if
sourceFormat is set to NEWLINE_DELIMITED_JSON.
final
-
maxBadRecords
→ Input<int>?
-
Optional The maximum number of bad records that BigQuery can ignore when reading data. If the number of bad records exceeds this value, an invalid error is returned in the job result. This is only valid for CSV, JSON, and Google Sheets. The default value is 0, which requires that all records are valid. This setting is ignored for Google Cloud Bigtable, Google Cloud Datastore backups and Avro formats.
final
-
metadataCacheMode
→ Input<String>?
-
Optional Metadata Cache Mode for the table. Set this to enable caching of metadata from external data source.
final
-
objectMetadata
→ Input<String>?
-
ObjectMetadata is used to create Object Tables. Object Tables contain a listing of objects (with their metadata) found at the source_uris. If ObjectMetadata is set, source_format should be omitted. Currently SIMPLE is the only supported Object Metadata type.
final
-
parquetOptions
→ Input<ParquetOptions>?
-
Additional properties to set if sourceFormat is set to Parquet.
final
-
referenceFileSchemaUri
→ Input<String>?
-
Optional Provide a referencing file with the expected table schema. Enabled for the format: AVRO, PARQUET, ORC.
final
-
runtimeType
→ Type
-
A representation of the runtime type of the object.
no setterinherited
-
schema
→ Input<TableSchema>?
-
Optional The schema for the data. Schema is required for CSV and JSON formats. Schema is disallowed for Google Cloud Bigtable, Cloud Datastore backups, and Avro formats.
final
-
sourceFormat
→ Input<String>?
-
Required The data format. For CSV files, specify "CSV". For Google sheets, specify "GOOGLE_SHEETS". For newline-delimited JSON, specify "NEWLINE_DELIMITED_JSON". For Avro files, specify "AVRO". For Google Cloud Datastore backups, specify "DATASTORE_BACKUP". Beta For Google Cloud Bigtable, specify "BIGTABLE".
final
-
sourceUris
→ Input<List<String>>?
-
Required The fully-qualified URIs that point to your data in Google Cloud. For Google Cloud Storage URIs: Each URI can contain one '' wildcard character and it must come after the 'bucket' name. Size limits related to load jobs apply to external data sources. For Google Cloud Bigtable URIs: Exactly one URI can be specified and it has be a fully specified and valid HTTPS URL for a Google Cloud Bigtable table. For Google Cloud Datastore backups, exactly one URI can be specified. Also, the '' wildcard character is not allowed.
final
