Skip to main content
Version: Next

Couchbase

Incubating

Important Capabilities​

CapabilityStatusNotes
Asset Containers✅Enabled by default
Classification✅Optionally enabled via classification.enabled
Data Profiling✅Optionally enabled via configuration profiling.enabled
Detect Deleted Entities✅Optionally enabled via stateful_ingestion.remove_stale_metadata
Domains✅Supported via the domain config field
Platform Instance✅The platform_instance is derived from the configured cluster name
Schema Metadata✅Enabled by default

This plugin extracts the following:

  • Buckets (databases), scopes and collections
  • Schemas for each collection (via schema inference)

The plugin will sample 10,000 documents by default. Use the setting schema_sample_size to define how many documents will be sampled per collection.

CLI based Ingestion​

Starter Recipe​

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:
  type: couchbase
  config:
    connect_string: "couchbases://127.0.0.1"
    username: "Administrator"
    password: "password"
    cluster_name: "testdb"
    classification:
      enabled: true
      classifiers:
        - type: datahub
    profiling:
      enabled: true

sink:
  # sink configs

Config Details​

Note that a . is used to denote nested fields in the YAML recipe.

FieldDescription
cluster_name
string
Couchbase cluster name.
Default: cbdb
connect_string
string
Couchbase connect string.
Default: couchbases://127.0.0.1
kv_timeout
integer
KV timeout.
Default: 5
maxSchemaSize
integer
Maximum number of fields to include in the schema.
Default: 300
options
object
Additional options to pass to ClusterOptions().
Default: {}
password
string
Couchbase password.
Default: password
platform_instance
string
The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://datahubproject.io/docs/platform-instances/ for more details.
query_timeout
integer
Query timeout.
Default: 75
schema_sample_size
integer
Number of documents to sample.
Default: 10000
username
string
Couchbase username.
Default: Administrator
env
string
The environment that all assets produced by this connector belong to
Default: PROD
domain
map(str,AllowDenyPattern)
A class to store allow deny regexes
domain.key.allow
array
List of regex patterns to include in ingestion
Default: ['.*']
domain.key.allow.string
string
domain.key.ignoreCase
boolean
Whether to ignore case sensitivity during pattern matching.
Default: True
domain.key.deny
array
List of regex patterns to exclude from ingestion.
Default: []
domain.key.deny.string
string
keyspace_pattern
AllowDenyPattern
regex patterns for keyspace to filter in ingestion.
Default: {'allow': ['.*'], 'deny': [], 'ignoreCase': True}
keyspace_pattern.ignoreCase
boolean
Whether to ignore case sensitivity during pattern matching.
Default: True
keyspace_pattern.allow
array
List of regex patterns to include in ingestion
Default: ['.*']
keyspace_pattern.allow.string
string
keyspace_pattern.deny
array
List of regex patterns to exclude from ingestion.
Default: []
keyspace_pattern.deny.string
string
profile_pattern
AllowDenyPattern
Regex patterns for tables to profile
Default: {'allow': ['.*'], 'deny': [], 'ignoreCase': True}
profile_pattern.ignoreCase
boolean
Whether to ignore case sensitivity during pattern matching.
Default: True
profile_pattern.allow
array
List of regex patterns to include in ingestion
Default: ['.*']
profile_pattern.allow.string
string
profile_pattern.deny
array
List of regex patterns to exclude from ingestion.
Default: []
profile_pattern.deny.string
string
classification
ClassificationConfig
For details, refer to Classification.
Default: {'enabled': False, 'sample_size': 100, 'max_worker...
classification.enabled
boolean
Whether classification should be used to auto-detect glossary terms
Default: False
classification.info_type_to_term
map(str,string)
classification.max_workers
integer
Number of worker processes to use for classification. Note that any number above 1 might lead to a deadlock. Set to 1 to disable.
Default: 1
classification.sample_size
integer
Number of sample values used for classification.
Default: 100
classification.classifiers
array
Classifiers to use to auto-detect glossary terms. If more than one classifier, infotype predictions from the classifier defined later in sequence take precedance.
Default: [{'type': 'datahub', 'config': None}]
classification.classifiers.DynamicTypedClassifierConfig
DynamicTypedClassifierConfig
classification.classifiers.DynamicTypedClassifierConfig.type ❓
string
The type of the classifier to use. For DataHub, use datahub
classification.classifiers.DynamicTypedClassifierConfig.config
object
The configuration required for initializing the classifier. If not specified, uses defaults for classifer type.
classification.column_pattern
AllowDenyPattern
Regex patterns to filter columns for classification. This is used in combination with other patterns in parent config. Specify regex to match the column name in database.schema.table.column format.
Default: {'allow': ['.*'], 'deny': [], 'ignoreCase': True}
classification.column_pattern.ignoreCase
boolean
Whether to ignore case sensitivity during pattern matching.
Default: True
classification.column_pattern.allow
array
List of regex patterns to include in ingestion
Default: ['.*']
classification.column_pattern.allow.string
string
classification.column_pattern.deny
array
List of regex patterns to exclude from ingestion.
Default: []
classification.column_pattern.deny.string
string
classification.table_pattern
AllowDenyPattern
Regex patterns to filter tables for classification. This is used in combination with other patterns in parent config. Specify regex to match the entire table name in database.schema.table format. e.g. to match all tables starting with customer in Customer database and public schema, use the regex 'Customer.public.customer.*'
Default: {'allow': ['.*'], 'deny': [], 'ignoreCase': True}
classification.table_pattern.ignoreCase
boolean
Whether to ignore case sensitivity during pattern matching.
Default: True
classification.table_pattern.allow
array
List of regex patterns to include in ingestion
Default: ['.*']
classification.table_pattern.allow.string
string
classification.table_pattern.deny
array
List of regex patterns to exclude from ingestion.
Default: []
classification.table_pattern.deny.string
string
profiling
GEProfilingBaseConfig
Configuration for profiling
Default: {'enabled': False, 'operation_config': {'lower_fre...
profiling.enabled
boolean
Whether profiling should be done.
Default: False
profiling.include_field_distinct_count
boolean
Whether to profile for the number of distinct values for each column.
Default: True
profiling.include_field_distinct_value_frequencies
boolean
Whether to profile for distinct value frequencies.
Default: False
profiling.include_field_histogram
boolean
Whether to profile for the histogram for numeric fields.
Default: False
profiling.include_field_max_value
boolean
Whether to profile for the max value of numeric columns.
Default: True
profiling.include_field_mean_value
boolean
Whether to profile for the mean value of numeric columns.
Default: True
profiling.include_field_median_value
boolean
Whether to profile for the median value of numeric columns.
Default: True
profiling.include_field_min_value
boolean
Whether to profile for the min value of numeric columns.
Default: True
profiling.include_field_null_count
boolean
Whether to profile for the number of nulls for each column.
Default: True
profiling.include_field_quantiles
boolean
Whether to profile for the quantiles of numeric columns.
Default: False
profiling.include_field_sample_values
boolean
Whether to profile for the sample values for all columns.
Default: True
profiling.include_field_stddev_value
boolean
Whether to profile for the standard deviation of numeric columns.
Default: True
profiling.limit
integer
Max number of documents to profile. By default, profiles all documents.
profiling.max_workers
integer
Number of worker threads to use for profiling. Set to 1 to disable.
Default: 20
profiling.offset
integer
Offset in documents to profile. By default, uses no offset.
profiling.profile_table_level_only
boolean
Whether to perform profiling at table-level only, or include column-level profiling as well.
Default: False
profiling.operation_config
OperationConfig
Experimental feature. To specify operation configs.
profiling.operation_config.lower_freq_profile_enabled
boolean
Whether to do profiling at lower freq or not. This does not do any scheduling just adds additional checks to when not to run profiling.
Default: False
profiling.operation_config.profile_date_of_month
integer
Number between 1 to 31 for date of month (both inclusive). If not specified, defaults to Nothing and this field does not take affect.
profiling.operation_config.profile_day_of_week
integer
Number between 0 to 6 for day of week (both inclusive). 0 is Monday and 6 is Sunday. If not specified, defaults to Nothing and this field does not take affect.
stateful_ingestion
StatefulIngestionConfig
Stateful Ingestion Config
stateful_ingestion.enabled
boolean
Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or datahub_api is specified, otherwise False
Default: False

Code Coordinates​

  • Class Name: datahub.ingestion.source.couchbase.couchbase_main.CouchbaseDBSource
  • Browse on GitHub

Questions

If you've got any questions on configuring ingestion for Couchbase, feel free to ping us on our Slack.