Service Catalog Version 0.96.9Last updated in version 0.95.1

Amazon Elasticsearch Service

View SourceRelease Notes

Overview

This service contains code to deploy an Amazon Elasticsearch Service cluster. See the Amazon Elasticsearch Service documentation and the Getting Started page for more information.

Features

• A fully-managed native Elasticsearch cluster in a VPC
• A fully functional Kibana UI
• VPC-based security
• Zone awareness, i.e., deployment of Elasticsearch nodes across Availability Zones
• Automatic nightly snapshots
• CloudWatch Alarms for alerting when CPU, memory, and disk metrics exceed certain thresholds

Learn

note

This repo is a part of the Gruntwork Service Catalog, a collection of reusable, battle-tested, production ready infrastructure code. If you’ve never used the Service Catalog before, make sure to read How to use the Gruntwork Service Catalog!

• About Amazon Elasticsearch Service
• Features of Amazon ES
• Developer Guide: Contains the main documentation on how to use Amazon ES and answers questions such as "What is Amazon Elasticsearch Service?"
• Streaming CloudWatch monitoring logs to Amazon Elasticsearch Service

Deploy

Non-production deployment (quick start for learning)

If you just want to try this repo out for experimenting and learning, check out the following resources:

• examples/for-learning-and-testing folder: The examples/for-learning-and-testing folder contains standalone sample code optimized for learning, experimenting, and testing (but not direct production usage).

• AWS Free tier: Using Amazon ES on Amazon’s free tier is a great way to get started, but it has limited features and does not include encryption at rest, ultra warm data notes, or advanced security options such as fine-grained access control. The free tier does allow multiple availability zones, VPC-based access control, TLS-only requests, and node-to-node encryption.

Production deployment

If you want to deploy this repo in production, check out the following resources:

• examples/for-production folder: The examples/for-production folder contains sample code optimized for direct usage in production. This is code from the Gruntwork Reference Architecture, and it shows you how we build an end-to-end, integrated tech stack on top of the Gruntwork Service Catalog.

• Amazon Elasticsearch Service pricing

Reference

Inputs

Required

domain_namestringrequired

The name of the Elasticsearch cluster. It must be unique to your account and region, start with a lowercase letter, contain between 3 and 28 characters, and contain only lowercase letters a-z, the numbers 0-9, and the hyphen (-).

instance_countnumberrequired

The number of instances to deploy in the Elasticsearch cluster. This must be an even number if zone_awareness_enabled is true.

instance_typestringrequired

The instance type to use for Elasticsearch data nodes (e.g., t2.small.elasticsearch, or m4.large.elasticsearch). For supported instance types see https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-supported-instance-types.html.

volume_sizenumberrequired

The size in GiB of the EBS volume for each node in the cluster (e.g. 10, or 512). For volume size limits see https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-limits.html.

volume_typestringrequired

The type of EBS volumes to use in the cluster. Must be one of: standard, gp2, io1, sc1, or st1. For a comparison of EBS volume types, see https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/ebs-volume-types.html.

zone_awareness_enabledboolrequired

Whether to deploy the Elasticsearch nodes across two Availability Zones instead of one. Note that if you enable this, the instance_count MUST be an even number.

Optional

advanced_optionsmap(any)optional

Key-value string pairs to specify advanced configuration options. Note that the values for these configuration options must be strings (wrapped in quotes).

Type Details

Any types represent complex values of variable type. For details, please consult `variables.tf` in the source repo.

Default:{}

advanced_security_optionsbooloptional

Enable fine grain access control

Default:false

alarm_sns_topic_arnslist(string)optional

ARNs of the SNS topics associated with the CloudWatch alarms for the Elasticsearch cluster.

Default:[]

allow_connections_from_cidr_blocksset(string)optional

The list of network CIDR blocks to allow network access to Aurora from. One of allow_connections_from_cidr_blocks or allow_connections_from_security_groups must be specified for the database to be reachable.

Default:[]

allow_connections_from_security_groupsset(string)optional

The list of IDs or Security Groups to allow network access to Aurora from. All security groups must either be in the VPC specified by vpc_id, or a peered VPC with the VPC specified by vpc_id. One of allow_connections_from_cidr_blocks or allow_connections_from_security_groups must be specified for the database to be reachable.

Default:[]

automated_snapshot_start_hournumberoptional

Hour during which the service takes an automated daily snapshot of the indices in the domain. This setting has no effect on Elasticsearch 5.3 and later.

Default:0

availability_zone_countnumberoptional

Number of Availability Zones for the domain to use with zone_awareness_enabled. Defaults to 2. Valid values: 2 or 3.

Default:2

cluster_high_cpu_utilization_periodnumberoptional

The period, in seconds, over which to measure the CPU utilization percentage

Default:60

cluster_high_cpu_utilization_thresholdnumberoptional

Trigger an alarm if the Elasticsearch cluster has a CPU utilization percentage above this threshold

Default:90

cluster_high_cpu_utilization_treat_missing_datastringoptional

Sets how this alarm should handle entering the INSUFFICIENT_DATA state. Based on https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html#alarms-and-missing-data. Must be one of: 'missing', 'ignore', 'breaching' or 'notBreaching'.

Default:"missing"

cluster_high_jvm_memory_pressure_periodnumberoptional

The period, in seconds, over which to measure the JVM heap usage percentage

Default:60

cluster_high_jvm_memory_pressure_thresholdnumberoptional

Trigger an alarm if the JVM heap usage percentage goes above this threshold

Default:90

cluster_high_jvm_memory_pressure_treat_missing_datastringoptional

Sets how this alarm should handle entering the INSUFFICIENT_DATA state. Based on https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html#alarms-and-missing-data. Must be one of: 'missing', 'ignore', 'breaching' or 'notBreaching'.

Default:"missing"

cluster_index_writes_blocked_periodnumberoptional

The maximum amount of time, in seconds, that ClusterIndexWritesBlocked can be in red status before triggering an alarm

Default:300

cluster_index_writes_blocked_treat_missing_datastringoptional

Sets how this alarm should handle entering the INSUFFICIENT_DATA state. Based on https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html#alarms-and-missing-data. Must be one of: 'missing', 'ignore', 'breaching' or 'notBreaching'.

Default:"missing"

cluster_low_cpu_credit_balance_periodnumberoptional

The period, in seconds, over which to measure the CPU credit balance

Default:60

cluster_low_cpu_credit_balance_thresholdnumberoptional

Trigger an alarm if the CPU credit balance drops below this threshold. Only used if instance_type is t2.xxx.

Default:10

cluster_low_cpu_credit_balance_treat_missing_datastringoptional

Sets how this alarm should handle entering the INSUFFICIENT_DATA state. Based on https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html#alarms-and-missing-data. Must be one of: 'missing', 'ignore', 'breaching' or 'notBreaching'.

Default:"missing"

cluster_low_free_storage_space_periodnumberoptional

The period, in seconds, over which to measure the available free storage space

Default:60

cluster_low_free_storage_space_thresholdnumberoptional

Trigger an alarm if the amount of free storage space, in Megabytes, on the Elasticsearch cluster drops below this threshold

Default:1024

cluster_low_free_storage_space_treat_missing_datastringoptional

Sets how this alarm should handle entering the INSUFFICIENT_DATA state. Based on https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html#alarms-and-missing-data. Must be one of: 'missing', 'ignore', 'breaching' or 'notBreaching'.

Default:"missing"

cluster_snapshot_periodnumberoptional

The maximum amount of time, in seconds, during with the AutomatedSnapshotFailure can be in red status before triggering an alarm

Default:60

cluster_snapshot_treat_missing_datastringoptional

Sets how this alarm should handle entering the INSUFFICIENT_DATA state. Based on https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html#alarms-and-missing-data. Must be one of: 'missing', 'ignore', 'breaching' or 'notBreaching'.

Default:"missing"

cluster_status_red_periodnumberoptional

The maximum amount of time, in seconds, during which the cluster can be in red status before triggering an alarm

Default:300

cluster_status_red_treat_missing_datastringoptional

Sets how this alarm should handle entering the INSUFFICIENT_DATA state. Based on https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html#alarms-and-missing-data. Must be one of: 'missing', 'ignore', 'breaching' or 'notBreaching'.

Default:"missing"

cluster_status_yellow_periodnumberoptional

The maximum amount of time, in seconds, during which the cluster can be in yellow status before triggering an alarm

Default:300

cluster_status_yellow_treat_missing_datastringoptional

Sets how this alarm should handle entering the INSUFFICIENT_DATA state. Based on https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html#alarms-and-missing-data. Must be one of: 'missing', 'ignore', 'breaching' or 'notBreaching'.

Default:"missing"

create_service_linked_rolebooloptional

Whether or not the Service Linked Role for Elasticsearch should be created within this module. Normally the service linked role is created automatically by AWS when creating the Elasticsearch domain in the web console, but API does not implement this logic. You can either have AWS automatically manage this by creating a domain manually in the console, or manage it in terraform using the landing zone modules or this variable.

Default:false

custom_endpointstringoptional

Fully qualified domain for your custom endpoint.

Default:null

custom_endpoint_certificate_arnstringoptional

ACM certificate ARN for your custom endpoint.

Default:null

custom_endpoint_enabledbooloptional

Whether to enable custom endpoint for the Elasticsearch domain.

Default:false

custom_tagsmap(string)optional

A map of custom tags to apply to the ElasticSearch Domain. The key is the tag name and the value is the tag value.

Default:{}

dedicated_master_countnumberoptional

The number of dedicated master nodes to run. We recommend setting this to 3 for production deployments. Only used if dedicated_master_enabled is true.

Default:null

dedicated_master_enabledbooloptional

Whether to deploy separate nodes specifically for performing cluster management tasks (e.g. tracking number of nodes, monitoring health, replicating changes). This increases the stability of large clusters and is required for clusters with more than 10 nodes.

Default:false

dedicated_master_typestringoptional

The instance type for the dedicated master nodes. These nodes can use a different instance type than the rest of the cluster. Only used if dedicated_master_enabled is true.

Default:null

ebs_enabledbooloptional

Set to false to disable EBS volumes. This is useful for nodes that have optimized instance storage, like hosts running the i3 instance type.

Default:true

elasticsearch_versionstringoptional

The version of Elasticsearch to deploy.

Default:"7.7"

enable_cloudwatch_alarmsbooloptional

Set to true to enable several basic CloudWatch alarms around CPU usage, memory usage, and disk space usage. If set to true, make sure to specify SNS topics to send notifications to using alarms_sns_topic_arns.

Default:true

enable_encryption_at_restbooloptional

False by default because encryption at rest is not included in the free tier. When true, the Elasticsearch domain storage will be encrypted at rest using the KMS key described with encryption_kms_key_id. We strongly recommend configuring a custom KMS key instead of using the shared service key for a better security posture when configuring encryption at rest.

Default:true

enable_node_to_node_encryptionbooloptional

Whether to enable node-to-node encryption.

Default:true

encryption_kms_key_idstringoptional

The ID of the KMS key to use to encrypt the Elasticsearch domain storage. Only used if enable_encryption_at_rest. When null, uses the aws/es service KMS key.

Default:null

iam_principal_arnslist(string)optional

The ARNS of the IAM users and roles to which to allow full access to the Elasticsearch cluster. Setting this to a restricted list is useful when using a public access cluster.

Default:[ "*" ]

internal_user_database_enabledbooloptional

Whether the internal user database is enabled. Enable this to use master accounts. Only used if advanced_security_options is set to true.

Default:false

iopsnumberoptional

The baseline input/output (I/O) performance of EBS volumes attached to data nodes. Must be between 1000 and 4000. Applicable only if volume_type is io1.

Default:null

is_publicbooloptional

Whether the cluster is publicly accessible.

Default:false

kms_key_error_periodnumberoptional

The maximum amount of time, in seconds, that KMSKeyError can be in red status before triggering an alarm

Default:60

kms_key_error_treat_missing_datastringoptional

Sets how this alarm should handle entering the INSUFFICIENT_DATA state. Based on https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html#alarms-and-missing-data. Must be one of: 'missing', 'ignore', 'breaching' or 'notBreaching'.

Default:"missing"

kms_key_inaccessible_periodnumberoptional

The maximum amount of time, in seconds, that KMSKeyInaccessible can be in red status before triggering an alarm

Default:60

kms_key_inaccessible_treat_missing_datastringoptional

Sets how this alarm should handle entering the INSUFFICIENT_DATA state. Based on https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html#alarms-and-missing-data. Must be one of: 'missing', 'ignore', 'breaching' or 'notBreaching'.

Default:"missing"

master_cpu_utilization_periodnumberoptional

The period, in seconds, over which to measure the master nodes' CPU utilization

Default:900

master_cpu_utilization_thresholdnumberoptional

Trigger an alarm if the Elasticsearch cluster master nodes have a CPU utilization percentage above this threshold

Default:50

master_cpu_utilization_treat_missing_datastringoptional

Sets how this alarm should handle entering the INSUFFICIENT_DATA state. Based on https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html#alarms-and-missing-data. Must be one of: 'missing', 'ignore', 'breaching' or 'notBreaching'.

Default:"missing"

master_jvm_memory_pressure_periodnumberoptional

The period, in seconds, over which to measure the master nodes' JVM memory pressure

Default:900

master_jvm_memory_pressure_thresholdnumberoptional

Trigger an alarm if the Elasticsearch cluster master nodes have a JVM memory pressure percentage above this threshold

Default:80

master_jvm_memory_pressure_treat_missing_datastringoptional

Sets how this alarm should handle entering the INSUFFICIENT_DATA state. Based on https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html#alarms-and-missing-data. Must be one of: 'missing', 'ignore', 'breaching' or 'notBreaching'.

Default:"missing"

master_user_arnstringoptional

ARN of the master user. Only used if advanced_security_options and internal_user_database_enabled are set to true.

Default:null

master_user_namestringoptional

Master account user name. Only used if advanced_security_options and internal_user_database_enabled are set to true.

Default:null

master_user_passwordstringoptional

Master account user password. Only used if advanced_security_options and internal_user_database_enabled are set to true. WARNING: this password will be stored in Terraform state.

Default:null

monitor_kms_keybooloptional

Whether to monitor KMS key statistics

Default:false

monitor_master_nodesbooloptional

Whether to monitor master node statistics

Default:false

node_count_periodnumberoptional

The period, in seconds, over which to measure the master nodes' CPU utilization

Default:86400

node_count_treat_missing_datastringoptional

Sets how this alarm should handle entering the INSUFFICIENT_DATA state. Based on https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html#alarms-and-missing-data. Must be one of: 'missing', 'ignore', 'breaching' or 'notBreaching'.

Default:"missing"

subnet_idslist(string)optional

List of VPC Subnet IDs for the Elasticsearch domain endpoints to be created in. If zone_awareness_enabled is true, the first 2 or 3 provided subnet ids are used, depending on availability_zone_count. Otherwise only the first one is used.

Default:[]

tls_security_policystringoptional

The name of the TLS security policy that needs to be applied to the HTTPS endpoint. Valid values are Policy-Min-TLS-1-0-2019-07 and Policy-Min-TLS-1-2-2019-07. Terraform performs drift detection if this is configured.

Default:"Policy-Min-TLS-1-2-2019-07"

update_timeoutstringoptional

How long to wait for updates to the ES cluster before timing out and reporting an error.

Default:"90m"

vpc_idstringoptional

The id of the VPC to deploy into. It must be in the same region as the Elasticsearch domain and its tenancy must be set to Default. If zone_awareness_enabled is false, the Elasticsearch cluster will have an endpoint in one subnet of the VPC; otherwise it will have endpoints in two subnets.

Default:null

Outputs

cluster_arn

The ARN of the Elasticsearch cluster created by this module.

cluster_domain_id

The domain ID of the Elasticsearch cluster created by this module.

cluster_domain_name

The name of the Elasticsearch domain.

cluster_endpoint

The endpoint of the Elasticsearch cluster created by this module.

cluster_security_group_id

If the domain was created inside a VPC, the ID of the security group created by this module for securing the Elasticsearch cluster.

kibana_endpoint

Domain-specific endpoint for Kibana without https scheme.