使用 EQL 检索 ElasticSearch 数据

使用 EQL 检索 ElasticSearch 数据

常用查询语法

• 查询两列相等的记录拉流数据

GET dev-vloud-collection.stats-2021.12.17/_search
{
    "query": {
        "bool": {
            "must": [
              {
                "script": {
                    "script": "doc['user.keyword'] == doc['remote.keyword']"
                }
            },
            {
                "script": {
                    "script": "doc['flow.keyword'].value == 'pull'"
                }
            }
            ]
        }
    }
}

• 查询包涵某字段的方法

POST /prod-vloud-collection.stats-2022.05.30/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "exists": {
            "field": "stats.extend.eglrender"
          }
        }
      ]
    }
  }
}

• 查询索引数据

GET /prod-vloud-collection.stats-2022.06.02/_search?pretty
{
    "query": {
        "match_all": {}
    }
}

• 手动修改已建立的索引模式不生效

POST /prod-vloud-collection.stats-2022.06.02/_mapping
{
  "properties": {
      "stats": {
        "properties": {
          "audioLevel": {
            "type": "float"
          }
        }
      }
    }
}

报如下错

{
  "error": {
    "root_cause": [
      {
        "type": "remote_transport_exception",
        "reason": "[jd-bj-bjy-elasticsearch009][10.1.81.96:9300][indices:admin/mapping/put]"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "mapper [stats.audioLevel] cannot be changed from type [long] to [float]"
  },
  "status": 400
}

可以改 template

{
  "properties": {
      "stats": {
        "properties": {
          "audioLevel": {
            "type": "float"
          }
        }
      }
    }
}

改模板只对新的数据上报有影响，旧的

POST /_reindex
{
  "source": {
    "index": "prod-vloud-collection.stats-2022.06.02"
  },
  "dest": {
    "index": "prod-vloud-collection.stats-v1-2022.06.02"
  }
}

DELETE /prod-vloud-collection.stats-2022.06.02

如果数据量大 reindex 操作可能超时，可以通过以下命令查任务进度

GET _tasks?actions=*reindex&detailed

{
  "nodes" : {
    "n58-kJiETXOmWtZ49uWI4Q" : {
      "name" : "jd-bj-bjy-elasticsearch008",
      "transport_address" : "10.1.80.42:9300",
      "host" : "10.1.80.42",
      "ip" : "10.1.80.42:9300",
      "roles" : [
        "ingest",
        "master",
        "data"
      ],
      "attributes" : {
        "xpack.installed" : "true"
      },
      "tasks" : {
        "n58-kJiETXOmWtZ49uWI4Q:494517246" : {
          "node" : "n58-kJiETXOmWtZ49uWI4Q",
          "id" : 494517246,
          "type" : "transport",
          "action" : "indices:data/write/reindex",
          "status" : {
            "total" : 5891483,
            "updated" : 0,
            "created" : 1850000,
            "deleted" : 0,
            "batches" : 1851,
            "version_conflicts" : 0,
            "noops" : 0,
            "retries" : {
              "bulk" : 0,
              "search" : 0
            },
            "throttled_millis" : 0,
            "requests_per_second" : -1.0,
            "throttled_until_millis" : 0
          },
          "description" : "reindex from [prod-vloud-collection.stats-2022.06.02] to [prod-vloud-collection.stats-v1-2022.06.02][_doc]",
          "start_time_in_millis" : 1654140713940,
          "running_time_in_nanos" : 521942923227,
          "cancellable" : true,
          "headers" : { }
        }
      }
    }
  }
}

如果任务执行超时可以取消任务

POST _tasks/n58-kJiETXOmWtZ49uWI4Q:494517246/_cancel

集群管理

这可以关闭 xpack 监控，如果 kibana 进不去可以进 es 中任意一台服务器敲个命令。

GET _cluster/settings
PUT _cluster/settings
{
  "persistent": {
    "xpack.monitoring.collection.enabled": true
  }
}

curl -XGET -u elastic:4321bjsl  -H 'Content-Type: application/json' 'http://10.16.40.179:9200/_cluster/settings'
curl -XPUT -u elastic:4321bjsl  -H 'Content-Type: application/json' 'http://10.16.40.179:9200/_cluster/settings' -d '{"persistent": {"xpack.monitoring.collection.enabled": false}}'

curl -XGET -u elastic:4321bjsl  -H 'Content-Type: application/json' 'http://10.16.40.179:9200/_cluster/settings'