Querying Elasticsearch from Google App Engine

Elasticsearch's documentations often gives examples where read-only actions are performed with an HTTP body payload and sent with the HTTP GET verb. For example, the following is from Elasticsearch's Search APIs reference:

$ curl -XGET 'http://localhost:9200/twitter/tweet/_search?routing=kimchy' -d '{
    "query": {
        "filtered" : {
            "query" : {
                "query_string" : {
                    "query" : "some query string here"
                }
            },
            "filter" : {
                "term" : { "user" : "kimchy" }
            }
        }
    }
}
'

With Google App Engine, any HTTP call made with the HTTP GET verb will have the body payload stripped. The call will need to be sent with the HTTP POST verb.

import json

import requests


url = 'http://123.456.789.012:9200/twitter/tweet/_search?routing=kimchy'
headers = {'content-type': 'application/json'}
payload = {
    "query": {
        "filtered" : {
            "query" : {
                "query_string" : {
                    "query" : "some query string here"
                }
            },
            "filter" : {
                "term" : { "user" : "kimchy" }
            }
        }
    }
}

resp = requests.post(url, data=json.dumps(payload), headers=headers)

But when Elasticsearch receives the search query with the HTTP POST verb it will think it's a call to create a record rather than perform a search. The use of _search in the endpoint URL will further confuse matters.

The solution I found was to run nginx on the Google Compute Engine Instance running Elasticsearch and convert any HTTP POST calls to HTTP GET in the nginx proxy configuration:

server {
    listen 80;

    ... various proxy settings ...

    location ~* / {
        ... http basic auth settings ...

        set $allow_method  0;

        if ($request_method = 'GET') {
          set $allow_method  1;
        }

        if ($request_method = 'POST') {
          set $allow_method  1;
        }

        if ($allow_method = 0) {
          # HTTP Error 405: Method not allowed
          # http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.6
          return 405;
          break;
        }

        proxy_pass http://elasticsearch/$uri$is_args$args;
        proxy_redirect off;

        # This will change all acceptable calls to HTTP GET
        # regardless of their original verb usage:
        proxy_method GET;
    }
}

Now you can send search queries with a body payload using HTTP POST and it will be converted by nginx before being handed off to Elasticsearch.

Thank you for taking the time to read this post. I offer both consulting and hands-on development services to clients in North America and Europe. If you'd like to discuss how my offerings can help your business please contact me via LinkedIn.