Web service adapters and timeouts: An example

×

Status message

New Feature: Guest Login function added to facilitate site exploration without registering. Try it out!

Here's an interesting problem that can occur when working with a web service adapter running under Adama at Araport.

We recently had an adapter from our collaborators at the Meyers Lab stop working. We will use 'curl' commands within a Unix terminal to diagnose the problem.

Let's try running the web service with 'curl' in a bash shell. We'll use the '-v' option of curl to turn on verbose mode which will show us the headers sent to (lines starting with '>') and from (lines starting with '<') the web server:

    $ export API=https://api.araport.org/community/v0.3
    $ export WS="$API/meyerslab/srna_phased_loci_v0.1"
    $ export TOKEN=b779cc78ea5675a789184bb68d4155d2
    $ export BEARER="Authorization: Bearer $TOKEN"
    $ curl -vkL -X GET -H "$BEARER" "$WS/search?regulatory_mechanism=phasi"
    *   Trying 129.114.60.24...
    * Connected to api.araport.org (129.114.60.24) port 443 (#0)
    * TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
    * Server certificate: .araport.org
    * Server certificate: Entrust Certification Authority - L1K
    * Server certificate: Entrust Root Certification Authority - G2
    > GET /community/v0.3/meyerslab/srna_phased_loci_v0.1/search?regulatory_mechanism=phasi HTTP/1.1
    > Host: api.araport.org
    > User-Agent: curl/7.43.0
    > Accept: */
    > Authorization: Bearer b779cc78ea5675a789184bb68d4155d2
    >
    < HTTP/1.1 500 Internal Server Error
    < Date: Thu, 11 Feb 2016 18:36:27 GMT
    < Server: WSO2-PassThrough-HTTP
    < Link: https://api.araport.org/community/v0.3/debug/6020530ffb324efe8472ec099f4b1b1b; rel="debug"
    < Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept, Authorization, Content-Range, Range
    < Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS
    < Content-Type: application/json
    < Access-Control-Allow-Origin: *
    < Access-Control-Allow-Credentials: true
    < Connection: close
    < Transfer-Encoding: chunked
    <
    {
        "exception": "TimeoutException",
        "message": "result channel tcp://172.17.0.1:36686 has been idle for more than 30 seconds",
        "status": "error",
        "worker_trace": null
    }
    * Closing connection 0

Uh! Oh! What happened? We've received a TimeoutException from Adama trying to connect to the external service.

Let's first try and see if the underlying web service that the adapter is calling is actually working. Back to the terminal!

    $ curl -vkL -X GET "https://mpss.udel.edu/web/php/pages/loci_list.php?SITE=at_sRNA&format=json&regulatory_mechanism=phasi"
    *   Trying 128.175.253.57...
    * Connected to mpss.udel.edu (128.175.253.57) port 443 (#0)
    * TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
    * Server certificate: mpss.udel.edu
    * Server certificate: GlobalSign Organization Validation CA - SHA256 - G2
    * Server certificate: GlobalSign Root CA
    > GET /web/php/pages/loci_list.php?SITE=at_sRNA&format=json&regulatory_mechanism=phasi HTTP/1.1
    > Host: mpss.udel.edu
    > User-Agent: curl/7.43.0
    > Accept: /
    >
    < HTTP/1.1 301 Moved Permanently
    < Date: Thu, 11 Feb 2016 18:36:52 GMT
    < Server: Apache/2.2.15
    < Location: https://mpss.danforthcenter.org/web/php/pages/loci_list.php?SITE=at_sRNA&format=json&regulatory_mechanism=phasi
    < Content-Length: 398
    < Connection: close
    < Content-Type: text/html; charset=iso-8859-1
    <
    * Closing connection 0
    * Issue another request to this URL: 'https://mpss.danforthcenter.org/web/php/pages/loci_list.php?SITE=at_sRNA&format=json&regulatory_mechanism=phasi'
    *   Trying 65.254.111.64...
    * Connected to mpss.danforthcenter.org (65.254.111.64) port 443 (#1)
    * TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
    * Server certificate: mpss.danforthcenter.org
    * Server certificate: Go Daddy Secure Certificate Authority - G2
    * Server certificate: Go Daddy Root Certificate Authority - G2
    > GET /web/php/pages/loci_list.php?SITE=at_sRNA&format=json&regulatory_mechanism=phasi HTTP/1.1
    > Host: mpss.danforthcenter.org
    > User-Agent: curl/7.43.0
    > Accept: /
    >
    < HTTP/1.1 200 OK
    < Date: Thu, 11 Feb 2016 18:40:12 GMT
    < Server: Apache/2.2.15
    < X-Powered-By: PHP/5.3.3
    < Set-Cookie: PHPSESSID=249vl1t8oskq3qu1tjgd2c1ob3; path=/
    < Expires: Thu, 19 Nov 1981 08:52:00 GMT
    < Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
    < Pragma: no-cache
    < Access-Control-Allow-Origin: *
    < Connection: close
    < Transfer-Encoding: chunked
    < Content-Type: application/json
    <
    {"phasiRNAs":[
        {"chromosome":2, "start":11721478, "end":11722200, "phase_length":21, "score":33.18304, "min_pvalue":0, "locus_id":"AT2G27400",
        "title":"TAS1a", "link":"https://mpss.danforthcenter.org/web/php/pages/GeneAnalysis.php?SITE=at_sRNA&chrnum=2&beg=11721478&end=11722200&phase_len=21",
        "image":"https://mpss.danforthcenter.org/web/php/helpers/showImg.php?file=/var/www/secured/tmpmap2/araport/at_sRNA.2.11720978-11722700.all_1.150.1._PA.png"},
        ...only showing first line of output...

OK. That worked, but let's take a closer look at the output from 'curl'. A connection is made to the web server at mpss.udel.edu, but we receive a 301 HTTP status code (Moved permanently) back from the web server along with a new location, https://mpss.danforthcenter.org/web/php/pages/loci_list.php?SITE=at_sRNA.... Another connection is then made to the web server at mpss.danforthcenter.org and we receive a 200 HTTP status code (Success) and the correct output in JSON format.

Why is this a problem when running under Adama? Well, as they say, this is a "feature, not a bug"! For security, web services running under Adama are behind a firewall and any external connections must be 'whitelisted' (allowed explicitly). The whitelist for an adapter is specified in the metadata.yml file (For details on Adama adapters see Building Community APIs with ADAMA). Let's take a look at a snippet of the current metadata file:

    whitelist:
        - mpss.udel.edu

Ah! Hah! There is the problem. Adama is allowing the initial connection to mpss.udel.edu, but blocking the subsequent connection to mpss.danforthcenter.org caused by the redirect. Hence, the timeout! The simpliest fix in this case would be to just add mpss.danforthcenter.org to the whitelist and redeploy the adapter. Since the adapter has to be updated anyway, a better (and SAFER) fix is to modify the whitelist and update the URL to the underlying web service in the code. This avoids the potential future problem of the redirect going away.

Note that this issue is also common at the initial stages of developing a web service when going from testing locally in Python to the first deployment under Adama. So make sure that the 'whitelist' variable in metadata.yml is set properly before deploying to Adama! You've been warned!