GA4GH API Demo

In this demo, we’ll install a copy of the GA4GH reference implementation and run a local version of the server using some example data. We then run some example queries on this server using various different methods to illustrate the basics of the protocol. The server can, of course, be run on any machine on the network, but for simplicity we assume that the client and the server are running on your local machine during this demo.

The instructions for installation here are not intended to be used in a production deployment. See the Installation section for a detailed guide on production installation. To run the demo, you will need a working installation of Python 2.7 and also have virtualenv installed. We also need to have zlib and a few other common libraries installed so that we can build some of the packages that the reference server depends on.

On Debian/Ubuntu, for example, we can install these packages using:

$ sudo apt-get install python-dev python-virtualenv zlib1g-dev libxslt1-dev libffi-dev libssl-dev

On Fedora 22+ (current), the equivalent would be:

$ sudo dnf install python-devel python-virtualenv zlib-devel libxslt-devel openssl-devel

First, we create a virtualenv sandbox to isolate the demo from the rest of the system, and then activate it:

$ virtualenv ga4gh-env
$ source ga4gh-env/bin/activate

Now, install the ga4gh package from the Python package index. This will take some time, as some upstream packages will need to be built and installed.

(ga4gh-env) $ pip install ga4gh-server --pre

(Older versions of pip might not recognise the --pre argument; if not, it is safe to remove it.)

Now we can download some example data, which we’ll use for our demo:

(ga4gh-env) $ wget https://github.com/ga4gh/ga4gh-server/releases/download/data/ga4gh-example-data_4.6.tar
(ga4gh-env) $ tar -xvf ga4gh-example-data_4.6.tar

After extracting the data, we can then run the ga4gh_server application:

(ga4gh-env) $ ga4gh_server
* Running on http://127.0.0.1:8000/ (Press CTRL+C to quit)
* Restarting with stat

(The server is using a default configuration which assumes the existence of the ga4gh-example-data directory for simplicity here; see the Configuration section for detailed information on how we configure the server.) We now have a server running in the foreground. When it receives requests, it will print out log entries to the terminal. A summary of the server’s configuration and data is available in HTML format at http://locahost:8000, which can be viewed in a web browser. Leave the server running and open another terminal to complete the rest of the demo.

To try out the server, we must send some requests to it using the GA4GH protocol. One way in which we can do this is to manually create the JSON requests, and send these to the server using cURL:

$ curl --data '{}' --header 'Content-Type: application/json' \
http://localhost:8000/datasets/search | jq .

In this example, we used the search_datasets method to ask the server for all the Datasets on the server. It responded by sending back some JSON, which we piped into the jq JSON processor to make it easier to read. We get the following result:

{
  "nextPageToken": null,
  "datasets": [
    {
      "description": null,
      "name": "1kg-p3-subset",
      "id": "MWtnLXAzLXN1YnNldA=="
    }
  ]
}

In this example we sent a SearchDatasetsRequest object to the server and received a SearchDatasetsResponse object in return. This response object contained one Dataset object, which is contained in the datasets array. This approach to interacting with the server is tedious and error prone, as we have to hand-craft the request objects. It is also quite inconvenient, as we may have to request many pages of objects to get all the objects that satisfy our search criteria.

To simplify interacting with the server and to abstract away the low-level network-level details of the server, we provide a client application. To try this out, we start another instance of our virtualenv, and then send the equivalent command using:

$ source ga4gh-env/bin/activate
(ga4gh-env) $ ga4gh_client datasets-search http://localhost:8000
MWtnLXAzLXN1YnNldA==    1kg-p3-subset

The output of this command is a summary of the Datasets on that are present on the server. We can also get the output in JSON form such that each object is written on one line:

(ga4gh-env) $ ga4gh_client datasets-search -O json http://localhost:8000
{"description": null, "name": "1kg-p3-subset", "id": "MWtnLXAzLXN1YnNldA=="}

This format is quite useful for larger queries, and can be piped into jq to extract fields of interest, pretty printing and so on.

We can perform similar queries for variant data using the search_variants API call. First, we find the IDs of the VariantSets on the server using the search_variant_sets method:

(ga4gh-env) $ ga4gh_client variantsets-search http://localhost:8000
MWtnLXAzLXN1YnNldDptdm5jYWxs    mvncall

This tells us that we have one VariantSet on the server, with ID MWtnLXAzLXN1YnNldDptdm5jYWxs and name mvncall. We can then search for variants overlapping a given interval in a VariantSet as follows:

(ga4gh-env) $ ga4gh_client variants-search http://localhost:8000 \
--referenceName=1 --start=45000 --end=50000

The output of the client program is a summary of the data received in a free text form. This is not intended to be used as the input to other programs, and is simply a data exploration tool for users. To really use our data, we should use a GA4GH client library.

Part of the GA4GH reference implementation is a client library. This makes sending requests to the server and using the responses very easy. For example, to run the same query as we performed above, we can use the following code:

from __future__ import print_function

from ga4gh.client import client

httpClient = client.HttpClient("http://localhost:8000")
# Get the datasets on the server.
datasets = list(httpClient.search_datasets())
# Get the variantSets in the first dataset.
variantSets = list(httpClient.search_variant_sets(
    dataset_id=datasets[0].id))
# Now get the variants in the interval [45000, 50000) on chromosome 1
# in the first variantSet.
iterator = httpClient.search_variants(
    variant_set_id=variantSets[0].id,
    reference_name="1", start=45000, end=50000)
for variant in iterator:
    print(
        variant.reference_name, variant.start, variant.end,
        variant.reference_bases, variant.alternate_bases, sep="\t")

If we save this script as ga4gh-demo.py we can then run it using:

(ga4gh-env) $ python ga4gh-demo.py

Host the 1000 Genomes VCF

The GA4GH reference server uses a registry of files and URLs to populate its data repository. In this tutorial we will use the command-line client to create a registry similar to that used by 1kgenomes.ga4gh.org. Your system should have samtools installed, and at least 30GB to host the VCF and reference sets.

Repo administrator CLI

The CLI has methods for adding and removing Feature Sets, Read Group Sets, Variant Sets, etc. Before we can begin adding files we must first initialize an empty registry database. The directory that this database is in should be readable and writable by the current user, as well as the user running the server.

$ ga4gh_repo init registry.db

This command will create a file registry.db in the current working directory. This file should stay relatively small (a few MB for thousands of files).

Now we will add a dataset to the registry, which is a logical container for the genomics data we will later add. You can optionally provide a description using the --description flag.

$ ga4gh_repo add-dataset registry.db 1kgenomes \
    --description "Variants from the 1000 Genomes project and GENCODE genes annotations"

Add a Reference Set

It is possible for a server to host multiple reference assemblies. Here we will go through all the steps of downloading and adding the FASTA used for the 1000 Genomes VCF.

$ wget ftp://ftp.1000genomes.ebi.ac.uk//vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz

This file is provided in .gz format, which we will decompress, and then with samtools installed on the system, recompress it using bgzip.

$ gunzip hs37d5.fa.gz
$ bgzip hs37d5.fa

This may take a few minutes depending on your system as this file is around 3GB. Next, we will add the reference set.

$ ga4gh_repo add-referenceset registry.db /full/path/to/hs37d5.fa.gz \
  -d “NCBI37 assembly of the human genome” \
  --species '{"id": "9606", "term": "Homo sapiens", "source_name": "NCBI", "source_version: "1.0"}' \
  --name NCBI37 \
  --sourceUri "ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz"

A number of optional command line flags have been added. We will be referring to the name of this reference set NCBI37 when we later add the variant set.

Add an ontology

Ontologies provide a source for parsing variant annotations, as well as organizing feature types into ontology terms. A sequence ontology instance must be added to the repository to translate ontology term names in sequence and variant annotations to IDs. Sequence ontology definitions can be downloaded from the Sequence Ontology site.

$ wget https://raw.githubusercontent.com/The-Sequence-Ontology/SO-Ontologies/master/so-xp-dec.obo
$ ga4gh_repo add-ontology registry.db /full/path/to/so-xp.obo -n so-xp

Add sequence annotations

The GENCODE Genes dataset provides annotations for features on the reference assembly. The server uses a custom storage format for sequence annotations, you can download a prepared set here. It can be added to the registry using the following command. Notice we have told the registry to associate the reference set added above with these annotations.

$ wget https://ga4ghstore.blob.core.windows.net/testing/gencode_v24lift37.db
$ ga4gh_repo add-featureset registry.db 1kgenomes /full/path/to/gencode.v24lift37.annotation.db \
    --referenceSetName NCBI37 --ontologyName so-xp

Add the 1000 Genomes VCFs

The 1000 Genomes are publicly available on the EBI server. This command uses wget to download the “release” VCFs to a directory named release.

$ wget -m ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/release/20130502/ -nd -P release -l 1
$ rm release/ALL.wgs.phase3_shapeit2_mvncall_integrated_v5b.20130502.sites.vcf.gz

These files are already compressed and indexed. For the server to make use of the files in this directory we must move the wgs file, since it covers chromosomes that are represented elsewhere and overlapping VCF are not currently supported. This file could be added as a separate variant set.

We can now add the directory to the registry using the following command. Again, notice we have referred to the reference set by name.

$ ga4gh_repo add-variantset registry.db 1kgenomes /full/path/to/release/ \
    --name phase3-release --referenceSetName NCBI37

Add a BAM as a Read Group Set

Read Group Sets are the logical equivalent to BAM files within the server. We will add a BAM hosting by the 1000 Genomes S3 bucket. We will first download the index and then add it to the registry.

$ wget http://s3.amazonaws.com/1000genomes/phase3/data/HG00096/alignment/HG00096.mapped.ILLUMINA.bwa.GBR.low_coverage.20120522.bam.bai
$ ga4gh_repo add-readgroupset registry.db 1kgenomes \
    -I HG00096.mapped.ILLUMINA.bwa.GBR.low_coverage.20120522.bam.bai \
    --referenceSetName NCBI37 \
    http://s3.amazonaws.com/1000genomes/phase3/data/HG00096/alignment/HG00096.mapped.ILLUMINA.bwa.GBR.low_coverage.20120522.bam \

This might take a moment as some metadata about the file will be retrieved from S3.

Start the server

Assuming you have set up your server to run using the registry file just created, you can now start or restart the server to see the newly added data. If the server is running via apache issue sudo service apache2 restart. You can then visit the landing page of the running server to see the newly added data.

Use the client package

If you only want to use the client and don’t need the server functionality, there is a seperate pypi package, ga4gh-client, which includes only the client. It is also much quicker to install. To install, simply run:

(ga4gh-env) $ pip install --pre ga4gh_client

This installs the ga4gh_client command line program, which provides identical functionality to the ga4gh_client which is installed via the ga4gh package:

(ga4gh-env) $ ga4gh_client datasets-search http://1kgenomes.ga4gh.org

Installing the ga4gh_client package also gives you access to the client’s libraries for use in your own programs:

>>> from ga4gh.client import client
>>> client.HttpClient
<class 'ga4gh_client.client.HttpClient'>

For more examples of using the GA4GH client visit this iPython notebook.

OIDC Demonstration

If we want authentication, we must have an OIDC authentication provider. One can be found in oidc-provider, and run with the run.sh script. We can then use this with the LocalOidConfig server configuration. So:

$ cd oidc-provider && ./run.sh

In another shell on the same machine

$ python server_dev.py -c LocalOidConfig

Make sure you know the hostname the server is running on. It can be found with

$ python -c 'import socket; print socket.gethostname()'

With a web browser, go to https://<server hostname>:<server port>. You may need to accept the security warnings as there are probably self-signed certificates. You will be taken through an authentication flow. When asked for a username and password, try upper and crust. You will find yourself back at the ga4gh server homepage. On the homepage will be a ‘session token’ This is the key to access the server with the client tool as follows:

(ga4gh-env) $ ga4gh_client --key <key from homepage> variantsets-search https://localhost:8000/current
MWtnLXAzLXN1YnNldDptdm5jYWxs    mvncall