variant_mapper package

`variant_mapper` package#

`variant_mapper.mapper` module#

class variant_mapper.mapper.BaseMapper(resolver=None, ref_genome=None, ref_genome_idx=None)#

Bases: object

A base class for the mapper, do not use directly

Parameters:

resolver (gwas_norm.variants.resolvers.BaseResolver) – A resolver with methods to attempt to rescue poor quality mappings and imputation of the ALT allele. If NoneType then the base class is used. This returns a no mapping result for both methods.
ref_genome (str) – The path to an indexed FASTA file containing a reference assembly that can be used by the mapper if needed. The reference assembly will be used in cases where variants have not been mapped so the source variant is normalised (if an INDEL) just incase it is the way the indel is provided that is causing it not to map.
ref_genome_idx (str) – The path to the index file for the reference genome. If this is NoneType it is assumed to have the same basename as the ref_genome.

DNA_REGEXP = re.compile('^[ATCGatcg-]+$')#

best_mapping(source_chr_name, source_start_pos, source_ref_allele, source_alt_allele, mapping_rows, resolve=True, existing_flags=0, input_row=None, strand=1, var_id=None)#

Run the mapping algorithm for a single source variant against the localised matching mapping rows.

Parameters:

source_chr_name (str) – The chromosome name of the source variant
source_start_pos (int) – The start position of the source variant
source_ref_allele (str) – The reference allele of the source variant
source_alt_allele (str) – The alternate allele of the source variant. If this is NoneType then the alternate allele will be attempted to be assigned from the appropriate mapping in mapping_rows.
mapping_rows (list of tuple or list) – Potential matching mappings for the source variant. These should represent rows from a mapping vcf file. So the chr_name at [0], start_pos at [1], var_id at [2], ref_allele at [3], alt_allele at [4] and an input row at [5] (list but can be empty). It is assumed that the alt allele is bi-alleilic.
resolve (bool, optional, default: True) – In the event of no mapping, that is the mappings do not reach sufficient quality. Should the resolution method get called. This is mainly here to stop infinite recursion should the resolution method have to call _map_variant again.
existing_flags (int, optional, default: 0) – Any existing mapping flags that need to be added to the map_bits. This is so recursive calls can pass mapping information to each other.
input_row (Any, optional, default: NoneType) – Any input rows that you want to store alongside the mapping. Typically these will be a list representing data in columns but could be anything.
strand (int, optional, default: 1) – The strand for the source variant, should be either 1 (forward) or -1 (reverse).
var_id (str or NoneType, optional, default: NoneType) – Any existing identifiers for the variant. This will be passed to the resolution methods in the event of no mapping.

Notes

Whilst this can be called directly usually it is called from the map_variant method. As the map_variant method will typically deal with things like INDEL normalisation (and subsequent re-localisation) as well. This assumes that a set of rows have already been localised, probably based on chr:start_pos (although these are checked by this function as well). So this will take localised rows and define the best mapping row or return a no mapping result if none of them are any good.

check_ref_alleles(chr_name, start_pos, ref_allele, alt_allele)#

Check which allele (ref/alt) matches with the reference genome sequence.

This will raise an error nothing matches and an attribute error is no genome is available.

Parameters:

chr_name (str) – The chromosome name
start_pos (int) – The start position in base pairs
ref_allele (str) – The current reference allele.
alt_allele (str) – The current alternative allele.

Returns:

ref_allele (str) – The validated reference allele
alt_allele (str) – The alternate allele (should not match the reference sequence)
mapping_bits (int) – Indicate that the site has been validated against the genome and also ref flip if the reference allele has changed.

Raises:

AttributeError – If no reference genome is defined
KeyError – If nothing matches or everything matches the reference assembly.

close()#: Close anything used by the mapping class

classmethod decode_mapping_flags(flags)#

Decode the bitwise mapping flags into human readable strings.

Parameters:: flags (int) – The bitwise flags to decode.
Returns:: decoded_flags – The bitwise flags decoded into human readable strings.
Return type:: list of str

classmethod get_mapping_error(source_coords, error=None, nsites=0, input_row=None)#

return a mapping with ERROR bits and the associated error (if available).

Parameters:

source_coords (tuple) – The coordinates of the variant that we are trying to map. Should have chr_name at [0], start_pos at [1], ref_allele at [2] and alt_allele at [3].
error (Exception, optional, default: NoneType) – An exception to add to the error mapping.
nsites (int, optional, default: 0) – The number of sites in the mapping.
input_row (Any) – The row from the input file that we tried to map. Typically this will be a list but could be anything.

Returns:

error_mapping – A mapping result with mapper.ERROR.bits

Return type:

mapper.MappingResult

classmethod get_mapping_var_id(mapping)#

Extract the variant identifier from a mapping, the variant identifier will be in the map_row (currently at index 2).

Parameters:: mapping (list) – This has the source variant at [0], mapping variant at [1], the mapping row at [2] and the data_bits (map_bits) at [3].
Returns:: var_id – The variant identifier, this baseclass version will return ''
Return type:: str

classmethod get_no_data_mapping(source_coords, nsites=0, input_row=None)#

return an empty no data mapping

Returns:: no_data_mapping – A mapping result with mapper.vc.NO_DATA.bits
Return type:: mapper.MappingResult

map_variant(*args)#

Placeholder for the map_variant method

Parameters:: *args – Ignored
Raises:: NotImplementedError – Indicate that it needs overriding

open()#: Initialise the source and mapping files

classmethod order_mappings(source_chr_name, source_start_pos, source_ref_allele, source_alt_allele, mapping_rows, strand=1)#

Order potential mappings (broadly localised mappings) from most relevant to least relevant.

There is a threshold criteria below which a localised mapping nothing is not deemed to map to the source variant attributes so will be omitted from the returned list entirely.

Parameters:

source_chr_name (str) – The chromosome name of the source variant
source_start_pos (int) – The start position of the source variant
source_ref_allele (str) – The reference allele of the source variant
source_alt_allele (str) – The alternate allele of the source variant. If this is NoneType then the alternate allele will be attempted to be assigned from the appropriate mapping in mapping_rows.
mapping_rows (list of tuple) – Potential matching mappings for the source variant. These should represent rows from a mapping vcf file. So the chr_name at [0], start_pos at [1], var_id at [2], ref_allele at [3], alt_allele at [4]. It is assumed that the alt allele is bi-allelic.
strand (int, optional, default: 1) – The strand for the match can be either 1 or -1. This is optional as most strand information is not available in files so we assume 1.

Returns:

mappings – The mapping_rows aligned with the source data in order from the mapping row with the best match to the source data to the mapping row with the worst match to the source data.

Return type:

list of list

classmethod quick_match(source, mapping)#

Define how well a variant from an untrusted source matches one from a trusted source.

Parameters:

source (tuple) – The first variant to match, this is assumed to be derived from an untrusted source and we are assessing how well it matches the variant from the trusted source. The untrusted label means that the alt-allele can be set to NoneType and the alleles can be I/D/R designations. This tuple should have chr_name (str), start_pos (int), strand (1/-1) followed by separate elements for all the alleles, ref and alt.
mapping (tuple) – The second variant to match, this is assumed to have “complete” information, i.e. derived from a trusted and robust source. The trusted source label means that the alt-allele must be present and all the alleles must be DNA sequences i.e. ATCG. This tuple should have chr_name (str), start_pos (int), strand (1/-1) followed by separate elements for all the alleles, ref and alt.

Returns:

source (tuple) – The mapping variant tuple of chr_name (str), start_pos (int), strand (1/-1) followed by separate elements for all the alleles, ref and alt.
mapping (tuple) – The mapping variant tuple of chr_name (str), start_pos (int), strand (1/-1) followed by separate elements for all the alleles, ref and alt.
data_bits (int) – The mapping bits between source and mapping variants

Raises:

ValueError – If the source allele types can’t be recognised or the mapping alleles are not DNA

Notes

This performs basic matching between two variants represented in tuples of chr_name (str), start_pos (int), strand (1/-1), followed by separate elements for all the alleles.

Currently, quick_match only supports bi-alleilic variants and assumes that the source alleles and the mapping alleles are all the same case. The source variant can be from the ambiguous source and the mapping variant should be from a solid data source. As such the alternate allele in the source can be NoneType but both alleles must be present in the mapping variant. Similarly, the source variant can be an I/D/R alleilic representation but the mapping variant must be made up from ATCG. The matching algorithm is as follows. Also, ensembl format - is not supported:

Attempt to match the chromosome, if there is no match then return vc.NO_DATA.
Attempt to match the start position, if there is no match then return CHR.
Extract the alleles from the source and mapping.
Is the source reference allele DNA, if so, see step 5. If not see step 9.
Is the source alt allele DNA or NoneType, if DNA see step 6. If NoneType see step 8. If neither of these - error.
Do the source and mapping alleles match? If so return, if not see step 7.
Flip strand and test the alleles again and return the result.
Test the source ref allele for a match to the mapping. If it matches return. If not see step 7.
Are the source alleles I/D/R? If so see step 10. if not raise an error.
I/D/R alleles are currently not handled.

property resolver#: Return the resolver used by the mapper (resolvers)

static sort_map(mapping)#

A sort key method for ordering the mapping rows.

Parameters:: mapping (tuple) – The mapping bits should be at element [3].

validate_genomic_coords(mapping_data)#

Validate source coordinates against the reference genome and update the mapping data to reflect the genomic reference sequence.

Parameters:: mapping_data (gwas_norm.variants.constants.MappingResult) – The mapping result to validate against the reference genome.
Returns:: checked_mapping_data – The mapping result that has been checked against the reference genome. If it validates against the reference genome, then the
Return type:: gwas_norm.variants.constants.MappingResult

Notes

This is not designed to error out, only to update the mapping result with any genomic validation data.

validate_resolver()#: Make sure the resolver is compatible with the data source for the mapper

classmethod var_id_match(mappings, source_var_id)#

Assess the mappings to see if they match the source_var_id. If they do adjust the map_bits accordingly.

Parameters:

mappings (list of list) – Each element is a localised mapping and each localised mapping list has the structure: source variant at [0], mapping variant at [1], the mapping row at [2] and the data_bits (map_bits) at [3].
source_var_id (str or NoneType)

Returns:

mappings – Each element is a localised mapping and each localised mapping list has the structure: source variant at [0], mapping variant at [1], the mapping row at [2] and the data_bits (map_bits) at [3]. The return is not strictly required as the mapping bits are updated in place.

Return type:

list of list

class variant_mapper.mapper.EnsemblVariantMapper(rest_client, *args, **kwargs)#

Bases: BaseMapper

A variant mapper that localises variant coordinates based on queries against the Ensembl REST API.

Parameters:

rest_client (ensembl_rest_client.client.Rest) – An object for interacting with the Ensembl REST API.
*args – Arguments to the gwas_norm.variants.mapper.BaseMapper
**kwargs – Keyword arguments to the gwas_norm.variants.mapper.BaseMapper

Notes

This is only suitable for small queries and not mapping millions of variants. If no reference genome is passed then any allele normalisation is based on queries against the Ensembl Rest API.

DNA_REGEXP = re.compile('^[ATCGatcg-]+$')#: A compiled regular expression for recognising DNA strings with deletion symbols - (re.Pattern)

classmethod get_mapping_var_id(mapping)#

Extract the variant identifier from a mapping, the variant identifier will be in the map_row (currently at index 2).

Parameters:: mapping (list) – This has the source variant at [0], mapping variant at [1], the mapping row at [2] and the data_bits (map_bits) at [3].
Returns:: var_id – The variant identifier
Return type:: str

map_variant(chr_name, start_pos, ref_allele, *args, alt_allele=None, strand=1, allele_norm=True, existing_flags=0, var_id=None, **kwargs)#

Map a single variant using the Ensembl REST API.

Parameters:

chr_name (str) – The chromosome name of the variant
start_pos (int) – The 1-based start position for the variant
ref_allele (str) – The reference allele for the variant, allowed values are ATCGatcg or - for a deletion.
*args – Any positional arguments (ignored).
alt_allele (str, optional, default: NoneType) – The alternate allele, allowed values are ATCGatcg or - for a deletion or NoneType.
strand (int, optional, default: 1) – The strand for the variant (if known), should be 1 for positive/forward strand or -1 for negative/reverse strand.
allele_norm (bool, optional, default: True) – In the event of no mapping, do you want to allele normalise and attempt to map again (if normalisation has occurred).
existing_flags (int, optional, default: 0) – Any existing mapping flags that you want to pass through to the final mapping. The end user should not need to touch this and it is mainly meant for recursive calls or for subclasses to use.
var_id (str or NoneType, optional, default: NoneType) – Any existing identifiers for the variant. This will be passed to the resolution methods in the event of no mapping.
*kwargs – Any keyword arguments (ignored).

query_region(chr_name, start_pos, end_pos)#

Query for the region using the REST API.

Parameters:

chr_name (str) – The chromosome name of the variant
start_pos (int) – The 1-based start position for the variant (post ensemblisation if an INDEL)
end_pos (int) – The end position for the variant

Returns:

rest_data – This should be the result of an ensembl_rest_client.overlap.Overlap.get_region_overlap() query.

Return type:

list of dict

class variant_mapper.mapper.TabixVcfVariantMapper(mapping_vcf_file, *args, **kwargs)#

Bases: BaseMapper, _BaseVcfInterface

A variant mapper that localises variant coordinates based on tabix queries of VCF files. This class operates on a Gwas Norm VCF mapping file.

Parameters:

mapping_vcf_file (str) – The path to the mapping VCF file
mapping_vcf_idx (str or NoneType, optional, default: NoneType) – The path to the mapping VCF index. If this is NoneType
**kwargs – Arguments to the mapper.BaseMapper

close()#: Close the tabix VCF file

classmethod get_mapping_var_id(mapping)#

Extract the variant identifier from a mapping, the variant identifier will be in the map_row (currently at index 2).

Parameters:: mapping (list) – This has the source variant at [0], mapping variant at [1], the mapping row at [2] and the data_bits (map_bits) at [3].
Returns:: var_id – The variant identifier, this baseclass version will return ''
Return type:: str

map_variant(chr_name, start_pos, ref_allele, alt_allele=None, strand=1, allele_norm=True, existing_flags=0, var_id=None, input_row=None, **kwargs)#

Attempt to map a variant by localising it’s chr:pos based on a tabix query.

Parameters:

chr_name (str) – The chromosome name of the variant
start_pos (int) – The 1-based start position for the variant
ref_allele (str) – The reference allele for the variant, allowed values are ATCGatcg or - for a deletion
alt_allele (str, optional, default: NoneType) – The alternate allele, allowed values are ATCGatcg or - for a deletion or NoneType
strand (int, optional, default: 1) – The strand for the variant (if known), should be 1 for positive/forward strand or -1 for negative/reverse strand.
allele_norm (bool, optional, default: True) – In the event of no mapping, do you want to allele normalise and attempt to map again (if normalisation has occurred).
existing_flags (int, optional, default: 0) – Any existing mapping flags that you want to pass through to the final mapping. The end user should not need to touch this and it is mainly meant for recursive calls or for subclasses to use.
var_id (str or NoneType, optional, default: NoneType) – Any existing identifiers for the variant. This will be passed to the resolution methods in the event of no mapping.
*kwargs – Any keyword arguments passed to best_mapping.

open()#: Open the tabix VCF file. Note that the file is not opened as a VCF file.

class variant_mapper.mapper.ScanVcfVariantMapper(source_variants, mapping_vcf, tabix_vcf=None, chr_name='chr_name', start_pos='start_pos', strand=None, ref_allele='ref_allele', alt_allele=None, var_id=None, header=True, buffer=1000, source_join_key=None, mapping_join_key=None, buffer_sort_key=None, **kwargs)#

Bases: BaseMapper

Map and annotate source variants from a flat input file (or file like object) against the mapping vcf file.

Parameters:

source_variants (iterator) – An object that has a __next__ method implemented and can serve up a row from the file. The row should be a represented as a list. The row that is given by the iterator does not have to have the start position as an integer, that is cast internally. However, if header is True, then the first row given by the iterator should be the header row. The source_variants file should also be sorted in the same way as the mapping_vcf mapping file.
mapping_vcf (gwas_norm.variants.mapper.VcfIterator) – An object that will iterate through the VCF mapping file, this file.
header (bool, optional, default: True) – Does the source_variants file have a header. If True this should be the first row given by the iterator.
tabix_vcf (list of gwas_norm.variants.mapper.TabixVcfVariantMapper or)
NoneType (NoneType) – A mapping VCF file containing rare variants, this file is searched with tabix rather than scanned and it is only searched if a variant is not available in the mapping_vcf. It is designed to be accessed infrequently only as a last resort.
optional (NoneType) – A mapping VCF file containing rare variants, this file is searched with tabix rather than scanned and it is only searched if a variant is not available in the mapping_vcf. It is designed to be accessed infrequently only as a last resort.
default (NoneType) – A mapping VCF file containing rare variants, this file is searched with tabix rather than scanned and it is only searched if a variant is not available in the mapping_vcf. It is designed to be accessed infrequently only as a last resort.
chr_name (str, optional, default: ‘chr_name’) – The name for the chromosome name column of source_variants file. If header is False, then this should be the column number.
start_pos (str, optional, default: ‘start_pos’) – The name of the start position column of source_variants file. If header is False, then this should be the column number.
ref_allele (str, optional, default: ‘ref_allele’) – The name of the reference allele column (or effect allele) of source_variants file. If header is False, then this should be the column number.
alt_allele (str or NoneType, optional, default: NoneType) – The name of the alternate allele column (or other allele), this should be NoneType if the source_variants file has no alt_allele column.
var_id (str or NoneType, optional, default: NoneType) – The name of the variant identifier column in the source_variants file. If header is False, then this should be the column number.
header – Does the source_variants file contain a header row. If so it should be the first row given by the iterator.
tmp_dir (str or NoneType, optional, default: NoneType) – The directory to write temp files. This is used if sort=True
buffer (int, optional, default: 1000) – The number of entries to buffer and check for any normalised alleles that would, change the sort order, the larger the buffer the less chance that any allele normalisation will interfere with the output sort order, this comes at the expense of memory.
source_join_key (function or NoneType, optional, default: NoneType) – A key function to use to extract the data values from each row of the input file that will be used to join to the corresponding data values in the mapping file (provided by the mapping_join_key). This should accept a row of data as a list and return a tuple of values to join on. The default (NoneType) will join on the chromosome name as a string and the base pair position as an integer.
mapping_join_key (function or NoneType, optional, default: NoneType) – A key function to use to extract the data values from each row of the mapping file that will be used to join to the corresponding data values in the input source file (provided by the source_join_key). This should accept a row of data as a list and return a tuple of values to join on. The default (NoneType) will join on the chromosome name as a string and the base pair position as an integer.
buffer_sort_key (function or NoneType, optional, default: NoneType) – A function to provide data values to be sorted on in the event that a mapped variant has been normalised. This attempts to ensure that the output rows are output in the correct sort order even when a potential disruptive event such as variant normalisation has occured. This should accept a gwas_norm.variants.constants.MappingResult and return a tuple used to sort the mapping results in the buffer. The default, NoneType, will use a function that provides the mapping chromosome name (as a string) and the mapping (normalisaed) base pair position as an integer.
**kwargs – Keyword arguments passed to the gwas_norm.variants.mapper.BaseMapper.

Raises:

ValueError – If the buffer value is < 0.

DNA_REGEX = re.compile('^[ATCGatcg]+$')#

MAPPING_FILE_TYPE#: alias of VcfIterator

check_new_chr_name(new_chr_name)#

Perform some basic sort order checks of a new chromosome name.

This is called when the source file being mapped moves to the next chromosome.

Parameters:: new_chr_name (str) – The next chromosome name.
Raises:: IndexError – If the input file and the mapping file have differing sort orders or if the input file is not sorted correctly (i.e. the chromosome names are not grouped)

check_norm(chr_name, start_pos, ref_allele, alt_allele=None, strand=None)#: Normalise the

check_ref(chr_name, start_pos, ref_allele, alt_allele=None, strand=None)#: Normalise the

close()#: Close source and the mapping files

extract_alt_allele(source_row)#

Extract the alternate allelle from the input row.

Parameters:: source_row (list of str) – The source row to extract the alt allele from. It is assumed that the row has the same index possitions as a row from a VCF file.
Returns:: alt_allele – The value for the alt allele.
Return type:: str

extract_nothing(source_row)#

A dummy function that returns NoneType, irrespective of what has been passed.

Parameters:: source_row (any) – Any arguments (ignored)
Returns:: nothing – An empty return type.
Return type:: NoneType

extract_positive_strand(source_row)#

Return a positive strand value. This is the default strand extraction function.

Parameters:: source_row (list of str) – The source row this is ignored.
Returns:: strand – The value for a positive strand 1.
Return type:: int

extract_strand(source_row)#

Return a positive row value.

Parameters:: source_row (list of str) – The source row to extract the strand from. Values that are interpreted as the strand are 1, -1, + and -. NoneType values results in a default strand of 1 being returned and anything else raises a ValueError.
Returns:: strand – The value for a strand. If the value at that position can’t be cast to an int and is NoneType, then a default strand of 1 is returned, any other value.
Return type:: int
Raises:: ValueError – If the strand value is not one of the recognised values.

extract_var_id(source_row)#

Extract the variant identifier from the input row.

Parameters:: source_row (list of str) – The source row to extract the variant identifier from. It is assumed that the row has the same index positions as a row from a VCF file.
Returns:: var_id – The value for the variant identifier.
Return type:: str

get_buffer_sort_key()#

Get the sort key to use to sort the buffer when a normalised variant is encountered.

Returns:: sort_key – The function that provided the key to sort on.
Return type:: function

get_mapping_join_key()#

Get the join key used by the mapping file.

Returns:: join_key – A function that will provide a yuple from an input row or a tuple of tuples with (input row index, datatype) that will be used to query the input row for the join keys
Return type:: function or tuple

static get_mapping_var_id(mapping)#

Extract the variant identifier from a mapping, the variant identifier will be in the map_row (currently at index 2).

Parameters:: mapping (list) – This has the source variant at [0], mapping variant at [1], the mapping row at [2] and the data_bits (map_bits) at [3].
Returns:: var_id – The variant identifier, this baseclass version will return ''
Return type:: str

get_source_join_key()#

Get the join key used by the source file that will be mapped.

Returns:: join_key – A function that will provide a yuple from an input row or a tuple of tuples with (input row index, datatype) that will be used to query the input row for the join keys
Return type:: function or tuple

property header#: Return the header row of the input source file (str).

init_mapping_file()#: Initialise the mapping file that will be scanned through using a join with the source file

init_source_file()#: Initialise the source file that will be mapped.

map_variant(join_row)#

Perform mapping on rows returned from a join scan against the mapping file.

In this scenario there could be multiple rows from the source file being mapped matched against multiple potential mappings. If there are multiple source_rows then they are all mapped and stored.

Parameters:: join_row (list or list) – The rows from the source/mapping file that are matched on chromosome name and start position. The source rows will be in a list at element [0] and the mapping rows will be represented as a list at element [1]. There maybe multiple source rows and multiple mapping rows. So the source rows are treated independently and mapped to the mapping rows.

classmethod mapping_correct(mapping_file)#

Determine if the mapping file is the correct type.

Parameters:: mapping_file (Any) – The mapping file object to test.
Raises:: TypeError – If the mapping file is not the correct type.

open()#: Initialise the source and mapping files

property output_sorted#

Return the output is sorted value (bool).

Notes

A flag indicating if the output file is sorted or not, if this is False then it means that an allele has been normalised in such a away that it puts it at the top of the sort order in the buffer. This may mean that the output requires sorting as the normalised value may be further up the sort order but that has already been output. If this is True, then everything should be ok. This value should be checked post mapping run.

sort_buffer(mapping_data)#

Perform a full sort on the buffer. This adds mapping data to the buffer and then sorts

Parameters:: mapping_data (variants.constants.MappingResult) – A mapping to check for sort order against the mapping buffer.

Notes

This will add the mapping data to the buffer and then perform the sort. The key from the source file JoinIterator is used to perform the sort.

property source#: Get the input source.

test_sort_order(mapping_data)#

Test the mapping data against the buffer to determine if the sort order is correct.

Parameters:: mapping_data (variants.constants.MappingResult) – A mapping to check for sort order against the mapping buffer.
Returns:: is_sorted – An indicator of the sort order status. True everything is the correct sort order False, then the mapping result belongs up the sort order somewhere.
Return type:: bool
Raises:: IndexError – If the buffer does not contain enough entries to test.

Notes

The key from the source file JoinIterator is used to perform the sort.

validate_resolver()#: Make sure the resolver is compatible with the data source for the mapper

class variant_mapper.mapper.VcfIterator(mapping_vcf_file)#

Bases: _BaseVcfInterface

An iterator that uses pysam to iterate through a VCF file from start to finish and provide rows of information as lists.

close()#

Close the tabix VCF file

Returns:: self – For chaining.
Return type:: gwas_norm.variants.mapper.VcfIterator

open()#

Open the tabix VCF file. Note that the file is not opened as a VCF file.

Returns:: self – For chaining.
Return type:: gwas_norm.variants.mapper.VcfIterator

`variant_mapper.resolvers` module#

class variant_mapper.resolvers.BaseResolver(*args, **kwargs)#

Bases: object

The base resolver class. Do not use directly

Parameters:

*args – Any arguments (ignored)
**kwargs – Any keyword arguments (ignored)

Notes

The idea behind a resolver class is to handle situations where a variant localises but can’t be mapped. So, the user can define their own methods for dealing with variant resolution using data extracted from the localisation source. Two methods should be implemented: resolve_poor_mapping and impute_alt_allele. The base class versions of these simply return a no mapping.

DEFAULT_INTERNAL_DELIMITER = '|'#: The internal delimiter value for flattened data string when MappingFileResolver.extract_summary_metadata_row is called (str)

METADATA_SUMMARY_ROW_HEADER = ['chr_name_mapper', 'start_pos_mapper', 'strand_mapper', 'ref_allele_mapper', 'alt_allele_mapper', 'nsites', 'map_info']#: The header column names that accompany can accompany the data returned by the MappingFileResolver.extract_summary_metadata_row (list of str).

MIN_ALT_IMPUTE_EVIDENCE = 11776#: The minimal amount of evidence that a variant must have in order to attempt alt allele imputation (int)

extract_metadata(row)#

Extract the required metadata from a mapped row.

Parameters:: row (pysam.VariantRecord) – A variant record with the populations (samples) and metadata (info) that is expected in a mapping file.
Returns:: metadata – The extracted metadata
Return type:: dict

Notes

The assumption here is that the row is derived from a VCF file that has the population allele numbers and counts in the sample sections and a VEP annotation in the iNFO field. Also, all variants should be represented as bi-allelic.

classmethod extract_summary_metadata_row(mapping, meta, *args, decode_map_info=False, **kwargs)#

Helper method to extract summary information as a list that can be written to file.

Parameters:

mapping (gwas_norm.variants.constants.MappingResult) – A named tuple with the following fields source_coords, mapping_coords, errors, mapping_bits.
mapping – The mapping result to provide the mapped coordinates.
meta (dict) – The extracted metadata information from the mapping variant. i.e. the result of calling obj.extract_metadata().
*args – Any other positional arguments (currently ignored)
decode_map_info (bool, optional, default: False) – Should the map info be decoded into a delimited string or remain as an encoded bitwise integer.
**kwargs – Any other keyword arguments (currently ignored)

Returns:

outrow – Summary mapping information that can be written to a flat csv file. The order of the columns are the same as MappingFileResolver.METADATA_SUMMARY_ROW_HEADER

Return type:

list

Notes

This baseclass version extracts the mapping coordinate information and the mapping bits (or decoded string depending on decoding mapping flags). This can be overridden to provide different information if needed.

See also

gwas_norm.variants.resolvers.MappingFileResolver.METADATA_SUMMARY_ROW_HEADER, gwas_norm.variants.resolvers.MappingFileResolver.extract_metadata

classmethod get_mapping_error(source_coords, error=None, nsites=0, input_row=None)#

return a mapping with ERROR bits and the associated error (if available).

Parameters:: error (Exception) – An exception to add to the error mapping.
Returns:: error_mapping – A mapping result with mapper.ERROR.bits
Return type:: mapper.MappingResult

classmethod get_no_data_mapping(source_coords, nsites=0, input_row=None)#

return an empty no data mapping

Returns:: no_data_mapping – A mapping result with mapper.vc.NO_DATA.bits
Return type:: mapper.MappingResult

impute_alt_allele(mappings, input_row=None)#

Attempt to assign an alternate allele based on data from all the mappings.

Parameters:

mappings (list of list) – The mapping_rows aligned with the source data in order from the mapping row with the best match to the source data to the mapping row with the worst match to the source data. Each tuple should have the structure of source coordinates (gwas_norm.variants.constants.MapCoords). mapping coords (gwas_norm.variants.constants.MapCoords), mapping bits and mapping row (the matching row from the mapping data source)
input_row (Any, optional, default: NoneType) – Mainly for passing through to any resolved mapped variants. This will be added to the source_row attribute.

Returns:

no_data_mapping – The BaseMapper implementation returns a mapping result with gwas_norm.variants.constants.NO_DATA.bits.

Return type:

mapper.MappingResult

Notes

This offers the option to resolve poor mappings using any available metadata in mappings.

resolve_poor_mapping(mappings, input_row=None)#

A method that is called in the case when there are no high quality mappings. In reality there is probably not much to be done but this offers the option to resolve poor mappings using any available metadata in mappings.

Parameters:

mappings (list of list) – The mapping_rows aligned with the source data in order from the mapping row with the best match to the source data to the mapping row with the worst match to the source data. Each tuple should have the structure of source coordinates (gwas_norm.variants.constants.MapCoords). mapping coords (gwas_norm.variants.constants.MapCoords), mapping bits and mapping row (the matching row from the mapping data source)
input_row (Any, optional, default: NoneType) – Mainly for passing through to any resolved mapped variants. This will be added to the source_row attribute.
var_id (str or NoneType, optional, default: NoneType) – Any existing identifiers for the variant. This will be passed to the resolution methods in the event of no mapping.

Returns:

no_data_mapping – The BaseMapper implementation returns a mapping result with gwas_norm.variants.constants.NO_DATA.bits

Return type:

mapper.MappingResult

static sort_map(mapping)#

A sort key method for ordering the mapping rows.

Parameters:: mapping (tuple) – The mapping bits should be at element [3].

validate_data_source(data_source)#

A method that can be used by the resolver to determine if the expected information is present in the data source.

Parameters:: data_source (any) – A data source to validate.

Notes

For example it can be used to determine if certain expected fields are present within a VCF header.

class variant_mapper.resolvers.PopulationResolver(*args, populations=None, allele_freq_method='mean', unsafe_alt_infer=0.05, freq_data_source=False, **kwargs)#

Bases: BaseResolver

A resolver class that handles some population arguments for specifying groups of populations that can be used to gather allele frequency info

ALLOWED_ALLELE_FREQ = ['mean', 'hierarchy']#: Allowed allele frequency methods (list of `str)

HIER_AAF = 'hierarchy'#: Keyword for hierarchical alternate allele frequency method (str)

MEAN_AAF = 'mean'#: Keyword for mean alternate allele frequency method (str)

METADATA_SUMMARY_ROW_HEADER = ['chr_name_mapper', 'start_pos_mapper', 'strand_mapper', 'ref_allele_mapper', 'alt_allele_mapper', 'nsites', 'map_info', 'alt_allele_freq', 'used_pops']#: The header column names that accompany can accompany the data returned by the MappingFileResolver.extract_summary_metadata_row (list of str).

static extract_hierarchy_aaf_counts(pop_weights, variant_pops)#

This performs a hierarchical population alternate allele frequency calculation.

Parameters:

pop_weights (list of tuple) – Each tuple should contain a tuple of population names at [0] with the most favoured populations nearer the start of the tuple (population group) and a weight for the population group at 1. All the weights across the population groups should add up to 1 (this is not checked here but is checked in the validation functions) variant_pops.
variant_pops (dict) – This has the population name as a keys and a tuple of (allele number int, allele count int) as values. If any data is missing for the population the allele number and allele count values will be NoneType.

Returns:

alt_allele_freq – The frequency of the alternate allele. If no allele frequencies are available for the sample then NoneType is returned.

Return type:

float or NoneType

Notes

The hierarchical method works as follows. The idea is that there are some populations that you will want to preferentially take allele counts from (i.e. with the highest sample size). However, maybe there is no data for the favoured population so fallback populations can be supplied. If none of the populations suffice then NoneType is the fallback. So, this method is designed to return allele frequency data in as many cases as possible. This method can accept several population hierarchy groups with weights for each group. So you could have a group of European ancestry population and a group of South Asian ancestry populations and calculate a weighted alternate allele frequency with 0.75 European and 0.25 South Asian. Note that this does not check the format of pop_weights so ensure that gwas_norm.variants.resolvers.PopulationResolver.validate_population_kwarg is called first.

static extract_hierarchy_aaf_freq(pop_weights, variant_pops)#

This performs a hierarchical population alternate allele frequency calculation.

Parameters:

pop_weights (list of tuple) – Each tuple should contain a tuple of population names at [0] with the most favoured populations nearer the start of the tuple (population group) and a weight for the population group at 1. All the weights across the population groups should add up to 1 (this is not checked here but is checked in the validation functions) variant_pops.
variant_pops (dict) – This has the population name as a keys and a tuple of (allele number int, allele count int) as values. If any data is missing for the population the allele number and allele count values will be NoneType.

Returns:

alt_allele_freq – The frequency of the alternate allele. If no allele frequencies are available for the sample then NoneType is returned.

Return type:

float or NoneType

Notes

static extract_mean_aaf_counts(pop_weights, variant_pops)#

This performs a (weighted) mean allele frequency calculation across all the supplied populations.

Parameters:

pop_weights (list of tuple) – Each tuple should contain a population name at [0] (str) and a weight for the population at 1. All the weights across the named populations groups should add up to 1 (this is not checked here but is checked in the validation functions)
variant_pops (dict) – This has the population name as a keys and a tuple of (allele number int, allele count int) as values. If any data is missing for the population the allele number and allele count values will be NoneType.

Returns:

alt_allele_freq – The frequency of the alternate allele. If no allele frequencies are available for the sample then NoneType is returned.

Return type:

float or NoneType

Notes

This will calculate the weighted mean of the alternate allele frequency across the supplied populations. If any of the populations lack data then NoneType is returned. Note that this does not check the format of pop_weights so ensure that gwas_norm.variants.resolvers.PopulationResolver.validate_population_kwarg is called first.

static extract_mean_aaf_freq(pop_weights, variant_pops)#

This performs a (weighted) mean allele frequency calculation across all the supplied populations.

Parameters:

pop_weights (list of tuple) – Each tuple should contain a population name at [0] (str) and a weight for the population at 1. All the weights across the named populations groups should add up to 1 (this is not checked here but is checked in the validation functions)
variant_pops (dict) – This has the population name as a keys and a tuple of (allele number int, allele count int) as values. If any data is missing for the population the allele number and allele count values will be NoneType.

Returns:

alt_allele_freq – The frequency of the alternate allele. If no allele frequencies are available for the sample then NoneType is returned.

Return type:

float or NoneType

Notes

static extract_no_aaf(*args, **kwargs)#

A dummy method that is called if no population data is available in the file. Under normal circumstances this should not be called.

Parameters:

*args – Positional arguments - ignored
**kwargs – Keyword arguments - ignored

Returns:

nothing – A non existant alternate allele frequency

Return type:

NoneType

list_populations()#

classmethod validate_population_kwarg(populations, allele_freq_method)#

Validate the populations that have been given to the object.

Parameters:

populations (list of str or tuple) – One or more populations that are specified in the data source. If the list contains strings, these should match the population names (sample names) in the data source (valid for allele_freq_method='mean' and allele_freq_method='hierarchy'). If the list contains tuples, there are several options. If the tuple has two elements and the first is a string (population name) with the second being a float between 0-1 (weight for the population) (valid for allele_freq_method='mean'), then this will give a weighted allele alternate frequency. If the tuple contains a tuple of strings at [0] and a float between 0-1 (weight for the population) then this is valid for allele_freq_method='hierarchy' and the first available population allele frequency is used to calculate a weighted alternate allele frequency.
allele_freq_method (str) – The method that is used to determine the alternate allele frequency. can be either, ‘mean’, ‘hierarchy’.

Returns:

valid_populations – Where either the tuple is valid for the mean allele frequency method or the hierarchical allele frequency method.

Return type:

list of tuple

Raises:

ValueError – If there are no populations to evaluate or the allele_freq_method is unknown.
TypeError – If there are any issues with the population data format

class variant_mapper.resolvers.MappingFileResolver(*args, **kwargs)#

Bases: PopulationResolver

A resolver for use with gwas_norm mapping files. See here

Parameters:

*args – Any arguments (ignored)
populations (NoneType or list of str or tuple, optional, default: NoneType) – One or more populations that are specified in the mapping VCF file. These are located where the sample fields are and the row format for them should be ‘AN:AC’. If this is NoneType then it is assumed that all populations that have been specified in the mapping file should be used. If the list contains strings, these should match the population names (sample names) in the mapping VCF file (valid for allele_freq_method='mean' and allele_freq_method='hierarchy'). If the list contains tuples, there are several options. If the tuple has two elements and the first is a string (population name) with the second being a float between 0-1 (weight for the population) (valid for allele_freq_method='mean'), then this will give a weighted allele alternate frequency. If the tuple contains a tuple of strings at [0] and a float between 0-1 (weight for the population) then this is valid for allele_freq_method='hierarchy' and the first available population allele frequency is used to calculate a weighted alternate allele frequency.
allele_freq_method (str, optional, default: mean) – The method that is used to determine the alternate allele frequency. can be either, ‘mean’, ‘hierarchy’. If the user want to override this class and add more then they should add them to are in a class variable gwas_norm.variants.resolvers.MappingFileResolver.ALLOWED_ALLELE_FREQ`
unsafe_alt_infer (float, optional, default: 0.05) – In the case when the ALT allele is not present, it will attempt to be inferred from matches based on chr_name, start_pos, ref_allele. If there are multiple possible matches, then the match with the greatest MAF is selected (based on the populations requested). However, if > 1 population has a MAF >= than this value then the ALT allele is not inferred and no mapping is returned. This must be a frequency between 0-1. This is designed to handle multiple common ALT choices.
**kwargs – Any keyword arguments to the base class

Notes

METADATA_SUMMARY_ROW_HEADER = ['chr_name_mapper', 'start_pos_mapper', 'strand_mapper', 'ref_allele_mapper', 'alt_allele_mapper', 'nsites', 'map_info', 'alt_allele_freq', 'used_pops', 'var_id', 'worst_consequence', 'worst_clinvar', 'cadd_raw', 'cadd_phred', 'sift', 'polyphen', 'datasets']#: The header column names that accompany can accompany the data returned by the MappingFileResolver.extract_summary_metadata_row (list of str).

extract_cadd(row)#

Extract the cadd annotations from the row.

Parameters:: row (pysam.VariantRecord) – A record derived from a mapping file.
Returns:: var_id – The variant identifier, if not available will be a .
Return type:: str

static extract_datasets(row)#

Extract the datasets that the variant has been found in.

Parameters:: row (pysam.VariantRecord) – A record derived from a mapping file.
Returns:: datasets – The datasets that contain this variant. If this field is not available then NoneType is returned.
Return type:: tuple or NoneType

static extract_id(row)#

Extract the variant ID from the row.

Parameters:: row (pysam.VariantRecord) – A record derived from a mapping file.
Returns:: var_id – The variant identifier, if not available will be a .
Return type:: str

extract_metadata(mapping)#

Extract the required metadata from a mapped row.

Parameters:: row (pysam.VariantRecord) – A variant record with the populations (samples) and metadata (info) that is expected in a mapping file.
Returns:: metadata – The extracted metadata
Return type:: dict

Notes

static extract_pops(row)#

Extract the population data for a variant.

Parameters:: row (pysam.VariantRecord) – A record derived from a mapping file.
Returns:: variant_pops – This has the population name as a keys and a tuple of (allele number int, allele count int) as values. If any data is missing for the population the allele number and allele count values will be NoneType.
Return type:: dict

classmethod extract_summary_metadata_row(mapping, meta, *args, decode_map_info=False, **kwargs)#

Helper method to extract summary information as a list that can be written to file.

Parameters:

mapping (MappingResult) – A named tuple with the following fields source_coords, mapping_coords, errors, mapping_bits.
meta (dict) – The extracted metadata information from the mapping variant.
decode_map_info (bool, optional, default: False) – Should the map info be decoded into a delimited string or remain as an encoded bitwise integer.

Returns:

outrow – Summary mapping information that can be written to a flat csv file. The order of the columns are the same as MappingFileResolver.METADATA_SUMMARY_ROW_HEADER

Return type:

list

Notes

This can be overridden to provide different information if needed.

`variant_mapper.norm` module#

class variant_mapper.norm.RefNorm(ref_fasta, index=None, cachesize=100000)#

Bases: object

Handles interactions with the reference genome sequence for reference allele lookups and INDEL normalisations.

Parameters:

ref_fasta (str) – An indexed FASTA reference sequence
index (str or NoneType, optional, default: NoneType) – An index file, if NoneType, then it is assumed to be the same name as the reference FASTA file.
cachesize (int, optional, default: 100000) – The size of sequences to store in memory

cache_ref_assembly_match(chr_name, start_pos, alleles)#

Given the coordinates and multiple alleles return a boolean list indicating which alleles match the reference genome assembly.

This is useful if you suspect that the ref and alt alleles have been swapped around for some reason.

This version will lookup against a pre-cached reference assembly sequence. The idea behind the cache is to store a string of reference assembly in memory that will be used for several lookups, for example if multiple consecutive sites are being queried. However, it does not seem to confer much (ANY) performance benefit but is left here for reference.

Parameters:

chr_name (str) – The chromosome containing the reference allele.
start_pos (int) – The start position of the reference allele.
alleles (list) – The allele to check against the reference genome assembly.

Returns:

matches – True if the allele matches the reference genome assembly, False if not

Return type:

list of bool

close()#: Close a FASTA reference genome

static get_minimal_alleles(start_pos, ref, alt)#

Get the minimal representation of a variant, based on the ref + alt alleles in a VCF this is used to make sure that multiallelic variants in different datasets, with different combinations of alternate alleles , can always be matched directly. Taken from: here <https://github.com/ericminikel/minimal_representation/blob/master/minimal_representation.py>.

Parameters:

start_pos (int) – The start position of the reference allele.
ref (str) – The reference allele.
alt (str) – The alternate allele.

Returns:

start_pos (int) – The start position of the minimally represented reference allele.
ref (str) – The minimally represented reference allele.
alt (str) – The minimally represented alternate allele.

property is_open#

normalise_alleles(chr_name, start_pos, ref, alt)#

Normalise INDEL alleles according to Tan et al. 2015 . This code also heavily based on this .

Parameters:

chr_name (str) – The chromosome containing the reference allele.
start_pos (int) – The start position of the reference allele.
ref (str) – The reference allele must be ATCGNatcgn-. The reference allele will be checked against the reference genome assembly.
alt (str) – The alternate allele must be ATCGNatcgn-.

Returns:

chr_name (str) – The chromosome containing the reference allele.
start_pos (int) – The start position of the normalised reference allele.
norm_ref (str) – The normalised reference allele.
norm_ref (str) – The normalised alternate allele.
was_normalised (bool) – True if the alleles have undergone normalisation, False if not

Raises:

common.SequenceError – If either the reference allele or alternate allele sequence is not ATCGNatcgn-
ValueError – If the ref and alt alleles are the same
KeyError – If the reference allele can not be found in the reference assembly

normalise_multi_alleles(chr_name, start_pos, ref, *alts)#

Normalise multi-allelic INDEL alleles according to Tan et al. 2015 . This code also heavily based on this .

Parameters:

chr_name (str) – The chromosome containing the reference allele.
start_pos (int) – The start position of the reference allele.
ref (str) – The reference allele must be ATCGNatcgn-. The reference allele will be checked against the reference genome assembly.
*alt (str) – One or more alternate allele must be ATCGNatcgn-.

Returns:

chr_name (str) – The chromosome containing the reference allele.
start_pos (int) – The start position of the normalised reference allele.
norm_ref (str) – The normalised reference allele.
norm_alts (tuple of str) – The normalised alternate alleles.
was_normalised (bool) – True if the alleles have undergone normalisation, False if not

Raises:

common.SequenceError – If either the reference allele or alternate allele sequence is not ATCGNatcgn-
ValueError – If the ref and alt alleles are the same
KeyError – If the reference allele can not be found in the reference assembly

open()#: Open a FASTA reference genome

ref_assembly_match(chr_name, start_pos, alleles)#

Given the coordinates and multiple alleles return a boolean list indicating which alleles match the reference genome assembly.

This is useful if you suspect that the ref and alt alleles have been swapped around for some reason.

Parameters:

chr_name (str) – The chromosome containing the reference allele.
start_pos (int) – The start position of the reference allele.
alleles (list) – The allele to check against the reference genome assembly.

Returns:

matches – True if the allele matches the reference genome assembly, False if not

Return type:

list of bool

search_assembly(chr_name, start_pos, end_pos)#

Search the reference genome assembly for sequences encompassed by chr_name, start_pos and end_pos.

Parameters:

chr_name (str) – The chromosome to extract from.
start_pos (int) – The 1-based start position of chr_name.
end_pos (int) – The end position

Returns:

sequence – The sequence encompassed by the coordinates.

Return type:

str

valid_ref(chr_name, start_pos, ref_allele)#

Verify that the given reference allele matches the reference assembly.

Parameters:

chr_name (str) – The chromosome containing the reference allele.
start_pos (int) – The start position of the reference allele.
ref_allele (str) – The reference allele to check against the reference genome assembly

Returns:

match – True if the reference allele matches the reference genome assembly, False if not

Return type:

bool

class variant_mapper.norm.EnsemblRefNorm(rest_client, *args, **kwargs)#

Bases: RefNorm

Handles interactions with the reference genome sequence for reference allele lookups and INDEL normalisations.

Parameters:

rest_client (ensembl_rest_client.client.Rest) – An object for interacting with the Ensembl REST API.
*args – Arguments to the gwas_norm.variants.norm.RefNorm
**kwargs – Keyword arguments to the gwas_norm.variants.norm.RefNorm

Notes

This version queries the Ensembl REST client to get the reference sequence information.

close()#: A dummy close method, to avoid opening reference genome files.

open()#: A dummy open method, to avoid opening reference genome files.

search_assembly(chr_name, start_pos, end_pos, strand=1, species='human', data_format='plain', **kwargs)#

Search the reference genome assembly for sequences encompassed by chr_name, start_pos and end_pos.

Parameters:

chr_name (str) – The chromosome to extract from.
start_pos (int) – The 1-based start position of chr_name.
end_pos (int) – The end position
strand (int, optional, default: 1) – The strand for the returned sequence.
species (str, optional, default: human) – The species for the sequence.
data_format (str, optional, default: plain) – The data format for the sequence, can be either fasta or plain. I have not tried fasta before.
**kwargs – Keyword arguments passed to ensembl_rest_client.client.Rest.get_sequence_region.

Returns:

sequence – The sequence encompassed by the coordinates.

Return type:

str

variant_mapper package

Contents

variant_mapper package#

variant_mapper.mapper module#

variant_mapper.resolvers module#

variant_mapper.norm module#

`variant_mapper` package#

`variant_mapper.mapper` module#

`variant_mapper.resolvers` module#

`variant_mapper.norm` module#