Error with AnchoredGTFDl

[PelFritz]

Hi,
I am using AnchoredGTFDl to extract promoter sequences, however when I call the load_all() I get the following error
"ValueError: all input arrays must have the same shape".

My assumption is that this arises from the fact that some gene coordinates are too close to the end of the chromosome and hence we do not get the appropriate extraction length. My code is below

`import numpy as np
from kipoiseq.dataloaders import AnchoredGTFDl

fasta_path = 'Zea_mays.Zm-B73-REFERENCE-NAM-5.0.dna.toplevel.fa'
gtf_path = 'Zea_mays.Zm-B73-REFERENCE-NAM-5.0.51.gtf'

dl = AnchoredGTFDl(gtf_path, fasta_path, num_upstream=1000, num_downstream=500,
gtf_filter='gene_biotype == "protein_coding"')

data = dl.load_all()`

As a work around I used the code below but I don't know if this is okay or there is some function to check extracted sequence length automatically.

`sequence = []
gene_id = []
for seq in dl:
if len(seq['inputs']) == 1500:
gene_id.append(seq['metadata']['gene_id'])
sequence.append(seq['inputs'])

sequence = np.array(sequence)
print(sequence.shape)`

Is there some way to assert sequence length to be the same?

[Hoeze]

Hi @PelFritz, thanks for posting this issue.

I think there are two valid solutions for this issue:

• Pad sequences with N's
• Filter too short sequences

This should be done here I guess:

kipoiseq/kipoiseq/dataloaders/sequence.py

Line 512 in b5c9b5f

sequence = self._fa.extract(interval)

@Karollus What's your opinion?

[Karollus]

Sorry for the very late reply, I somehow missed this.

I think the most feasible solution (without changing the behaviour of the extract function- which would impact a lot of dataloaders) is to have a padding or filtering afterwards. This is a bit ugly - it runs a check for every extracted sequence, but I see no other real solution. I think N-padding is probably better than filtering (filtering can be done by done relatively easily by the user by excluding genes too close to the chromosome end, padding is harder to achieve for the user as it requires editing the fasta - so it seems that providing padding would be more useful). @Hoeze, does that sound reasonable?

I can try to prototype it in the next few days

[Hoeze]

@Karollus maybe we can have an additional flag in the dataloader that chooses the behavior:
a) raise an error
b) log a warning and just remove the sequence
c) silently pad the sequence