Learn About Our Meetup

5000+ Members



Join our meetup, learn, connect, share, and get to know your Toronto AI community. 



Browse through the latest deep learning, ai, machine learning postings from Indeed for the GTA.



Are you looking to sponsor space, be a speaker, or volunteer, feel free to give us a shout.

[P] Have any of you had experience with loading prebatched data into Keras or PyTorch?

Hey guys.

I’m developing a CNN at the moment, and to avoid memory issues I’ve batched my image files into blocks of 500 in my directory (each block of 500 images is a .h5 file). I’m just struggling at the moment in creating a thread-safe generator for either Keras or PyTorch that can loop through all the .h5 files in the directory, load the 500 images, and then push a batch_size quantity of images from that block (let’s say 2 images, if batch_size = 2 ) to the neural network to train on.

I’ve managed to do this somewhat successfully in Keras, using a generator, loops, and yield statements, however this isn’t really suitable for multiprocessing. So now I’m attempting to do this via a Keras sequence or a PyTorch Dataset. I would appreciate any insight that you could offer, and I’ve linked the relevant SO for more information.

submitted by /u/xandrovich
[link] [comments]

Toronto AI is a social and collaborative hub to unite AI innovators of Toronto and surrounding areas. We explore AI technologies in digital art and music, healthcare, marketing, fintech, vr, robotics and more. Toronto AI was founded by Dave MacDonald and Patrick O'Mara.