I’m planning on sorting ~100,000 images to use as data for a computer vision application. With this much data, shaving a little time off of each picture would add up quickly. I was wondering whether there are any tools or techniques to make this as quick and easy as possible.

For my specific task I’m simply looking to go through an entire folder of images and discard those that aren’t ‘good pictures’, no complicated sorting required.

Thanks in advance, and sorry if this is the wrong tag.

I’m doing video prediction research i.e. predicting the next frame in a video sequence. In essential it’s just a mapping from the past frame into the future frame. I wonder how I can incorporate a scalar input in addition to the input frame.

Since I’m just using CNN operations and never making any flattening of the feature maps, I cannot concatenate the scalar input directly. I have found this which suggest that one could treat the bias as the scalar input of some CNN layer but doing so you are not directly adding any parameters to the scalar input.

Does anyone have any experience with this? All info, papers etc are appreciated!

https://github.com/Santosh-Gupta/ScientificSummarizationDataSets

A few months ago, I released several a dataset from ~7 million papers for ~12 million datapoints. I think the most exciting part were the datasets designed by a similar methodology of Alexios Gidiotis, Grigorios Tsoumakas [https://arxiv.org/abs/1905.07695] who discovered that there are many papers with structured abstractions, whose sections correspond to entire sections within the papers.

Having a dataset of these abstract sections and full paper sections is probably the best dataset available for research paper summarization, as far as I know.

Using some of the text processing methods in Gidiotis, Tsoumakas, and using Semantic Scholar’s Science Parse, I was able to create a dataset from Arxiv and the Semantic Scholar Corpus.

I have now released a model using a slightly modified version of the BertSum repo [ https://github.com/nlpyang/BertSum https://arxiv.org/abs/1903.10318 ]. The model was trained on a batch size of 1024 for 5000 steps, and then a batch size of 4096 for 25000 steps.

The datasets and model are all available here.

https://github.com/Santosh-Gupta/ScientificSummarizationDataSets

I also included text processing and training setups for Pointer-Generator and the Tensor2Tensor transformers abstractive summarizers. At the time they were the best for abstractive summarization, but for the purposes of my future project, I needed the most accurate summarizer, which needed an extractive method.

I have a medical data set of patients ( about 50k) and each patient has around 20 – 30 records of their vitals for each hour. The Target variable is a binary variable which is 1 if the Patient contracted the Illness or 0 otherwise , so for many patients they first few row’s are 0 are and then it switches to 1 (signifying that the patient caught the illness at that point of time ).

Till now i have been treating this data as non-temporal and considering each row to be a unique record , which has been working pretty well but i would rather treat the data as temporal , any suggestions on what techniques i can use?

Also i am currently using Autoencoders to reduce the dimentionality of the data and running a CNN over the reduced data.

Thanks in advance !

Has anyone had any experience with this? I have a rather wide feature set in my time series data, many features with vastly different scales. I’m attempting to batch-train an LSTM autoencoder. So far, I have come to the following conclusions:

• I’m reluctant to use first differences as I don’t want to destroy level information.
• I don’t want to z-score normalize data before training as many components are non-stationary
• I don’t want to use sliding min-max or sliding z-score as that destroys any volatility information between subsequent minibatches

​

So far, the following thoughts have come to mind:

• Using layer normalization, yielding equal normalization statistics for all features accross my minibatch. Destroys however information between individual features in a given sample
• Manual z-score operations inside my network. Take the resulting statistics, pass through linear layer to adapt dimensionality and initialize the LSTM hidden layer. So far, doesn’t really work. But a variant of this (perhaps concatenating to the lstm output…?) seems to be my current focus.
• This approach seems promising:
  • https://arxiv.org/pdf/1902.07892.pdf
• … but somehow, it doesn’t yield good results either. Seems very learning rate dependent which is a bit of a bummer.

Any thoughts or successful ideas?

For a small project I have been looking at applying primarily RNN architectures to a multivariate time series problem. However, research that provides reasonable assumptions on how to model the time series in terms of the residual and a neural network seems to be hard to find. Although for long term simulation of scenarios this seems to be a core aspect of analyzing time series, considering classical time series research.

I would be very grateful for pointing out good papers concerning neutral nets in time series or would like to talk about what kind of stochastic model people having experience chose and why. As a starting point I suggest RNN architectures can fit the trend, seasonality and an ARMA(p,q) model. However modelling time dependent variance and extremal values seems to be a rather hard task for neural nets.

Sorry for not having great academic english and looking forward to your answers!

I guess I’ll mark this is a discussion, in case any one is not tired of this topic, our (I would say fairly objective and comprehensive) summary of it:

https://www.skynettoday.com/briefs/openai-rubiks-cube

Would welcome feedback! The hope is that people without much knowledge of AI (but enough interest to read an article of this length) could stumble upon it and gain some clarity about the significance of OpenAI’s latest research.

I need to develop a small project for a data science bootcamp interview next week (insightdatascience.com).

One idea I had was to create an art valuation bot and art generation bot. The outline would look something like this:

1. Scrape abstract art data (image and price data) from an art sales website.
2. Use a pretrained network and fine-tune it on my dataset for price regression.
3. Greate a GAN on the same dataset, then uses it to generate novel art images.
4. Evaluate GAN with the price regression model to determine the newly generated pieces with the highest predicted value.

Some potential challenges I see:

1. Image size: Resizing the dataset to have a standard resolution will really affect the appearance of the newly generated images. Can I do without this? Perhaps I will resize while maintaining the original aspect ratio.
2. Noise: Given that I don’t have access to an art valuation dataset and am just trying to train based on an art sales website, the price does not necessarily indicate that the art piece is valuable. Hopefully, there is enough signal in the dataset to offset this effect.

Does anyone have thoughts on project-based data science boot camps like Insight Data Science (https://www.insightdatascience.com/ )

If you see any more potential issues with my project outline, or have a suggestion for improving it, please leave a comment or PM 🙂

The company TVEyes says it is now transcribing 1,500 hours of podcasts every day (press release). TVEyes clients receive alerts when they are mentioned in one of the 13,000 most popular podcasts.

I’ve never worked with audio, so I’m curious how impressive it is to be able to process 62.5 hours of spoken audio per hour. Is this something that could be done with an off-the-shelf algorithm and a $5,000 server, or are we talking about PhD-level expertise and massive computing power?