Follow and add your project to r/datascienceproject

Examples:

Predicting Collegiate Basketball Win Shares from ESPN Ratings and High School Statistics

Impact of training hours on employee performance

Any data science applications in Physics?

Hi all! I’m curious about the best ways to manage large image and video datasets for computer vision projects.

I’m an ML engineer on a team of ~10, supported by 5 data labellers.

I was wondering how other teams in CV space manage:

-Storing the datasets in a central (hosted?) location, and version controlling them as needed, with minimal overhead

-Allowing for querying and visual exploration of the datasets for quick adjustment or examination of labels

-Efficiently pulling a dataset or subset of a dataset to a local machine.

-Automating the flow of datasets as much as possible, i.e. train x model on y subset of z dataset.

-Compressing less frequently used data as much as possible for “cold storage” and handling uncompression/recompression when the data is needed for training or when new data is added

​

So far we’ve used 3 solutions:

1. Storing everything on a local machine sitting under an unoccupied desk, and everybody manually updated the data there
2. Storing compressed tar files of the data on AWS storage and retrieving/updating it manually every so often.
3. Assigning one of the data labellers to spend some time as a “dataset manager” and try to do this for us.

Each of these has had its own set of problems, and I feel I waste a lot of time dealing with the overhead of this stuff.

How do you guys deal with this situation? Is there an “industry standard” correct way of managing this stuff? Like a github for CV? At places like Waymo/Tesla for example where they are constantly growing and updating their dataset to improve weak points, I would think an elegant solution for this has been devised.

One caveat is that I’d like to avoid using things like AWS and Azure ML “low code” services that might do some data management for you but then take away most of the freedom of working in TF/Pytorch, and make the model into a black box.

Today, the Chinese firm reached a major milestone in its AI roadmap, announcing Ascend 910, the “world’s most powerful AI processor”, and MindSpore, an AI computing framework. With this launch, the firm has unveiled all the key components of its full-stack, all-scenario AI portfolio.

Speaking at Huawei’s headquarters in Shenzhen, Eric Xu, the company’s rotating chairman, spoke regarding the new products, noting:

“We have been making steady progress since we announced our AI strategy in October last year. Everything is moving forward according to plan, from R&D to product launch. We promised a full-stack, all-scenario AI portfolio. And today we delivered, with the release of Ascend 910 and MindSpore. This also marks a new stage in Huawei’s AI strategy.”

The Ascend 910, in conjunction with the MindSpore framework, is used to train AI models. Surpassing even Huawei’s own expectations in terms of its performance, the new processor delivers 256 TeraFLOPS of computing speed for half-precision floating point (FP16) operations, and 512 TeraFLOPS for integer precision calculations (INT8). Its maximum power consumption is 310W, lower than the initially planned 350W, despite being around two times faster in training AI models based on standard deep neural networks like ResNet-50. The new AI processor belongs to Huawei’s series of Ascend-Max chipsets.

Similarly, with the release of MindSpore, Huawei believes it has furthered its AI framework development goals. These comprised of reduction in training times and costs, efficient execution, and more adaptability. Aside from fulfilling these requirements, MindSpore also puts privacy protection and security at the forefront. To clarify this a bit further, the new AI framework deals only with processed information, not actual user data. It also has built-in model protection tech to ensure that the AI models being utilized are trustworthy.

Furthermore, with regards to actual performance, MindSpore has 20% fewer lines of code than other frameworks in a typical neural network for natural language processing (NLP). Combined with its “AI Algorithm As Code” design concept, the product provides a high degree of adaptability and efficiency. Aside from Ascend processors, support is also offered for GPUs, CPUs, and other types of processors. Importantly, Xu also offered some insight into the framework’s future, noting that “MindSpore will go open source in the first quarter of 2020. We want to drive broader AI adoption and help developers do what they do best.”

Despite the new unveilings, Huawei isn’t slowing down its pursuit of AI dominance, and will be revealing more such products in its upcoming Connect 2019 conference, scheduled to be held between September 18 and 20 in Shanghai, China.

https://www.neowin.net/news/huawei-launches-ascend-910-the-world039s-most-powerful-ai-processor-and-mindspore/

What’s up redditors, wanted to share a video I made using pytorch to make art, produced via machine learning algorithms. With the use of a green screen and neural networks, I was able to style transfer actual art onto the singer. If you have any questions I would be more than glad to answer, let me know what you think!

Video: https://www.youtube.com/watch?v=TW99ygvKFq4

I’ve implemented SVRG (Stochastic Variance Reduction Gradient Descent) optimizer for Keras. The goal is to make this optimizer available in Keras as well, which may be beneficial in the case of RL as some papers claimed it is advantageous over Adam.

The paper: https://papers.nips.cc/paper/4937-accelerating-stochastic-gradient-descent-using-predictive-variance-reduction.pdfLink to the project: https://github.com/tilkb/SVRGoptimizerKeras

The goal of the Shared Visual Representations in Human and Machine Intelligence (SVRHM) workshop at NeurIPS 2019 is to discuss and disseminate relevant findings and parallels between the computational neuro/cognitive science and machine learning/artificial intelligence communities.

In the past few years, machine learning tools — especially deep neural networks — have permeated the vision/cognitive/neuro science communities to become the leading computational models that describe many cognitive tasks. Huge strides are also being made on the machine learning/artificial intelligence community with biologically inspired algorithms providing large efficiency gains in both computational and learning capabilities. However, many mysteries remain with regards to the alignment of human and machine perception, and there are cases where we see divergent rather than convergent representations. To resolve such questions, this workshop aims to bring fruitful discussions between scientists and engineers with multi-disciplinary backgrounds to review the recent progress in shared visual representations in both humans and machines, and in doing so identifying road-blocks and areas of interest to further accelerate the growth of both fields.

The workshop will include a series of talks and panel discussions from a diverse group of speakers from both industry and academia who will share their research at the intersection of humans and machines that pushes the field of vision forward. The aim of our Call for Papers is to bring together scientists and engineers to share their work in progress at the Poster Session that are applicable to the scope of the Workshop.

The following areas provide a sense of suitable topics for 2-4 page paper submissions:

• Biological inspiration and inductive bias in vision
• Human-relevant strategies for robustness and generalization
• New datasets (e.g., for comparing humans/animals and machines)
• Biologically-driven self-supervision
• Perceptual invariance and metamerism
• Biologically-informed strategies to mitigate adversarial vulnerability
• Foveation, active perception, and attention models
• Intuitive physics
• Perceptual and cognitive robustness
• Nuances and noise in perceptual and cognitive systems
• Creative problem-solving
• Differences and similarities between humans and deep neural networks
• Canonical computations in biological and artificial systems
• Alternative architectures for deep neural networks
• Reverse engineering of the human visual system via deep neural networks

We will be awarding an NVIDIA Titan RTX and an Oculus Quest as best paper and poster prize respectively at the conference.

Link to the workshop with additional details for the Call for Papers: https://www.svrhm2019.com/

Link to Paper workshop submission: https://cmt3.research.microsoft.com/SVRHM2019

Questions regarding the workshop should be sent to: [info@svrhm2019.com](mailto:info@svrhm2019.com)📷

Sincerely,

The Organizers

Arturo Deza, Joshua Peterson, Apruva Ratan Murty, Tom Griffiths

The SVRHM workshop is currently sponsored by NVIDIA, MIT’s Center from Brains, Minds and Machines (CBMM), National Science Foundation (NSF), Oculus and MIT’s Quest for Intelligence.

I was reading about logarithmic loss and I became curious about a few things, so I did an exploratory analysis using a “good” model and a “bad” model.

Some things I looked at include: distribution of log loss, mean vs. median and calculating it separately for the target and non-target classes.

I’d like to know if anyone has any thoughts or ideas to discuss regarding some of the more nuanced aspects of the metric.

https://emilyswebber.github.io/LogLoss/

I am currently working on a project with an interesting task that I am struggling with an approach. I am tasked to implement ML algorithms to replace fuzzy matching for MDM purposes. The fields will include PII such as SSN, First Name, Last Name, Address etc. I am quite new to Fuzzy Matching would love to hear some opinions on some other approaches to this problem! The development language for Python (I did work through FuzzyWuzzy and I believe the current implementation uses something similar to this and the similarity score)