I’ve been asked to convert a fully-trained CNN to a simple FC network with fixed architecture (it’ll be used on a small chip if I remember correctly). They understand the classification performance will drop but it needs to be done anyway. I’ve set up the student network such that it just takes the flattened image as the input but I’m unsure what my targets are. I have the data the teacher network was trained on so I guess I can train the student using those inputs with the correspoding teacher output (rather than one-hot targets in the dataset). But my real question is can I just generate random input images and use whatever the teacher outputs as a target for the student to train on? Is that what is usually done to generate a lot of training data for the student network?

Hi, I’m trying to implement ‘Born Again Neural Networks(BAN)’ by T.Furlanello(https://arxiv.org/abs/1805.04770), and I have some questions. Can anybody help with this please?

If you have read some papers on Knowledge Distillation, you would know that some papers released before BAN used so called temperature. In those papers, the logits were divided with temperature(usually positive integer) and then these outputs were gone through within softmax.

In BAN paper, however, the authour didn’t mention on temperature. So I didn’t stabilize the logits (in other words, I set temperature as 1), but I found that I failed. There were no dramatic difference between Original network and Distilled Network. I guess if i just set temperature as 1 and if the networks overfits, the output distribution wouldn’t provide a meaningful ‘dark knowledge’..

for example, there would no difference between [1, 0, 0, 0,0] and [0.999, 0.000 …, 0.000 …, 0.000 …, 0.000 …]

so.. do i have to apply temperature even though the author didn’t mention on the paper? I’m wondering if I can succeed without applying temperature. Thank you for reading.

​

forewarning I am out of my element with this idea

I’m building a lichtenberg machine DIY from a microwave power transformer, i was wondering if its possible to hook up like an Arduino and a web cam or some type of camera and use machine learning to teach the ai lighting patterns and eventually get it to create its own patterns based off what i feed it Litchenberg wise. now please don’t flame me im not an expert on anything AI so maybe Arduino isn’t the way to go. preferably would like to do this in Python. if there’s any resource or constructive input id be willing to read.

I found this article published recently in Harvard Data Science Review by Michael Jordan (the academic) a joyful read. Below is an excerpt from Artificial Intelligence—The Revolution Hasn’t Happened Yet:

Most of what is labeled AI today, particularly in the public sphere, is actually machine learning (ML), a term in use for the past several decades. ML is an algorithmic field that blends ideas from statistics, computer science and many other disciplines to design algorithms that process data, make predictions, and help make decisions. In terms of impact on the real world, ML is the real thing, and not just recently. Indeed, that ML would grow into massive industrial relevance was already clear in the early 1990s, and by the turn of the century forward-looking companies such as Amazon were already using ML throughout their business, solving mission-critical, back-end problems in fraud detection and supply-chain prediction, and building innovative consumer-facing services such as recommendation systems. As datasets and computing resources grew rapidly over the ensuing two decades, it became clear that ML would soon power not only Amazon but essentially any company in which decisions could be tied to large-scale data. New business models would emerge. The phrase ‘data science’ emerged to refer to this phenomenon, reflecting both the need of ML algorithms experts to partner with database and distributed-systems experts to build scalable, robust ML systems, as well as reflecting the larger social and environmental scope of the resulting systems.This confluence of ideas and technology trends has been rebranded as ‘AI’ over the past few years. This rebranding deserves some scrutiny.

https://hdsr.mitpress.mit.edu/pub/wot7mkc1/

Hi all, I published a summary of a recent paper published by researchers from MIT, University of Toronto and Cambridge. It takes a deeper look into the Lottery Ticket Hypothesis, an idea that neural networks have a pruned version which can train just as well as the original version with significantly less size, when applying the right weights (this is a “winning ticket”). The result could help better understand “winning tickets” and is generally interesting. I hope you’ll like it and happy to get feedback. Full summary here: https://www.lyrn.ai/2019/07/02/exploring-the-lottery-ticket-hypothesis/

Tacotron and derivative models described in recent papers often have appendixes with short speech samples that appear to have crossed the threshold of sounding perfectly human. However, those samples are only a few seconds in duration so I wonder if any attempts were made to use those models in long form text transcription and how those would rate against a proper human narrator.