I have stumbled across the NIPS paper from Olson, Wyner and Berk on decomposing deep neural networks into decorrelated sub-networks to explain how neural networks generalise well (as de facto ensembles) also on small data sets.

Since I am trying to get more acquainted with tensorflow and pytorch, I tried to implement the decomposition strategy presented, however, am running into some issues obtaining convergence results similar to the ones presented in the paper. I am new to this subreddit and hope that maybe someone could point me to issues in my replication efforts.

​

The authors use the following network architectures they subsequently decompose into subnetworks via linear programming:

– 10 hidden layers (with elu activation and he initialization) with 100 each

– binary classification task

– Adam optimizer with learning rated of 0.001 and 200 training epochs

​

I am using the following code to simulate data similar to the synthetic data set used in the paper:

​

np.random.seed(0)

def classify(x):

if np.linalg.norm(x) < 0.6: # the paper mentions a radius of 0.3, the images look like 0.6 though

return 1

else:

return 1 if np.random.random() < 0.15 else 0

x_ = np.linspace(-1, 1, 20)

X_train = np.asarray([(x_[i], x_[j]) for i in range(20) for j in range(20)], dtype=np.float32)

y_train = np.apply_along_axis(classify, 1, X_train).reshape(400, 1)

​

Then I have set up the network architecture as follows:

​

X_input = tf.placeholder(tf.float32, [None, 2], name='input_data')

y = tf.placeholder(tf.float32, [None, 1])

​

X = tf.layers.dense(X_input, units=M, kernel_initializer=tf.contrib.layers.variance_scaling_initializer(), activation=tf.nn.elu, name='dense_0')

for i in range(1, L):

X = tf.layers.dense(X, units=M, kernel_initializer=tf.contrib.layers.variance_scaling_initializer(), activation=tf.nn.elu, name='dense_' + str(int(i)))

​

logits = tf.layers.dense(X, units=1, activation=None, name='logits')

predicted = tf.nn.sigmoid(logits, name='predicted')

is_correct = tf.equal(tf.round(predicted), y, name='is_prediction_correct')

accuracy = tf.reduce_mean(tf.cast(is_correct, tf.float32))

​

cross_entropy = tf.nn.sigmoid_cross_entropy_with_logits(labels=y, logits=logits)

l = tf.reduce_mean(cross_entropy)

optimizer = tf.train.AdamOptimizer(learning_rate=0.001).minimize(l)

​

To my understanding this should match the implementation details of the paper, however, while the paper limits the training epochs to 200 and even mentions, that

“In practice, we found that networks without dropout achieved 100% training accuracy after a couple dozen epochs of training.”

my implementation only converges after 500+ epochs at best.

Does anyone here have ideas about what I could be doing wrong?

I’m looking for up to date datasets to benchmark various algorithms against the performance (both speed and accuracy) of published models.

​

I’ve found some dataset but the main issue is that they are either:

​

a) very old and small, e.g. most datasets hosted by UCI, which are rather “easy” to “solve” nowadays and most papers using them came out decades ago. Even barring that, a lot of the papers dealing with the data are not ideal for benchmarks per-say because they are not very specific in their methodology for splitting into train/test/validate.

​

OR

​

b) They are focused on images, e.g. cifrar 100 is pretty decent, and there are loads of high quality models with known accuracy and available source code… but, I can’t find the equivalent of cifrar 100 for, say, financial timeseries prediction, or STT, or geospatial movement predictions for cars… or any problem other than image classification -_-

​

Are there any well maintained list of datasets that specifically have various models benchmarked against them ? Or would it be better to just do reverse-search on this problem, as in, look for interesting papers that came out in the last few years and use the datasets they used.

As a data scientist, there are a ton of things you have to do every day. Which ones do you look forward to? And which tasks do you usually dread working on?

I have some thoughts I’d like to share on some of the challenges I face in my day in day out work.

No, I won’t be writing another “Gentle Introduction/ blah, blah, demystified” type post, but what I feel will be some serious and original insights into my particular corner of the ML/DS world.

What is a good blog or online publication that would accept my posts. I am OK with a review process, but one time I had to wait for 4 months before my post was reviewed and published (almost as long as an academic paper…), so I’m hoping a place that would publish the posts relatively quickly.

​

Any suggestions? Please do not say Towardsdatacience, the quality of that blog has plummeted in the last year.

Hi all, I’d like to share my undergraduate thesis on Mental Simulation with Self-Supervised Spatiotemporal Learning. We propose that one way to understand mental simulation in humans is to approach it as a problem of video prediction. The code is based off the recent ICLR 2019 paper Eidetic 3D LSTM: A Model for Video Prediction and Beyond.

Paper: https://github.com/kevinstan/video_prediction/blob/master/paper/mental_sim.pdf

Code: https://github.com/kevinstan/video_prediction

Abstract: Mental simulation — the capacity to imagine objects and scenes in order to make decisions, predictions, and inferences about the world — is a key feature of human cognition. Evidence from behavioral studies suggest that representations of visual imagery are spatial and sensitive to the causal structure of the world. Inspired by how humans anticipate future scenes, we aim to leverage state-of-the-art techniques in deep learning and computer vision to tackle the problem of spatiotemporal predictive learning in a self-supervised manner. We perform explorations across three architectural design choices: (i) the importance of 2D-convolution vs. 3D-convolution inside the cell of recurrent neural networks, (ii) the effectiveness of residual connections in stacked long short-term memory models for remembering spatial information over long time horizons, and (iii) the balance between $l_1$ norm and $l_2$ norm components in the objective function. Our extensive evaluations demonstrate that finetuning with residual connections achieves state-of-the-art performance on the Moving MNIST and KTH Action benchmark datasets. Potential application areas include weather forecasting, traffic flow prediction, and physical interaction simulation.

​

Comments and feedback are highly appreciated. Thanks!

Hi all!

​

I want to wish you a good evening and would like to share with you some tool I developed for my work in the particle physics community.

​

Probably we are not the only ones who like to prototype and train ML models with xgboost in Python and then have to deploy them to our high performance multithreading C++ production environment. This is exactly the step that my tool which I call “FastForest” wants to make easy like a breeze!

​

The mission of the library is to be:

• Easy: deploying your xgboost model should be as painless as it can be
• Fast: thanks to efficient structure-of-array data structures for storing the trees, this library goes very easy on your CPU and memory
• Safe: the FastForest objects are immutable, and therefore they are an excellent choice in multithreading environments
• Portable: FastForest has no dependency other than the C++ standard library

​

I hope that this might be of use for someone else too 🙂 I think it might, because so far, the only solutions to this C++ deployment problem that I found on the web are either to use the sparsely documented xgboost C API or to transform your models into hardcoded C++ (which is cool as well of course). What I tried to write here is a very lightweight and clean C++ solution which can load models dynamically.

​

Check it out here: https://github.com/guitargeek/XGBoost-FastForest