These are my questions:

1. 25,000 games are played against itself. After each game, is the MCTS reset for the next game, or is it kept?

2. Does each neural network consist of 1000 batches of 2048 game positions, or can it have more than 2,048,000 inputs?

3. After a new neural network is chosen, is the MCTS reset (thrown away) or kept? Are parts of it reset? Like all of the nodes in the MCTS have P’s from the prior neural network, so do these get recalculated? Are the W’s kept the same even though they were all calculated using V’s from the previous neural network?

Thanks.

Hi,

Does anybody know whether the 40G WebText for GPT-2 contains lots of patents? As early as the 36th step of fine-tuning, GPT-2 can start generating patent-like text correctly with three special tags (“<|startoftext|>”, “<|endoftext|>”, “@@@”) in our training data. It is really unreasonably effective. Anybody in similar situation during fine-tuning?

Available on web: (1) the first 100 steps of fine-tuning, (2) 1000 generated patent claims.

Basic specs are (up to) 28 CPU cores @ 2.5GHz, 1.5TB memory, and dual Radeon Pro Vega IIs.

Full specs here.

I haven’t seen a price listed yet, so I guess considering it based on the appropriateness of the hardware rather than value.

Edit: 3rd party estimates of the price are around $45k for the fully loaded system. Based on price and downsides of AMD vs NVIDIA it seems like a poor choice.

For a long time, I have been using old style spreadsheets to log the results of my experiments with columns like “dataset-version”, “git commit” “PARAMS” “results”…etc

However, I find that this became a pain to consistently fill and update those spreadsheets especially when performing hyper-param search.

There are many frameworks built to manage your ML experiments. I have collected a list below (thanks to the comment here). I would like to know if you have a favourite of the ones below or another

​

SACRED https://github.com/IDSIA/sacred

Studiohttps://github.com/studioml/studio
Datmohttps://github.com/datmo/datmo
Lorehttps://github.com/instacart/lore

FORGE https://github.com/akosiorek/forge
Sumatra https://pythonhosted.org/Sumatra/
RandOpthttps://github.com/seba-1511/randopt
Pachydermhttps://github.com/pachyderm/pachyderm
feature Forgehttps://github.com/machinalis/featureforge
Model Chimphttps://github.com/ModelChimp/modelchimp
PolyAxonhttps://github.com/polyaxon/polyaxon
Kubeflowhttps://github.com/kubeflow/kubeflow
Weights and Biaseshttps://www.wandb.com/

​

ps: This maybe a repost but it is worth revisiting this topic since new frameworks are out.

Are there any open source projects (or free products) that I can use to combine Deep Learning with Distributed Computing? Because I recently discovered what serverless computing is and I realized that I needed a cloud provider, so the first thing that I thought of was Apache’s products, like MXNet, Hadoop and Spark but I don’t know if they’re as useful as serverless computing, given that serverless uses a more dynamic approach.

P.S I’m aiming to built one for an Image Recognition algorithm

Dear community,

I am right now pursuing a PhD at the intersection of molecular science and machine learning during which I am researching new advanced sampling techniques and generative models for molecular structures respecting physical constraints.

While there are plenty of opportunities to just stay in academia I m curious about companies who do applied research in related fields, like searching for new materials using computational methods (in particular modern machine learning techniques) or designing functional proteins using such techniques.

I am personally very interested to eventually work on something “real” that exists in the physical world (e.g. designing a new catalysis membrane for waste decomposition vs building new image filters for Snapchat). But besides current endeavors in rather pure research e.g. as done by DeepMind (AlphaFold) or Google AS I don’t know smaller/less known companies with maybe much more focus on real applications. There are obviously pharmaceutical firms also starting to work in this field, but right now I am more curious about possible other applications.

Do you have any ideas or suggestions what companies might be worth to look at?

Thanks a lot!

We propose an evaluation framework for predictive uncertainty estimation that is specifically designed to test the robustness required in real-world computer vision applications. Using the proposed framework, we perform an extensive comparison of the popular ensembling and MC-dropout methods on the tasks of depth completion and street-scene semantic segmentation. Our comparison suggests that ensembling consistently provides more reliable uncertainty estimates.

arXiv: https://arxiv.org/abs/1906.01620

Code: https://github.com/fregu856/evaluating_bdl

Video: https://youtu.be/CabPVqtzsOI

Project page: http://www.fregu856.com/publication/evaluating_bdl/

https://i.ytimg.com/vi/OAb_p-SXSeM/maxresdefault.jpg

If input/output layers get bigger, the range [-x,x] and standard deviation decreases, wouldn’t this make the initialized nodes be more similar? Wouldn’t it make sense to spread out the initialization of many nodes instead of making them less spread out?