I called for “call for contributions” recently, but it didn’t end well. People were too obsessed with keeping their secrets and know little outside of ML.

So I searched myself for challenging problems in science, with high meaningful impact, potential for ML to make breakthrough, ready dataset and benchmark, and I found this ProteinNet for protein folding. These scientists seem to think for the sake of science as a whole, and want to see how ML can help advance their field. You are welcome to use it for your side project if you are already tired of old time CV or NLP tutorials.

So this is more of an exploratory question. I am deploying models using a TF serving docker image with the flag TF_FORCE_GPU_ALLOW_GROWTH. I am deploying a small fashion mnist model, resnet (99MB), and inception v3(92MB) models. Because of the flag, the tf model server initially occupies only ~300 MB approx, then on sequential requests to the models it increases as follows (according to nvidia-smi):

~300 MB | after inception request ~4306MB | after resnet request ~ 8402 MB

if I send a request to resnet first, the GPU usage does not increase at all (Even when I add more models):

~300 MB | after resnet request ~7888MB | after inception request ~ 7888 MB

Why does the GPU usage not increase after adding more models? Are they flushed from memory when new models are loaded for inference? How can I accurately estimate how many similar sized models can be loaded on one GPU enabled machine without the trial and error method? Is there a pattern to what fraction of GPU memory is progressively allocated?

Note: This is run on an EC2 instance with available GPU memory 11441MiB [ Tesla K80 ] when I trey to run the same on a machine with lower capacity [Quadro P2000 – 5059 MB], I face a similar situation where there is no increase in memory usage. However, I also get the following in the logs:

2019-12-11 05:10:54.727985: W external/org_tensorflow/tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.25GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2019-12-11 05:10:54.736610: W external/org_tensorflow/tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.26GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available 

I’ve yet to witness a solution that containerizes my laptop environment and replicates my experiment in the cloud. Suppose I run a jupyter experiment on my laptop. Now with a modicum of fiddling I want to fire off 10 instances of the same experiment in AWS with different learning rates (and have my results neatly collated, perhaps in git branches).

I acknowledge the efforts of MLflow, Kubeflow, Comet.ml, deepkit, guide.ai, dvc, sacred, speedrun, and trains but none of these at first glance addresses basic replication in the cloud. Moreover several of them like Kubeflow are prohibitively complex for academics on a budget.

Requesting comments and/or sympathies.

I was wondering if there are any papers examining the matching of distributions in a cycleGAN setting for other data distributions such as financial data

Where do I even begin this rant?

I am a machine learning intern. We have a labelling problem in which we want to classify strings into category “Something” and category “Not Something”. These are not sentences so we can’t use any standard NLP library. My boss is convinced we should turn these strings into ASCII codes, in order to make them “non categorical”, with each feature being the ASCII code for the character in question.

I tried to gently assert that even though they’re numbers, that doesn’t mean that they’re quantitative data – is the average of B and D, C? (He answered yes to that, btw.).

I told him if the word ‘apple’ appears in the beginning of the string and in the other row appears in the end of the string, it won’t be put in the same cluster necessarily. He says the SVM will pick up the pattern – say you have for features 0, 1 and 2 the values 65, 112 and 112 and in another row for features 10, 11 and 1 the same values, the SVM will “detect the pattern” and put them closer together. “That’s not how support vector machines work.” “Oh really, how many have you done?”

I ran it anyway – it gives results with 98% accuracy because in this case “Something” and “Not Something” tend to have radically different lengths. To show him it doesn’t detect patterns, I put a bunch of zeros behind the string and it obviously did not correctly recognise the label. He says that doesn’t prove anything, it’s just a “vulnerability”.

I am at a loss here. Does anyone have a source I can share with him? Or an alternative way of solving my problem?