Abstract

Deep learning models frequently trade handcrafted features for deep features learned with much less human intervention using gradient descent. While this paradigm has been enormously successful, deep networks are often difficult to train and performance can depend crucially on the initial choice of parameters. In this work, we introduce an algorithm called MetaInit as a step towards automating the search for good initializations using meta-learning. Our approach is based on a hypothesis that good initializations make gradient descent easier by starting in regions that look locally linear with minimal second order effects. We formalize this notion via a quantity that we call the gradient quotient, which can be computed with any architecture or dataset. MetaInit minimizes this quantity efficiently by using gradient descent to tune the norms of the initial weight matrices. We conduct experiments on plain and residual networks and show that the algorithm can automatically recover from a class of bad initializations. MetaInit allows us to train networks and achieve performance competitive with the state-of-the-art without batch normalization or residual connections. In particular, we find that this approach outperforms normalization for networks without skip connections on CIFAR-10 and can scale to Resnet-50 models on Imagenet.

https://papers.nips.cc/paper/9427-metainit-initializing-learning-by-learning-to-initialize

Colab and similar products can help with the reproducibility of code and more importantly the code underlying academic results. These free resources not only make code transparency easier, from here forward, it makes unpublished Python code highly suspect. There are no limitations to sharing code and data anymore and no limitation in accessing this code, the data and the necessary processing power to analyse the results.

from medium

Recently, I did a little research in the literature for “Topic segmentation” since “Text segmentation” seems to be more related to identifying text in images. From the results, it appears that the most recent survey is from 2011 [1], while the most recent papers in big conferences are from 2008 to 2013 [2, 3, 4].

​

Is this the current state of the problem, or there are more recent and relevant works?

​

It’s also possible that I’m using the wrong terms. So, for clarification, I’m most interest in segmenting a collection of documents in a small and well-known number of sections / topics.

​

[1] Purver, Matthew. “Topic segmentation.” In Spoken language understanding: systems for extracting semantic information from speech (2011)

[2] Eisenstein, Jacob, and Regina Barzilay. “Bayesian unsupervised topic segmentation.” In Proceedings of the Conference on Empirical Methods in Natural Language Processing (2008)

[3] Riedl, Martin, and Chris Biemann. “TopicTiling: a text segmentation algorithm based on LDA.” In Proceedings of ACL 2012 Student Research Workshop (2012)

[4] Du, Lan, Wray Buntine, and Mark Johnson. “Topic segmentation with a structured topic model.” In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics (2013)

^same as title. Any favorites?

A look at 17 of the most popular projects, research papers, demos, and more from this subreddit

https://heartbeat.fritz.ai/best-of-machine-learning-in-2019-reddit-edition-5fbb676a808

Do very simple text-preprocessing (a.k.a dirty work) with PreNLP Package !

I’m working in NLP part, and implementing a package to do iterative but necessary works for NLP. Here are some exmaples to preprocess text for following NLP tasks.

​

• Frequently used normalization functions for text pre-processing are provided in prenlp. General use cases are as follows:

​

>>> from prenlp.data import Normalizer >>> normalizer = Normalizer() >>> normalizer.normalize('Visit this link for more details: https://github.com/') Visit this link for more details: [URL] >>> normalizer.normalize('Use HTML with the desired attributes: <img src="cat.jpg" height="100" />') Use HTML with the desired attributes: [TAG] >>> normalizer.normalize('Hello 🤩, I love you 💓 !') Hello [EMOJI], I love you [EMOJI] ! >>> normalizer.normalize('Contact me at lyeoni.g@gmail.com') Contact me at [EMAIL] >>> normalizer.normalize('Call +82 10-1234-5678') Call [TEL] 

• Quick tour to Text classification : The following example code trains fastText classification model on IMDB. The code below has only 16 lines of code (except blank lines and comments).

​

import fasttext import prenlp from prenlp.data import Normalizer from prenlp.tokenizer import NLTKMosesTokenizer # Data Preparation imdb_train, imdb_test = prenlp.data.IMDB() # Preprocessing tokenizer = NLTKMosesTokenizer() normalizer = Normalizer(url_repl=' ', tag_repl=' ', emoji_repl=' ', email_repl=' ', tel_repl=' ') for dataset in [imdb_train, imdb_test]: for i, (text, label) in enumerate(dataset): dataset[i][0] = ' '.join(tokenizer(normalizer.normalize(text.strip()))) # both prenlp.data.fasttext_transform(imdb_train, 'imdb.train') prenlp.data.fasttext_transform(imdb_test, 'imdb.test') # Train model = fasttext.train_supervised(input='imdb.train', epoch=20) # Evaluate print(model.test('imdb.train')) print(model.test('imdb.test')) # Inference print(model.predict(imdb_test[0][0])) 

For more details, follows: https://github.com/lyeoni/prenlp

p.s. And I want to know what you want to implement on this issue. I’ll implement that on this package.