Table of Contents

A simple collection of notes from my explorations of Kubeflow cluster.

An example to show how we can use Huggingface Roberta Model for fine-tuning a classification task starting from a pre-trained model. The task involves binary classification of smiles representation of molecules.

FullNotebook for this post

Full Notebook

Adapted from https://d2l.ai/chapter_recurrent-modern/seq2seq.html

Create a simple GRU layer using pytorch. Feed a tensor of shape batch_sizexnum_stepsxinput_size and observe the GRU output. Next feed the same input tensor one time-step at a time ensuring that the previous timestep hidden state becomes initial state for the current timestep. The outputs should be same.

CROSS ENTROPY LOSS (logits, class) = negative log of softmax of logits[class]

```python from future import print_function

First use nn.GRU with 3 layers for processing sequences. Then use nn.GRUCell for doing the same.

Implements simple character level name classification using Pytorch. Training is done using about 20K names across 18 languages. The names are clubbed into three categories : English, Russian, Other for simplicity. Using SGD as optimizer produces poor results, Adam performs better, Nadam even better.

Implements simple character level name classification using Keras LSTM and Dense layers. Training is done using about 20K names across 18 languages. The names are clubbed into three categories : English, Russian, Other for simplicity. Using SGD as optimizer produces poor results, Adam performs better, Nadam even better.

Simple Generative Adversarial Network to generate datapoints from a simple one-dimensional function (adapted from https://machinelearningmastery.com/how-to-develop-a-generative-adversarial-network-for-a-1-dimensional-function-from-scratch-in-keras/).

```python from future import absolute_import from future import division from future import print_function import os import numpy as np import itertools

Given the smiles of a molecule and the smiles of a possible substructure, find the atoms of the substructure in the molecule. Visualize the molecule with the substructure atoms highlighted in green.

You will need gpu and cuda with pytorch. Data used in the code

Tensorflow implementation of End-To-End Memory Networks for the language modeling task. I tried to name the variable as closely as possible to that in the paper following the equations to help understand the paper. Don’t forget to change the “input_file” to your input file. Some of the ideas are borrowed from earlier implementation. The python notebook of the code can be found here.

Deepchem provides a wonderful framework and library for developing deep learning and machine learning predictive models for small molecules. However, its understandably complex pythonic architecure and equally inexplicable lack of documentation (except the raw python function descriptions and a handful of tutorials) make it very hard to get benath the surface and engineer it to fit your own needs, particularly so if you are not a physics, chemisty and deep learning and programming major. Here I will chronicle my efforts to just enable training with deepchem where we will be feeding the data using feed_dict (people who uses tensorflow will understand this term) to the tensorflow graph, not using the standard fit_generator or fit functions of deepchem (which kind of makes deepchem a blackbox difficult to understand).

This writeup shows how to use graph convolutions for a regression like problem using the DeepChem library.

This post shows how to implement a simple graph convolutional deep learning method to predict interfaces between protein residues, i.e. given a pair of interacting proteins, can we classify a pair of amino acid residues as interacting or not. This is based on the paper published in NIPS 2017 (Protein Interface Prediction using Graph Convolutional Networks).

Input test data

Given a piece of text, NER seeks to identify named entities in text and classify them into various categories such as names of persons, organizations, locations, expressions of times, quantities, percentages, etc. Here we just want to build a model to predict \(N_c =\) 5 classes for every word in a sentence: PER (person), ORG (organization), LOC (location), MISC (miscellaneous) and O(null class, not a NER).

A simple example demonstrating the use of TensorArray to create a recursive neural network, essentially a tree structured neural network.

A small example on how to use Tensorflow TensorArray.

A lot of materials on word2vec models such as Skipgram and CBOW are available that explain the models really well. This post is just a drop in that ocean trying to clarify some of the details that I found useful in understanding the internals and explaining the models in line with the (almost the same) terminology used in the NLP lectures CS224n.Two other resources that I find very useful are word2vec Parameter Learning Explained and word2vec Explained: deriving Mikolov et al.’s negative-sampling word-embedding method.

Example of solving a parameterized model with Tensorflow - define the logistic regression with multiple classes to predict.

This post develops on the Ising model concepts from my previous blog, see Ising model. Consider the problem of reconstructing a black-and-white image (i.e. each pixel is either 1 or -1) from the corrupted observations. We assume that there is an underlying (hidden) noise-free image from which the observed image (Figure 1) is generated by adding noise: randomly flip the pixel values with a small probability (say 20%). Given the observed noisy image (modeled by random variable \(\textbf{Y}\)), we want to recover the original noise-free image (modeled by random variable \(\textbf{X}\).

Please refer to my earlier posts on MRF and GRF for getting used to the notations. If we consider clique potentials for sizes of upto 2, the energy function is the sum of clique potentials of all cliques of size 1 and size 2,

Lets see what is a GRF and how it is connected to the MRF. If you are new to my blogs, please visit my blog on MRF to get familiar with the notations. A GRF can be thought of as a graphical representation of a family of random variables \( \textbf{X} =\{ X_1,X_2,…,X_n \} \) on set \( S=\{1,2,…,n\} \) of sites.The relationship between the random variables is defined using a neighborhood system. A GRF obey’s Gibb’s distribution given by,

A short intro to MRFs. Let \( \textbf{X} =\{ X_1,X_2,…,X_n \} \) be a family of random variables defined on a set \( S=\{1,2,…,n\} \) of sites. As an example, \( S \) can represent the pixel positions of an \( m \times m \) image in a 2-D lattice \( \{(i,j) | 1 \leq i,j \leq m\} \) where the double indexing can be recoded to univariate indexing by \( (i,j) \rightarrow (i-1)m+j \) so that \( S=\{ 1,2,…,m^2 \} \).

A demonstration of how to use python package sklearn for a basic machine learning task : classification.

Next you can update your site name, avatar and other options using the _config.yml file in the root of your repository (shown below).