Recent posts
Visual Search Engines
Posted on 08 Oct 2016
Visual Search Engines The search engines we normally use are text based, like Google. You type in some text and the search engine retrieves the most relevant documents for that query term. The query text can be in the form of audio, “Siri, tell me about Albert Einstein”, which opens the Wikipedia page of Albert Einstein. Similarly, there can be visual search engines in which the query term is an image and the search engine finds similar looking images, or related images. Google supports visual search currently, but I don’t have the statistics as to what number of people actually...
CMP: Chip Multiprocessors
Posted on 08 Oct 2016
Understanding Sources of Inefficiency in General-Purpose Chips By: Rehan Hameed, Wajahat Qadeer, Megan Wachs, Omid Azizi, Alex Solomatnikov, Benjamin C. Lee, Stephen Richardson, Christos Kozyrakis, and Mark Horowitz Paper Presentations or notes: 1 2 3 4 5 6 Convolution Engine: Balancing Efficiency and Flexibility in Specialized Computing By: Wajahat Qadeer, Rehan Hameed, Ofer Shacham, Preethi Venkatesan, Christos Kozyrakis, and Mark Horowitz Paper Presentations or notes: 1
TensorFlow Playground
Posted on 04 Oct 2016
TensorFlow Playground This post is an effort to understand how neural networks work. The visualizations are images obtained by experiments using TensorFlow Playground. Kudos to TensorFlow for making such an amazing framework! Right now, I have added the experiments that I found the most interesting. Learning is an ongoing process and new interesting insights will be added. Most of these might be obvious to a lot of people, but, I am noting them here because all these nitty grittes are really important in knowing how to fine tune the actual network, or in other words, how the inputs get translated...
Adversarial Autoencoders
Posted on 05 Sep 2016
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, Brendan Frey Autoencoder Are used to optimize the reconstruction of the input at the output layers. So, whatever it got as the input, it wants to produce the same thing as the output. In an autoencoder there is one encoder and one decoder. The encoder takes the input and the decoder gives the output. The encoder encodes the input to some representation, this representation is important because it is fed to the decoder. The decoder upon receiving the encoded representation of the input will decode it back, or try to gauge what...
DenseCap: Fully Convolutional Localization Networks for Dense Captioning
Posted on 30 Jun 2016
Justin Johnson, Andrej Karpathy, Li Fei-Fei This paper brings forth the new idea of dense caption. The dense caption is that every almost every object has a bounding box and its contents are described by a caption. I like the fact that their open world detection task is like image retrieval. Although, intuitively I think that if anyone searches for “white tennis shoes”, they would want the white tennis shoes to be one of major objects of interest in the image rather than the example images below. However, it is necessary to remember that the image dataset here contain non...
Generation and Comprehension of unambiguous object descriptions
Posted on 24 Jun 2016
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan Yuille, Kevin Murphy Spatial positions need bounding box Evaluation of the mid level vectors because it could be an encoding - model will communicate better with itself in its own encoding. CO-Training vs multi view system Multiple models on diff views and then improve iteratively Training is semi supervised in this paper - this is like - bootstrapping? Examples show interesting visual features that can encode ‘behind’. Assumption is that the training dataset must have such words. Hard ground truth is the most confusing caption. Mutual information is used in the...
Dynamic Memory Networks for Visual and Textual Question Answering
Posted on 18 Jun 2016
Caiming Xiong, Stephen Merity, Richard Socher This paper can be thought of as rephrasing the question to go towards the answer, so the memory always represents the question in some way. This idea was said during a paper discussion and I think its a different way to think about QA tasks. I am not sure how are priors used and how are they derived. Training: How many passes for training? The system can also use sentences that it had previously thrown away. What is the intuition behind this? The system allows sentences to interact with other sentences without changing gradient,...
Learning to Execute
Posted on 12 Jun 2016
Wojciech Zaremba, Ilya Sutskever This paper uses RNNs (also called Tree Neural Networks) variant LSTMs to execute small programs by evaluating symbolic mathematical expressions and logical formulas. Addition task These programs are easy to solve by a human however the computer needs to understand what print, +, 12 and other such characters mean, when not in a programming environment. The system has not been taught the operators or anything about their combination beforehand. The above task is addition. Another task is the memorization task in which the order of numbers presented has to be remembered. These programs are limited by...
Curriculum learning
Posted on 12 Jun 2016
Yoshua Bengio, Jerome Louradour, Ronan Collobert, Jason Weston The basic idea is: Start Simple then Simplify When we are being taught, we are given easier examples first and then the level of difficulty is increased. This is a really cool idea. Its something that is so natural in real life and is applied in neural networks. True in biology as well, we are given weak strains of virus so that our platelets can learn to fight against the actual much stronger virus. I think this is the general idea of what a vaccine is, I can surely be wrong though...
Order Embeddings of images and language
Posted on 09 Jun 2016
Ivan Vendrov, Ryan Kiros, Sanja Fidler, Raquel Urtasun All problems relating images and text involve the visual semantic hierarchy and have some partial order over images and langauge. Example: Captions are abstractions of images. If a system is able to learn representations that represent the partial order structure then it should help in solving the image-text problems. A method for learning the ordered representations which is the basis for many tasks involving image and languages. Distributed representations or embeddings are used to model the image-caption relations because they can map text and images to high dimensional vector space. This mapping...
Apache Spark Maker
Posted on 06 Jun 2016
Twitter Feed of Apache Spark Maker - IBM where I was an invited panelist among John Akred, Chief Technology Officer, Silicon Valley Data Science; Todd Holloway, Director of Content Science and Algorithms, Netflix; Matthew Conley - Data Scientist, Tesla Motors; Nick Pentreath, Spark Committer and STC Engineer, IBM; and Dr. Eitel J.M. Lauría, Professor and Graduate Director at the School of CS & Math, Lead Data Scientist of the Learning Analytics, Marist College. Video: IBM+Spark→ links pt. 3• https://t.co/0O2167XNcz• @MLnick @BigDataAnalysis @toddmholloway https://t.co/grUDVrHqdm @EitelLauria @SiddhaGanju— Brian Krent (@BrianKrent) June 7, 2016 .@TeslaMotors data scientist Matt Conley: great data scientists are...
Visual Question Answering Experiments
Posted on 27 May 2016
Toolkit for Visual7W visual question answering dataset python predict_baseline.py --dataset visual7w-telling --mode open --topk 100 --split val --result_path results parsed parameters: { "result_path": "results", "split": "val", "topk": 100, "mode": "open", "dataset": "visual7w-telling" } Initializing data provider for dataset visual7w-telling... BasicDataProvider: reading datasets/visual7w-telling/dataset.json writing predictions to results/result_visual7w-telling_open.json... python evaluate.py --dataset visual7w-telling --mode open --topk 100 --split val --results results/result_visual7w-telling_open.json --verbose 1 Initializing data provider for dataset visual7w-telling... BasicDataProvider: reading datasets/visual7w-telling/dataset.json 2016-05-26 17:10:20,488 Open-ended QA evaluation 2016-05-26 17:10:28,096 Evaluated 10,000 QA pairs... 2016-05-26 17:10:28,655 Evaluated 20,000 QA pairs... 2016-05-26 17:10:29,124 Done! 2016-05-26 17:10:29,124 Evaluated on 28,020 QA pairs with top-100 predictions. 2016-05-26...
Distributed Representations of Words and Phrases and their Compositionality
Posted on 26 May 2016
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, Jeffrey Dean The Skip-Gram model’s greatest advantage is its efficiency in learning high quality vector representations of words from large amounts of unstructured text data. The training objective is to learn a word representation that is good at predicting nearby words. Training of Skip-Gram model does not need dense matrix multiplications which makes the training efficient such that an optimized single machine implementation can train on more than 100 billion words in one day. These learned vectors explicitly encode linguistic patterns and regularities which can also be represented as linear translations. This...
Sequence to Sequence Learning with Neural Networks
Posted on 25 May 2016
Ilya Sutskever, Oriol Vinyals, Quoc V. Le Deep Neural Networks are utilized in an end to end approach to sequence learning while making minimal assumptions about the sequence structure. A multilayered LSTM maps the input sequence to a vector of a fixed dimensionality and a deep LSTM decodes the target sequence from the vector. The LSTM was able to learn phrases and sentence representations that were sensitive to word order and invariant to the voice (active or passive). If the word order of the source sentences is reversed then the LSTM’s performance increases because this introduces short term dependencies between...
The Neural Conversation Model
Posted on 24 May 2016
Oriol Vinyals and Quoc V.Le Neural Networks to map sequences to sequences which in this regard are statements and their responses. The advantage of using neural networks is that the system becomes are simple and general, the same end-to-end system can be used for machine translation, conversations and question answering. The only difference in conversations is that the input sequence is the concatenation of what has been covered so far (the entire context) and the output sequence is the reply. Translation in comparison to conversations is a much easier task. This is again fortified by the fact that the objective...
Visual Question Answering
Posted on 18 May 2016
Does knowing the question help in image understanding? Dataset MS COCO VQA dataset Format Image followed by its three questions-answers pairs. Example Question: What is the mustache made of? Inference: A male is present who has a mustache Truth: Girl is wearing banana mustache Other question: What color are her eyes? (Female with mustache can be inferred from this) Question: Do you like this house? Inference: Outdoor scene showing a house. Truth: No help while answering other questions. Other questions: Who is riding a bike? Is the boy on a skateboard? Observations If the questions are independent then they don’t...
Experimenting with the Neural Conversation Model
Posted on 01 May 2016
These are the experiments for The Neural Conversation Model carried out during Spring 2016, CMU. The (private for now) Github repo is maintained here. This is the model used for the experiments. The datasets have been developed by me. Idea The problem at hand is to model dialogs so that they appear human like. When the system communicates with a person through text, it learns about their knowledge and interests. By using Memory Networks built using LSTMs, a modified form of RNN’s, for the predefined toy tasks T ∈ {“where is” , “what is in”}, we can build a smart...
Deep Learning Summer School
Posted on 19 Apr 2016
whoami Hi! If you are from the Deep Learning Summer School, please check out my CV here. My current projects are: Image Captioning: Working towards developing captions for images using attention models, the results from MS COCO leaderboard will be shortly added. Video Question Answering: Implementing different language models to make the question answering more dynamic. Intelligent and Proactive Dialog (Intelligent Chat Bot): Which can carry context from one dialog exchange to another and can use this information to initiate a new dialog. This is being developed using memory networks and is tested on an indigenous CMU corpus of dialogs...
Amazing AI
Posted on 18 Apr 2016
Till now, all the work that I have done on deep learning or machine learning takes only one kind of input, it’s either text, images, audio or video. But, in real life humans can take in more than one of these as inputs and process their thoughts. Keeping this as my dream and longer term goal, I have started working on multimodal deep learning, which combines the learning’s from various different inputs. This is all towards a greater dream of achieving an Amazing AI. There are several phrases in the development of an Amazing AI system and I have identified...
Strata+Hadoop Conference, San Jose
Posted on 03 Apr 2016
Links: O’Reilly Blog post I about my talk, also featured on their data newsletter. O’Reilly Blog post II, also featured on their data newsletter. O’Reilly Conference details Twitter Data Today features my post "Atom Smashing using #machinelearning at #CERN" https://t.co/1UQWmRhhul Thank you @nicoletache1 @strataconf !!!!— Siddha Ganju (@SiddhaGanju) April 5, 2016 "Spark was the shining lighthouse that showed us the way of realtime analytics" @SiddhaGanju smashtests @ApacheSpark https://t.co/1ShSG2r9N9— Katharine Kearnan (@KatKearnan) April 1, 2016 Yes, the data at @CERN is Big with a capital B. @SiddhaGanju #StrataHadoop pic.twitter.com/XGeBton0Pe— amcasari (@amcasari) March 31, 2016 . @SiddhaGanju taking about the unreal data...
Translating Videos to Natural Language Using Deep RNNs
Posted on 12 Feb 2016
Translating Videos to Natural Language Using Deep RNNs Natural languages and videos have had little history because of limited training data which has the video and its respective descriptive sentences. It has also been hindered by the lack of rich models that can capture the joint dependencies of a sequence of frames and words. This paper proposes a way to translate the video to natural languages with a single deep neural network using transfer knowledge from auxiliary tasks. Each frame is modeled independently by a CNN and the sequence is modeled by a RNN (LSTM). Pre-training on images and text...
Show and Tell - Neural Image Caption Generator
Posted on 11 Feb 2016
Show and Tell: Neural Image Caption Generator Computer Vision and Natural Language processing are connected via problems that generate a caption for a given image. This caption is like the description of the image and must be able to capture the objects in the image and their relation to one another. Then, this caption must be expressed in a semantically correct form in a natural language. This paper presents a neural net trained using SGD. This model combines the sub-networks for vision and language models. If the model is pre-trained on a larger corpus then the resulting captions can make...
Deep Visual Semantic Alignments for Generating Image Descriptions
Posted on 11 Feb 2016
Deep Visual Semantic Alignments for Generating Image Descriptions They use an ensemble of CNN?s over images, BRNN?s over sentences and a structured objective that aligns these two modalities through a multimodal embedding. This Multimodal RNN is used to generate image captions. This model has been tested on the Flickr8K, Flickr80K and MS COCO datasets. This paper treats the language as a rich label space in contrast to the other works which usually focus on labeling images with a fixed set of visual categories. These closed vocabularies constitute the convenient modeling assumptions but are restrictive in comparison to the varying degree...
Learning Knowledge Graphs for Question Answering through Conversational Dialog
Posted on 09 Feb 2016
Learning Knowledge Graphs for Question Answering through Conversational Dialog The open dialog system (called KNOWBOT) can learn about a domain by relating concepts in science question to propositions in a fact corpus (a knowledge base) and can store new concepts and relations within a knowledge graph. The knowledge is acquired from open and natural language dialogs without any knowledge of what the user might say. Hence, it is without any fixed ontology or structure or any domain model. This dialog learning is built on top of a question answering system that is able to refine knowledge from other sources. KNOWBOT...
Shallow and Deep Neural Networks
Posted on 03 Feb 2016
Shallow Neural Networks Have the summation over all the functions, hence, these have only one hidden layer. Deep Neural Networks The final value is obtained through functions inside another functions, hence, these have many hidden layers. These hidden layers may be of differnet types. These are more efficient for representing certian classes of functions. Deep Models are able to extract more information and hence perform better, because they have more intermediate layers to learn the functions. Effect of having more layers on output performance – to be updated based on experiments
Memory Neural Networks
Posted on 23 Jan 2016
Memory Networks Being able to learn more information for longer duration of time is possible by having a large enough memory which is compartmentalized suitably to accurately remember facts from the past. RNN’s alone have difficulty in performing memorization due to exploding gradients and vanishing gradients. The memory network has memory indexed as array indexes with components including the input feature map (I), generalization (G), output feature map (O) and response (R). On being given an input, it is converted to a feature map which is updated in the memory by the generalization rule. The output features are computed and...
Image Captioning - Baby steps - Types of images
Posted on 20 Jan 2016
Since I am researching on image captioning and have avid interest in photography I first decided to understand how many types of images are there. There are so many questions that came up from them. Can we understand that this is an image taken from a kaleidoscope or is it a panoramic photo. Is it a sunrise or a sunset? Is clip art image recognition possible and how How to differentiate between past and present photos of same object in the same environment. Can the computer just classify correct what kind an image is ? What category an image belongs...
AI Complete Question Answering
Posted on 17 Jan 2016
Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks The following are the tasks and their examples. It must be noted that the computer learns the pattern in the language and not the actual language as is shown in this example: Sbdm ip im vdu yonrckblms. Abf ip im vdu bhhigu. Sbdm yigaus ly vdu hbbvfnoo. Abf zumv vb vdu aivgdum. Mduku ip vdu hbbvfnoo? A:yonrckblms Mduku znp Abf fuhbku vdu aivgdum? A:bhhigu Which is indeed a difficult task. Basic Factoid QA with Single Supporting Fact John is in the playground. Bob is in the office. Where is John?...
From Visual Concepts and Back
Posted on 16 Jan 2016
From Visual Concepts and Back Visual detectors, language models, and multimodal similarity models are learnt directly from a dataset of image captions. Part of Speech tagging is used to train visual detectors Word detector outputs are conditional inputs to a max entropy language model Global semantics by re-ranking captions (sentences with high likelihood) using sentence level features, such as a linear weighting of them and, a deep multimodal similarity model. Weights are learnt using Minimum Error Rate Training. Sentence re-ranking is done by MERT because the language model gives several (M) best sentences. MERT uses a linear combination of features...
Question Answering Datasets
Posted on 16 Jan 2016
Datasets and papers for the AI complete task of question answering. Papers Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks End-To-End Memory Networks Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systems The Goldilocks Principle: Reading Children’s Books with Explicit Memory Representations Freebase Semantic Parsing on Freebase from Question-Answer Pairs Large-scale Simple Question Answering with Memory Networks Datasets Aristo Challenge Questions, AI2 Machine Comprehension Test (MCTest), Microsoft bAbI-tasks (forked repo) Winograd Schema Challenge
Learning a Recurrent Visual Representation for Image Caption Generation
Posted on 15 Jan 2016
Learning a Recurrent Visual Representation for Image Caption Generation Model Features Bi-directional mapping between images and their sentence based descriptions which are the captions are learnt using RNN?s. Because of bidirectional nature the ability to generate caption given image and using same model to reconstruct the visual features given this visual description is possible. The recurrent visual memory learns to remember long term visual concepts (sentence generation and visual feature reconstruction). Dynamic representation within model It captures the visual aspects of the scene as each new word is read and the representation is updated to reflect the new information. The...
Towards a Visual Turing Challenge
Posted on 12 Jan 2016
Towards a Visual Turing Challenge A Visual Turing test is an open domain task of question answering based on real world images that resembles the the Turing Test. This is inspired from the holistic architectures for challenging grounding, natural language generation from image and video, image to sentence alignment and question answering problems. Evaluation Interpreting and evaluating the answer of a system becomes difficult and ideally relies on human judgment However, these are not objective methods that can?t be evaluated automatically at a large scale. The evaluation methodology that assigns scores over a large output domain is challenging as the...
AI Hard and AI Complete Tasks
Posted on 09 Jan 2016
Here, I am consolidating what an AI Hard task is and why is image captioning considered one. Image captioning is a AI hard task as it involves: Grounding natural language symbols to the physical world Semantically understanding content of images Multi-modal knowledge from several sub-domains is required to solve AI Hard tasks Well defined evaluation metric (currently it is the number of questions that are correctly answered by a VQA system.) An image is understood when it is possible to generate a caption which encompasses the salient features in the image.
Exploring Image Captioning Datasets
Posted on 09 Jan 2016
PASCAL SENTENCE DATASET Link: http://vision.cs.uiuc.edu/pascal-sentences/ PASCAL stands for Pattern Analysis Statistical Modeling and Computational learning. It has 3 tasks: Image Classification Object Detection Object Segmentation The dataset has 20 classes, including aeroplane, bicycle, boat, bottle, bus, car, cat, chair, cow, dining table, dog, horse, motorbike, person, potted plant, sheep, train, TV. For selecting this dataset, no quality filter was applied, the complete dataset has been directly downloaded from Flickr. Because of no filtering, there are complex scenes, scaling, view points of different objects, unnatural lighting. Training set consists of 10,103 images with 23,374 objects such that there are approximately 500...
Deep Learning Frameworks
Posted on 30 Dec 2015
The purpose of this post is to write down what I know about these frameworks, as I learn more, I will keep on updating with more information. These are the deep learning frameworks that I have used or am using: Tensor Flow Both C++ and Python Released by Google Theano Its a python library that allows one to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently. Developed at University of Montreal. Some important and beneficiary features of Theano are: Tight integration with NumPy Use numpy ndarray in Theano-compiled functions. Hence Theano has been powered for large-scale computationally intensive...
FMRI Brain Data
Posted on 18 Dec 2015
This describes our attempts at the Machine Learning project for 10601(A). Thanks to my team-mate Mona Ramadan and our professors Dr. Aarti Singh and Dr Gordon for guiding and mentoring us throughout. Abstract Machine leaning techniques are used to study the brain activity from fMRI data. The data under investigation is composed of pre-processed signal recordings of 5903 voxels taken from a single fMRI scan. The subjects are performing a simple task of pressing a button when a moving bar reaches a line, but not to press the button if they get an early or late stop signal. Support vector...
Qualitative Evalvations of Word2Vec for Recommendations
Posted on 15 Dec 2015
This project uses Word2Vec for understanding why a review for a business given by a user is rated ‘3 stars’ or ‘1 star’. The report is here. I would like to thank our mentor, Prof. William Cohen for accepting and guiding us throughout this project and my teammates, Sravya Popuri and Srikant Avarsala. Special thanks to Manaal Faruqui who helped us understand the complexities of Word2Vec.
Word2Vec
Posted on 09 Nov 2015
Word2Vec Word2Vec is a class of two-layer neural network models that, given an unlabelled training corpus, produce a vector for each word in the corpus that encodes its semantic information. Hence, Word2Vec processes text, it takes an input text corpus and outputs a feature vectors for words in that corpus, these numerical vectors can be understood by deep nets. Word2Vec offers an advanced vector representations of a term, along with other distributed representations. These are based on the distributional hypothesis, which motivates that the meaning of a word can be gauged by its context. Hence, if two words occur in...
Learning from Image Captioning
Posted on 05 Nov 2015
Visual Question Answering has a very important effect. It may be used to help the blind people see. They can ask questions after being given the image description. All of this will help to understand the image and its contents in depth. Most papers have described their training was done using Amazon Mechanical Turk. Launched in November 2005, Amazon Mechanical Turk is named after a legendary automaton from the 18th century, the Turk, which could play chess. The wooden man, adorned with a turban, appeared to be powered by the machinery of a clock. The Turk was a sensation: a...
Paper Reviews
Posted on 05 Nov 2015
In the paper review posts I will write about the papers that I have read. The posts are in no particular order. I am writing this in order to consolidate my thoughts and gain an insight into how neural networks work. Additionally, reading all these papers helps to the understand the basics, learn about the datasets and the various evaluation metrics that exist. There might be some questions that I write which I am not able to fully grasp or sometimes are just a hunch. They don’t reflect on anything at all.
Grace Hopper Conference
Posted on 19 Oct 2015
Grace Hopper Conference Previously, I got the opportunity to be a part of Grace Hopper Conference, India and this year I got to attend Grace Hopper Conference, Texas too! I got the chance to be motivated by Sheryl Sandberg, Susan Wojcicki, Hilary Mason, Megan Smith, Moira Forbes, CMU’s very own Manuela M. Veloso and so many more inspirational women. I met and was amazed by the energy of so many more great women. GHC Scholar I was a Grace Hopper Scholar for 2015. Thank you Capital One for sponsorship! It’s a great effort that companies like Capital One support and...
Long Term Memory
Posted on 03 Oct 2015
Here is the implementation of LSTM in Theano and tested on IMDB dataset Supervised Sequence Labelling with Recurrent Neural Networks by Alex Graves is a great book. It describes LSTM’s and RNN’s LSTM’s were introduced by Sepp Hochreiter and Jurgen Schmidhuber in their paper. They describe six experiments where LSTM’s are able to perform better than RNN’s compared on various basis. LSTM’s do not have the vanishing gradient problem. They work even when there are long delays, and can handle signals that have a mix of low and high frequency components Word2Vec uses (LSTMs) Long Short Term Memory networks [1]....
CERN Webfest: Open Cosmics
Posted on 25 Aug 2015
CERN Webfest The CERN Webfest is a hackathon organized for the CERN Summer students. Participating in the 2015 CERN Webfest and being a summer intern at CERN was an amazing experience for me. Our team, Open Cosmics presented an idea about opening cosmic data from the cloud chambers to the public and developing a grid so that people can access and contribute to the data freely. We aimed this as a Citizen CyberScience Center project and along the way we were introduced to CrowdCrafting, a unique platform for online assistance in performing tasks that require human cognition, knowledge or intelligence...
CERN Openlab Summer Internship III
Posted on 17 Aug 2015
Rolling Forecast A rolling forecast is performed which helps us to predict binary popularity values for each week. Each week’s data is added to the existing data and a new model is created. This follows the notion, more data leads to better data analysis. Prediction can be done in various ways following the implementation of several machine learning algorithms, mainly, Naive Bayes, Stochastic Gradient Descent and Random Forest. Their models are then combined into an ensemble to check which algorithm offers the best true positive, true negative, false positive or false negative value. This project also includes plotting the results...
CERN Openlab Summer Internship II
Posted on 17 Aug 2015
Dataset A dataset describes a process completely. A process is any interaction taking place in the LHC. An example would be proton-proton collision in the LHC, taking place at a single vertex. A single process may be composed of many collisions taking place at the same vertex. The weekly collection of data is uniquely represented by the name of the dataset. It describes which weeks’ data it contains, 20140101 - 20140107 will describe the first week of year 2014. A datasets format is defined by three distinct parts process/software/tier, where: process: is a process type, examples include Higgs Process, TopQuark,...
CERN Openlab Summer Internship I
Posted on 17 Aug 2015
CERN Openlab Project This summer I was humbled to have been chosen as an intern at the European Council for Nuclear Research, CERN, Canton of Geneva, Switzerland. I was in the Information Technology Databases, which is usually called IT-DB. My project was to evaluate Apache Spark as an analytics framework for CERN as Big Data Analytics Infrastructure. I had never used Apache Spark before. There was an EDx course which was teaching Apache Spark and I enrolled myself for it. I also read some books on Apache Spark, mainly, Learning Spark and Advanced Analytics with Spark. The goal of this...
GitHub custom domain
Posted on 01 Mar 2015
This is a neat CLI package to get ones own cutom git.io domain. It is simple to use. Just do: gitio 'https://github.com/username_or_handle' 'custom_domain' So, gitio ‘https://github.com/sidgan’ ‘sg’ To check if the domain has already been taken, just go to git.io/custom_domain and see if it is available or not. So, hurry!! before someone takes the custom git.io domain that youv’e always wanted.
My induction into the Mozilla Developers Network
Posted on 20 Feb 2015
Mozilla Developers Network I had been editing some of the docs regarding Firefox OS. It was great work. I have always loved open source and this was just another contribution. I had been lurking on the #mdn IRC channel for some time now. One fine day, there came the day of the meeting of all the people in the Mozilla Development Network and I was also present. I introduced myself, and no doubt the people welcomed me. I was all \o/ and happily doing the Irish jig about the great meeting going on. They gave a list of all the...
ShanghAI III
Posted on 14 Feb 2015
Experiment All the experiments were done with my team mate Elena Okhapkina and primary teaching assistant Martin F. Stoelen along with secondary TA Yijiang Ren. Robots and animal behaviour[1] The swiss robots behave as animals and they are out to hunt for the cubes. Providing vision to the robots enables them to see and interpret what it is that they are seeing. In the video below, whenever the swiss robot sees the cube, the console writes 'I see a red cube'. This is done by understanding the patch of colors that appear on the screen and then comparing them to...
ShanghAI II
Posted on 13 Feb 2015
The initial ideas that we started out with were worked upon, implemented on WebBots and along the way got modified slightly. The end result, I believe is full of better ideas and implementations. Robotics derives a lot of inspiration from animal behaviour and this iteration of ShanghAI is all about how robotics is coupled with animal behaviour. The act of evolution has perfected almost every task and we just need to learn from it[1]. Initial Ideas Machine learning to calculate the optimum numerical values of all parameters viz. pitch, yaw. Adaptive boosting for finding the best values of the parameters...
Signing GPG Keys
Posted on 10 Feb 2015
Signing GPG Keys Signing another person’s public keys means that you certifying and stating that you trust that person and their work. BigLumber is a site that allows people to sign each others public keys. You just create your account and upload your ASCII armoured public key. The method of using ASCII armored version to import keys is often preferred because the key comes directly from the user. The keyserver may contain a corrupt key or may be unavailable so the ASCII armored version is given preference. To create the ASCII armoured public key simply do: gpg --output key.asc --export...
Data Visualization
Posted on 16 Jan 2015
Data Visualization When you have a lot of data at hand, often the best thing to do is to plot it on a graph. That way one can visualize it and see how sparse or dense the data points are. It gives a basic idea about the data that we have to analyze. Data visualization as the name suggests, lets one see the data. Seeing several tables or mounds of csv files does not tell us anything apart from the fact that we just have a lot of data. I have used data visualization extensively to understand what data implies....
Deep Learning for Audio Recognition
Posted on 10 Jan 2015
Deep Learning Deep learning is also called deep structural learning or hierarchical learning. It is a set of algorithms in machine learning that attempts to model high-level abstractions in data by using model architectures composed of multiple non-linear transformations. It is the part of a broader family of machine learning methods based on learning representations of data. Various deep learning architectures such as deep neural networks, convolutional deep neural networks, and deep belief networks have been applied to fields like computer vision, automatic speech recognition, natural language processing, and music/audio signal recognition and these have produced state-of-the-art results on various...
Creating and Revoking GPG keys
Posted on 05 Jan 2015
GPG Keys OpenGP standards define the semantics for secure information exchange. GPG is the software that follows OpenPG standards. GPG is an acronym for GNU Privacy Guard. Alice and Bob are the foo-bar, the spam-eggs of the world of computer security. Symmetric key cryptography takes place by two different methods. The first in which there are two keys, both same, and the second in which both keys are different. GPG helps in symmetric key cryptography. Assuming Alice and Bob to be two people who want to communicate and send messages to each other. Both will possess their own private and...
Who Said it?
Posted on 01 Jan 2015
“I love reading Jeffrey Archer.” It is possible to guess which of your friends said it. The computer can also guess who! In a group of people, who is more likely to have said something can be predicted if the computer has heard them talk before. The computer can analyze their thought process while interacting with them and, in due course of time, predict the most probable speaker, given a statement. Introduction ‘Who Said It?’ aims at mirroring the analytical power of the human brain. Just as the human brain can process a given amount of data to deduce something,...
I'm at ShanghAI!!
Posted on 30 Dec 2014
These are my beginner steps in robotics. I have always been fascinated by robotics, and given my interest in artificial intelligence, this area has always been a region of wanderlust for me. However, my interest in robotics was limited to being dazed by the advancements in its field, reading about various types of bots and shaking hands with them. I have never built one, but now, after attending classes in the Global Virtual Lecture Hall of ShanghAI, I definately am capable enough to give it a try. ShanghAI has helped me to delve deeper into exactly how they work, it...
Spider
Posted on 29 Dec 2014
Spider or Ant According to the philosophies of computer science a spider is the same as a ant. Yes! that, in fact is true because this almost jargon is talking about web crawling or web scuttering which is further used for automatic indexing and web scraping. I used the spider algorithm to get data from a few sites. The first hack was using lynx and Beautiful Soup and, later, a better and elegant solution using Python?s urlparse and urllib instead of lynx. Beautiful Soup is indeed beautiful, as beautiful as the morning Sun shining on the dew covered grass. Excuse...
Pipeline for ML
Posted on 14 Dec 2014
Command Line Pipeline for Machine Learning Problems This project deals with aggregating methods for machine learning. Machine learning is a broad spectrum term. It involves churn prediction, recommendation systems and time series analysis just to name a few. Initially I had to work on Kaggle problems trying to gain first hand experience at cleaning data and figuring out what are the most efficient algorithms on which kinds of problems. The first problem that I started on was the Titanic Trainer Challenge. My first prediction lay at a mere 50\%, barely, just touching the last position. However, continually working on the...
Gender Identifier
Posted on 10 Dec 2014
Name based gender identification using NLTK This project is based on a simple idea that usually the female names end in vowels like ‘a’, ‘e’ and ‘i’, whereas male names usually end in ‘k’, ‘o’, ‘r’, ‘s’ and ‘t’. Using this feature, we can generate a confidence value if the input name belongs to a male or a female. So far, I have not found any such study on Indian names and so decided to do one. The study on English names can be found here. The English name corpus has been included in the ‘names’ package of NLTK. It...
Docker Installation
Posted on 30 Nov 2014
Docker Docker like its name lets a developer create a software and ship it easily. Docker quickly assembles applications from its components and tries to eliminate all problems when it comes to shipping a software. It is the ultimate hackers software because it conforms with the identity of fuck it, ship it. It helps in versioning, that means you can create several versions of the same software and use whichever one you want. Like on Github versioning is possible, imagine the same with Docker. You can even create versions of your operating system with Docker much like Windows has the...
Grace Hopper Conference & Hackathon
Posted on 25 Nov 2014
Bangalore, India, 2014 Grace Hopper was a computer scientist and a US Navy Admiral who is credited to several laurels in the field of computers alone that she was nicknamed ‘Amazing Grace’. Amongst her computer genius, is her invention of the first compiler and the idea of a machine independent, high-level, programming language which ultimately led to the development of COBOL. To celebrate Grace Hopper and her achievements, the Anita Borg Institute holds the Grace Hopper Celebration of Women in Computing partnering with ACM India. It is India’s largest gathering of Women Technologists. This year, I got the opportunity of...
Hadoop Installation
Posted on 31 Oct 2014
Requirements: Java See installation of the latest version here A dedicated Hadoop user Add a new Hadoop user and group that will access all the Hadoop files. This is the user in whose directory Hadoop will be installed and will communicate via SSH to the local user. sudo addgroup hadoop sudo adduser --ingroup hadoop hduser Chaos In my first Hadoop installation, I did not create a separate Hadoop user and only then could I understand why the existence of a separate user was necessary. Creating a separate user helps tremendously in terms of file permissions, security and backups. I did...
Data Scientists: Who, What, How?
Posted on 10 Oct 2014
The newspaper reads “Data Scientist: The Sexiest Job of the 21st Century”. For a minute you are like “I thought medical and engineering were the only two domains”. Then you smirk “Does it even pay?”. Then somehow you start to read the article. “Best paid job of the century”. Ahem Ahem! You close the newspaper, and realize you know nothing. Who’s a data scientist you ask? The data scientist venn diagram by Drew Conway explains it quite well. A data scientist is a blend of maths and statistics, hacking skills and substantive expertise. So what does a data scientist do?...
SSH Keys
Posted on 23 Sep 2014
SSH SSH stands for Secure Shell. The SSH keys are nearly impossible to crack because they consist of a private and public key both of which are a long string of random characters. For extra protection one can also add a passphrase which means that anytime these keys are used, the user will be prompted for a passphrase. SSH keys enable one user on a machine to log onto another users account on the same machine. Alternatively, they also allow one user to remotely log onto a remote machine. Generating SSH keys SSH keys can have various types of secure...
New York University Abu Dhabi Hackathon
Posted on 01 Mar 2014
I was gratified to be selected to represent India at the New York Abu Dhabi Hackathon. Hackathons are always great and this was another great one. We developed a web app ‘Orphan Locator’. Before the hackathon, ideation had already begun on Google Groups and this forced us to put our thinking caps and suggest ideas. As more ideas came in, people started discussing their implementation and use cases and it felt like we were already part of the hackathon. Basic Idea: Lost Person Application An application for lost or missing people was answering the need of the hour because Gaza...
IET India Scholarship Awards
Posted on 27 Oct 2013
India Scholarship Awards We had to present a solution for Inclusive Growth using Technology at the India Scholarship Awards 2013, organised by the Institution of Engineering and Technology. In this national competition, participants had to go through four levels of competition and the performance of students was judged by innovative ideas, technically and economically viable solutions and problem solving capacity as potential leaders in future. A total of 5000 students participated. Following this I was honored to be the Women IET Ambassador as well. I also got the opportunity to represent India at the Community Volunteers Conference held by the...
Nokia Do Good Hackathon
Posted on 20 Aug 2013
Nokia Do Good Hackathon This was my first ever hackathon. It was a such a great experience that I just love hackathoning now. The exhilarating experience that teaches you so much and makes one rack their brains is simply awesome. Though I loved to keep my nose in a good book, now I love to participate in hackathons too. Eclectic minds from all over Delhi had come and their lighting talks inspired me. I met Nalin Savara at this hackathon. He motivated me to go forth and leave no stone unturned while putting my best effort into all the work...