If you'd like to get updates on how my progress goes, feel free to subscribe here or follow me on twitter!
In 2020 and beyond, I plan to dive more deeply into independent research projects in Machine Learning and Theoretical Neuroscience. It has been on my mind for nearly a year now, and the new year (decade?) seems as excellent time as any to formalize my thoughts and start. An important point to note is that while I'm extremely early in my career as a researcher, and any other career generally, I have been doing research formally both in academia and in industry for more than a year now.
In this post, I'll briefly walk through some context around my motivations, why I'm framing it as a personal PhD, sources of inspiration, some areas I'm interested in, and the mechanics of my planned independent research journey. Please feel free to jump to specific parts of the post if others are not salient or useful for you.
Motivations
I'm currently finishing 2 separate Master's degrees from Georgia Tech and Harvard, where I focus on both applied and theoretical machine learning and neuroscience. In 2018, following building a popular Machine Learning project with a friend, I decided I wanted to understand ML, particularly Deep Learning, fundamentally and theoretically. My background prior to this was studying Cognitive Neuroscience at UCSD, and I wanted to deepen this aspect of my understanding as well. I had to make decisions regarding which research areas to pursue formally for both of my Master's theses, and settled on studying both Deep Learning and Theoretical Neuroscience.
The following year was an intense one: I received the Pioneer grant, jumped into research with full force, and faced some of the highest highs and lowest lows I've ever experienced. Doing research was hard, especially because of its open ended nature. I was used to tight feedback loops and clearly delivered objectives. I was used to low complexity problems that required high velocity of execution: building Minimum Viable Products, talking to users, doing multiple choice exams. Instead, Research was full of unknown unknowns, and the plethora of topics from mathematics, computer science, neuroscience confused me. On top of this, the sheer amount of new preprints and papers published daily was intimidating.
But, as time passed, I found research immensely satisfying. Uncovering new truths, learning new skills to apply, and being in a world of unknown unknowns is thrilling. Thanks to my advisor, mentors, and peers, it became a thoroughly enjoyable process. The theory of neural networks fascinated me, and the application of various disparate concepts from applied mathematics, physics and biology impressed me to no end. I managed to find some work as a Research Fellow in industry, and also learned about bridging the gulf between concepts emerging in research and application to real world use cases. My own understanding of deep learning grew immensely. By the end of 2019, my peers and I managed to get a paper published, and also have a few more on the way. I even applied to several doctoral programs.
The future is uncertain, however. PhD programs in ML are notoriously difficult to gain admission to these days. While my research record is impressive for the relatively short period of time that I have been active, it is dwarfed by those that have numerous first author publications in top conferences and several years of research experience. Surprisingly, this seems to be the norm for admission to a top program now. On top of this, I'm not sure that I can pursue further graduate school without abandon, as I have other obligations that may prevent me from dedicating another 4-5 years to study.
However, whether I choose to go to industry or continue in academia, I want to continue doing research in any capacity possible. I'm thoroughly interested in the topics of neural architecture search, neuroscience derived primitives, unsupervised learning, and compositionality in neural networks. I have a research project I just started that combines some of these ideas and explores them further, something I will be sharing soon.
I want to continue to do research in these areas, collaborate with all the brilliant folks I've met and will meet in the future, and be a part of the broader research community in any capacity I can. Formalizing my involvement, even as a part time independent researcher, is what I am dubbing the DIY Phd.
Sources of Inspiration and the DIY PhD
Over the past few months, there have been several posts about doing independent research in ML, other CS areas, and generally. In particular, I was inspired by posts from Nadia Eghbal and David Laing. The clear takeaways from their advice and experiences is that doing independent research is really useful when it revolves around producing work on a topic in a structured manner. Producing work, and keeping yourself on the path to doing so, involves a high level of self imposed structure and accountability. Nadia also points out that many non tenure research careers are invariably gig based, which can be heartening or disheartening depending on how you work and your priorities.
Andreas Madsen had an excellent post recently about his time doing independent research, why he suggests avoiding it, and advice he has for those who set out to do so anyway. His experience emphasized the sheer competitiveness of modern ML PhD admissions, and how arduous getting published in a top journal is. At its core, production of useful research is really the guiding light of any researcher's career, and a PhD seems to be training to produce good research. The question becomes, is it possible to do research outside of the traditional academic structure of these programs.
I think it is possible, though perhaps extremely difficult. I think doing independent research is really only something to consider if there are questions in your field of interest that you are absolutely dying to explore and answer, whether you get a career out of it or not. I find myself in such a position, and I'm happy to embrace it both as a personal hobby and professionally, if the opportunity arises.
The DIY PhD should revolve around the production of useful research, research artifacts, and other work that contribute to the field, topic, or question you're interested in. It should be public work that is shared with the community, and its value is ultimately up to others to decide. Personally, my aim is to publish a few more papers this year and a book in the area of Neural Architecture Search, something I'm currently working on.
Areas of Interest
I'm broadly interested in building modular, robust learning systems and applying them to real world use cases. Specifically, I want to discover and better understand neural network priors, automate the building of large, sparse, modular neural networks, and apply these theoretical breakthroughs to real world tasks. In the past, I've worked in the areas of Neuroscience inspired Machine Learning, Disentangling Representations, Neural Architecture Search, and intelligent tutoring systems.
The Mechanics of Independent Research
It seems that the name of the game when it comes to doing any independent work, research or otherwise, is to have a useful output. This can be a paper, source code, a book, or some other piece of content. In my case and in the field of Machine Learning, papers and code are the primary artifacts that researchers output.
To produce this work in a consistent way that mirrors a traditional PhD program, it is imperative to have structure and support networks in place. Personally, I am involved in several communities filled with independent researchers and makers. I have also been a part of the ML research community for more than a year, and have made several friends with which I am already planning some projects for this year, which I will be sharing soon. I think this gives me a slight advantage, as I understand the research process to some degree and will rigorously try to
Sharing work is also imperative, and to that end, I will primarily be using this blog to publish short form write-ups, research notes, and daily updates on my work and research. Preprints will be shared on arXiv and summarized here. Interestingly enough, I am also contemplating setting up a youtube channel where I would livestream the research I do, allowing anyone who wants to chime in or collaborate to do so in a live setting. Let me know what you think of this idea!
Going Forward
This is very much an experiment, and much of the communication and structure I outlined above may change through the course of this year. I really wanted to take the chance to open up a conversation about this topic, see what others who are far more experienced than I think, and get the ball rolling on something I've been planning for a long time. I don't yet know whether I will do a formal PhD, or if I'll find myself in industry doing some applied work, but I do know I want to take the next few years to stay close to my field. My focus is on doing useful work, not chasing recognition, and I hope to get the chance to learn and build something awesome alongside many other brilliant people.
If you'd like to get updates on how my progress goes, feel free to subscribe here or follow me on twitter!