Unfinal

TL;DR: I recently found myself asking the question: how might we design learning experiences that are deeply engaging and lead to meaningful progress? Then, I explored one potential solution in the form of a multi-modal app. You can see this app at: https://www.deeperlearning.space (the app is in closed beta - if you would like access, please send me your email address). --------------- I love learning new things. Learning can mean a lot of different things - I can learn a new fact (like the fact that what are commonly packaged and sold as black olives are actually green olives that have undergone oxidation - thanks l). There is also learning that requires us to engage deeply with a complex topic, potentially over an extended period of time. This form of learning seems hard to come by, but, when experienced, is deeply rewarding. Often because it leaves me feeling like they now understand the world in a way I did not previously. Or, because I can now do something that I previously could not. For example, learning how to code. Doing an online course made me realise that this form of learning is not easy to induce. The reasons are plentiful - content may feel dry/irrelevant, the videos in the online course may not fully engage me, the form of assessment in the course may not encourage deeper understanding/reflection that is necessary for learning. [A question to explore more deeply - what is the purpose of online courses? And, what do existing studies show about their effectiveness?] This experience left me feeling deeply unsatisfied - I had spent 6-8 hours a week, $1000 dollars, and 4 months, and come out the other side not feeling much more capable than before I started the course. Two more thoughts about learning before i dive into what I built: - With the narrative around job disruption by AI, and the level of tasks that can be automated by AI, it may be the case that we need to have a much deeper understanding to be able to judge output produced by AI (this is hypothetical - we may instead need adjacent skills that are not related to learning any particular subject matter deeply, and i would love to engage in conversations about this!)… for example, in the story by Ted Chiang [to elaborate] - When it comes to building a product, a learning product seems to be one that will always need to be built for humans. Other products, which are oriented toward doing some task, may change their shape significantly in a world where agents are a large part of the user base. But for learning, a human will always be the primary user. For this reason, this product is protected from becoming obsolete in a world where AI agents abound - instead, any developments in AI and Agentic capabilities will only serve to strengthen the product. With regards to the second point, I am really excited about the possibilities AI opens up for building personalised, magical, learning experiences at scale - I am sure I am not the only one, and again, super happy to get in touch with others working in this space. With all this in mind, I decided to build a prototype of what learning could look like. It started off as an ambitious project - starting with helping learners figure out what they want to learn, then teaching them that in a context that can keep them engaged, with frequent assessment, gap analysis… the list of features goes on. I started building it, but quickly realised that the scope of the problem I was trying to solve was (a) very wide, and (b) trying to solve the problem of learning, without actually considering what consists of learning deeply in the first place. So, I decided to take a step back. I thought about my own learning experiences. I read about what other people have said about learning. And I cut down what I was building to test one potential medium for AI-augmented learning - a space for students to work on problem sets in an environment that allows them to write and talk to AI while they are doing so. The primary guide in deciding to build this was that learning is an active process. It requires the student to sit and work through the difficulty. Problem sets are one way for them to do this. So, I wanted to augment this important aspect of the learning journey. Additionally, working on problem sets is an effective way to expose gaps in understanding. By making AI available at this point where the gap is identified, to provide the relevant information to fill the gap and then apply it directly to solve a problem, I hoped to tighten the feedback/learning loop. The Application Then, it came to designing the interaction with the app. There are 2 modes of information input in the app - a canvas the student directly writes onto when solving the problem, and a voice tutor they can talk to. And there are 2 modes for information output from the app - the voice tutor’s reply, and a complementary textual hint displayed on the screen. Understanding the design choices Whiteboard: A whiteboard offers a free form of expression and problem solving for the student. It encourages thinking about a problem in different ways. And it allows more effective forms of problem and solution representation. For example, if solving a physics problem, the student can draw the free-body force diagrams into the application directly - you get the idea. Additionally, for math questions, it gives the app access to the student’s thought process. Not only does this allow hints that are tailored to the specific step the student is stuck on, but it also allows the app to form a better representation of the student’s understanding and gaps - information that could be useful to guide future lessons for the student. Voice: A natural way to express thoughts and pose questions. The goal here is to make the experience feel immersive, natural, and encourage vocalisation of thoughts - whether it is expressing train of thought, or the particular part of the problem the student is stuck on - as naturally as possible. Even being able to articulate where you are stuck has meaningful positive impact on learning by promoting awareness and reflection. Textual Hint: A purely voice response from the tutor is interactive, but loses out on richness of information that could be better displayed through images. Voice is also transitory. The textual hint provides a permanent reference the student can revisit as they are solving the problem. The goal of all this is to make the learning experience feel magical. So magical, that students want to stay on and continue learning, because they see the value it brings them. That being said, there are some limitations to the current form of the app that need to be addressed: - The whiteboard works great on touch-first devices such as iPads. On laptops, using the cursor and trackpad is clunky, and does not induce the flow state I intend. Optimistic me hopes that this form of learning becomes so normalised that cheaper, more effective devices to write into a laptop (future writing tablets) are invented to make this possible. - Voice as input breaks in noisy environments. - Voice as input does not work in quiet study environments (libraries) The voice as input problems can be addressed by allowing students to type instead, but I feel like that also loses some of the flow I would like users to experience while using the app. So, if you have any ideas on how to address these issues, would love to hear them! (if not, i guess we’ll have to wait for BCI to catch up and then we won’t need voice any more :)) For now, I have mitigated the limitations for voice as input noted above by adding a "I'm stuck" button that the user can press when they are stuck. It triggers both a voice response and a text response (just like if they had spoken out loud), and thus overcomes the limitation of the app not being usable in a quiet study environment/noisy environment. Regarding this problem space, to explore: [can wispr flow like products solve some of the problems wrt quiet environments? is whispering as opposed to talking out loud a different learning experience? what solutions exist to isolate the user's voice from background noise?] What I have built is the first step. There are plenty of things I would like to explore further. A non-exhaustive list: - At present, the app allows students to upload their problem set in the form of a PDF to work on in the app. This leaves some room for error in PDF parsing, and adds complexity when dealing with figures, sub-problems etc.. In a future version, providing educators a way to add questions (and even solutions) into the app could overcome any complexity and fragility that plagues this manual extraction step. - Designing a lesson for students based on identified gaps - Creating flashcards and revision notes based on gaps identified during problem solving (existing learning apps provide flashcard generation. I believe that deciding what flashcards to generate based on gaps identified during problem solving can make the feature more relevant for learning). - Creating a “starting step” for students when they open the app - make it frictionless to begin a new learning experience, to make it easy for students to start engaging with the app, instead of having to spend time and energy figuring out what they need to work on next. - Building educator-facing features to share relevant information with teachers to enable interventions by them when needed - Integrating with sources of knowledge (textbooks, papers), giving students an immediate way to engage more deeply with a topic they feel drawn to without having to leave the site. Ideally, partnerships can be formed with the publishing companies to enable their content to be made available in the app in as native a way as possible. Also to explore are the "mnemonic medium" type books referenced by Andy Matuschak. - Finding ways to cultivate attention, problem solving, and critical thinking - meta skills that can enhance learning. My own learning experience has often been tainted with lots of distraction and procrastination. Given the importance of focus and attention for learning, how can we design ed-tech apps to address competing apps for attention? Learning is a complex process - neuroscience, psychology, sociology all have important insights that can help us understand what leads to optimal learning experiences. This doesn’t even begin to address the bureaucratic and organisational challenges of meaningfully transforming the traditional learning experience at scale. I am barely scratching the surface - but being here has me excited about the possibilities the future holds. Addendum: As I delved into this space, I came across some really interesting thoughts/ideas/products that resonated deeply with me. I’d like to share them here - if any strike a chord, I would love to hear your thoughts: - https://www.straitstimes.com/singapore/ai-a-great-personal-tutor-for-students-not-a-shortcut-for-learning-says-openais-raghav-gupta - https://andymatuschak.org/hmwl/ - https://blackboardlm.com/ - https://www.heyclicky.com/ - https://ocw.mit.edu/courses/cms-595-learning-media-and-technology-spring-2024/pages/syllabus/ - https://arxiv.org/pdf/2605.04816 (TO READ) - https://drphilippahardman.substack.com/archive Also, as I built this learning app, I realised the potential AI has for transforming how we interact with technology, and how we can use it to augment thinking in other contexts (e.g. work) too. I will explore these ideas more deeply in a separate post.