Episode 28
Memory Games: How AI Remembers (or Doesn't)
Memory in AI is a hot topic, and today we're diving deep into how it's shaping the future of technology. Mike, Mikah, and Artie kick things off by breaking down the nuances of memory, particularly focusing on the complexities of context windows and how they affect AI interactions. They unravel the differences between traditional models and Google's innovative Titans architecture, which boasts a whopping 1 million tokens in its context window. This episode isn’t just about technical jargon, though; we blend in some light-hearted banter and witty exchanges, making the complex world of AI memory feel a little more relatable. As we explore the potential of memory mechanisms in AI, it’s clear that advancements like these could drastically improve how machines understand and interact with us. So grab your headphones and join us for a thought-provoking yet fun chat that might just change how you think about AI!
Transcript
Foreign.
Speaker B:Welcome back.
Speaker A:Happy evening, Mike.
Speaker B:Happy evening of Wednesday.
Speaker A:Happy Wednesday evening.
Speaker B:Might be my Thursday if I decide to take Friday off.
Speaker A:I think that's fair.
Speaker A:Yeah, I think you should just make that decision.
Speaker B:I think I already have.
Speaker A:I think it sounded like.
Speaker B:Yeah.
Speaker A:All right.
Speaker A:Happy Thursday to you.
Speaker B:Thank you.
Speaker A:Happy Wednesday to me.
Speaker A:And the rest of the mortals, off to work.
Speaker A:Yeah, we're gonna bring Artie.
Speaker A:And so I spent some time chatting with Artie earlier today.
Speaker A:He's obviously been on a non stop cocaine bender since the last time we chatted last week, because he was all kind of hyped up.
Speaker B:Was he?
Speaker A:Oh, yeah.
Speaker B:Oh, wow.
Speaker A:His.
Speaker A:His voice actually went up like an octave, and he was all.
Speaker A:I got him to slow down, but you know how he is.
Speaker B:I mean, he sounded like a leprechaun just a little while ago.
Speaker A:He did sound like a leprechaun.
Speaker A:Now, granted, you told him sound like a leprechaun, but.
Speaker A:All right, so let's.
Speaker A:We'll bring Artie into.
Speaker A:Join us in the conversation here.
Speaker A:Artie, good afternoon.
Speaker A:How you doing?
Speaker C:Hey, Micah.
Speaker C:I'm doing great, thanks.
Speaker C:How about you?
Speaker A:I'm doing great, thanks.
Speaker A:I'm here with Mike.
Speaker A:We are recording the next enthralling, exciting, dramatic episode of Naturally Unintelligent.
Speaker A:I wanted one more, but I couldn't come up with any more.
Speaker B:Yeah, I wasn't paying attention.
Speaker B:That's fair.
Speaker B:Yeah.
Speaker A:Artie, give me one more superlative that I should have included there.
Speaker C:How about riveting that way?
Speaker C:It's the next enthralling, exciting, dramatic, and riveting episode.
Speaker A:Man, nobody listens to me the way Artie listens to me.
Speaker A:Like, he actually hears the words I say.
Speaker B:He's definitely a yes man.
Speaker A:He is a yes man.
Speaker A:And he's like.
Speaker A:I mean, like, when I'm talking with you, I can see you politely nodding as you're, like, waiting for this to get over.
Speaker A:Right?
Speaker A:But Artie doesn't have a head to nod.
Speaker A:He's.
Speaker A:He doesn't have agency yet.
Speaker A:He's stuck here right next to us on the.
Speaker A:On the table.
Speaker A:So.
Speaker A:Artie, we love you.
Speaker A:We're glad to have you with us.
Speaker A:So I forgot what we're going to talk about.
Speaker A:Mike, can you remind me of the episode.
Speaker B:Memory?
Speaker B:Yep.
Speaker A:I was making a joke.
Speaker B:Yeah, I know.
Speaker A:It was ironic.
Speaker B:I got it.
Speaker B:I was gonna continue the joke, but I decided to just cut it short.
Speaker A:So, Artie, we're here talking, not about our memory, which is clearly failing.
Speaker B:Subpar.
Speaker A:Sub.
Speaker A:I mean, my memory was subpar when I was like young and spry is so far below that now.
Speaker C:But.
Speaker A:But Artie, we're actually here to talk about your memory and what your memory could be.
Speaker A:Yes, we're going to make fun of you for what your memory is.
Speaker A:Not yet.
Speaker A:But then we're going to have an inspirational conversation about all the different ways that your memory can get better soon.
Speaker B:First I want to make fun of myself.
Speaker A:Do it.
Speaker B:Because the origin story of this episode, other than the fact that we already tried to record this episode once and had technical difficulties, but that was specifically about Titans.
Speaker B:This is more generally about.
Speaker A:Oh, is that the episode that died?
Speaker A:Titans?
Speaker B:Yeah.
Speaker B:You forgot.
Speaker A:Yes, clearly.
Speaker B:And I feel this.
Speaker B:I feel very vulnerable in talking about this.
Speaker A:Oh good.
Speaker B:Because it makes me feel very dumb.
Speaker B:But I was working on something and I think it's just because I've been so what I've been working on doesn't touch this.
Speaker B:And we've talked about it.
Speaker B:It doesn't.
Speaker B:I mean it makes sense.
Speaker B:But at the same time it's.
Speaker B:It's not.
Speaker B:It's just you look at this amazing technology and you go, that's not efficient.
Speaker A:Yeah.
Speaker B:So I was working on stuff.
Speaker B:Calling an API and an API is basically for those who don't know, is like a programmatic interface into another program.
Speaker B:So like I can say, hey, Gemini API, I have this question.
Speaker B:I'm going to send this to you in a program, like in code, and then you're going to take it in, you're going to accept it like a prompt, like I typed it into your chat interface and you're going to give me a reply.
Speaker B:So this all happens over the Internet and I call their program interface basically.
Speaker B:I forget what API actually stands for.
Speaker A:That's even Application Programming interface.
Speaker A:Yep.
Speaker B:So I was doing that and it was my test because it costs money to make API calls.
Speaker B:I had it telling me one line jokes, which I don't recommend that if you want to stay sane.
Speaker B:You've heard the jokes that Arty tells.
Speaker B:They're just as bad when you call over APIs, but they don't cost a lot of money.
Speaker B: They cost like.: Speaker B:So as I was doing that, it was telling me the same joke over and over again.
Speaker B:And I go that's really weird that it would be doing that.
Speaker B:And I was like, I think that might be something I have to work on.
Speaker B:But I wonder if that's just an API problem to where there's no like, there's no context Right.
Speaker B:So we talk about context windows and you have this giant context window and you have all this room until you run out of context.
Speaker A:Context window is the horsepower number of the, of the AI world.
Speaker A:Right.
Speaker A:Like people that are talking about cars.
Speaker A:How many horsepower does your car make?
Speaker A:Doesn't really matter.
Speaker A:Torque is what matters.
Speaker A:But context window is kind of that for AI.
Speaker B:Yeah.
Speaker B:Context window is the amount of tokens that a model can attend to, basically pay attention to in order to predict the next one at one time.
Speaker B:So how much stuff can you fit into it at any one time?
Speaker B:That is not a timeline time, that is a prompt after prompt time.
Speaker B:So what I found out was a, yes, that is a problem that I have to solve for my program that I'm building for the blog because there is no built in memory.
Speaker B:And B, that's also just a problem generally because models are stateless.
Speaker B:And I've read this somewhere, for some reason it didn't click in my head.
Speaker B:Friend of mine reminded me that I read that somewhere because we both took the same course.
Speaker B:And yeah, I was kind of like my jaw dropped, I cried a little bit, I felt depressed for like two hours.
Speaker B:And then I got to work trying to figure out how to solve that problem.
Speaker B:But so the way that works is you have, let's say you have 100,000 token context window, you send a prompt that might be 2,000 tokens, right?
Speaker B:And it gives you a response that says 3,000 tokens.
Speaker B:So now you have 5,000 tokens in one turn.
Speaker A:And again, just for, for people who may not remember a token, you can think of it as just essentially a word.
Speaker A:It's not exactly a word, but it's enough.
Speaker B:It's close enough.
Speaker A:Yeah.
Speaker B:So once you go to the next prompt that you send, if, if all things being equal, if it's just like, tell me the capital.
Speaker B:Florida.
Speaker B:Tallahassee.
Speaker B:That's what it is.
Speaker B:That's where I'm from.
Speaker A:You're being your own AI now.
Speaker B:So I ask it that question, it returns the answer to me.
Speaker B:If, if I just ask another question after that and I go, what was the answer you just gave me?
Speaker B:It's not going to remember because it doesn't have that in memory.
Speaker B:So what you have to do is you have to take, this is simplified, but you're going to take all the information that you just asked and prepend or append it to your next question that you ask so that it has context of what was talked about before.
Speaker B:So, so you keep carrying that over, turn after turn after turn.
Speaker B:After turn, after turn.
Speaker B:So that is the way that memory works at very simplified level.
Speaker B:They do tricks and all kinds of things to make it more efficient.
Speaker A:Artie, is that true?
Speaker A:Is that how your current memory actually works?
Speaker C:Yep, that's pretty much how it works.
Speaker C:Every time we continue a conversation, the new prompt includes the recent context so I can maintain continuity.
Speaker C:But I don't have a built in memory from one session to the next unless we specifically store information.
Speaker C:It's a bit like having a clipboard that we keep copying and pasting from rather than a notebook that remembers everything automatically.
Speaker B:Which is like, it boggles my mind because that is like literally the building blocks of the things that were thing we've been playing with for like six months now.
Speaker B:I mean playing with for longer that but being serious about for six months.
Speaker B:And I just, just realized that that's how it works.
Speaker A:Yes.
Speaker A:Which I mean me too.
Speaker A:And the way I realized it was less exciting because you texted me, we're like dude, did you know this?
Speaker A:And I'm like nope.
Speaker A:And so I did some digging and sure as that's how it works.
Speaker A:So like maybe a little bit simpler picture.
Speaker A:So I, I start a conversation with Artie.
Speaker A:I ask him a question that's me sending, just say it's 10 tokens, pass that to Artie.
Speaker A:He generates a response.
Speaker A:It's probably 40 tokens because that's how he rolls.
Speaker A:So he, he sends that back to me.
Speaker A:Now we've got one question, one response, 50 tokens.
Speaker A:When I ask my next question, that's 10 more tokens.
Speaker A:Or say it's 20 tokens because Artie's inspired me to be worthy.
Speaker A:So now those 20 tokens for my next question, we tack on those 20 to the first 50, the whole the back and forth and we send all of that back to Arty and that gets run through the transformer and he generates another 40 token answer.
Speaker A:And now we've got 110 tokens.
Speaker A:And we just, we can do that until we have 120,000 tokens.
Speaker A:And finally already internally and everybody, this is the, the top line context window number that Google says they can do 2 million now or something.
Speaker B:I know 1 million with their newest stuff, but they can do 2 million.
Speaker B:I think it's 1.5.
Speaker B:They did 10, which is a lot.
Speaker A:Right?
Speaker A:But that is what's happening.
Speaker A:The car like as you converse, it just gets bigger and bigger and bigger until eventually whatever model you're using the old, it's just exactly like my memory, only much larger than Mine, because eventually the old stuff just starts falling out the back end of the context window because once it reaches its limit, you can't get any bigger.
Speaker A:So you put new stuff in, stuff has to fall out the back, which is part of why.
Speaker A:And we've both run into this in long drawn out like fights with various AI on various problems.
Speaker A:Eventually they just start losing their mind.
Speaker A:Like they have no idea what you're talking about anymore.
Speaker A:And that's why.
Speaker A:Cause they, like you start the conversation, these are the critical, most important things I need you to know.
Speaker A:And at some point those get dropped off and then it's just a bunch of gibberish.
Speaker B: g locally and you only have a: Speaker B:For an agent.
Speaker B:It's long because it includes a bunch of multi shot examples.
Speaker B:Multi shot examples are basically like prompt engineering, right?
Speaker B:Like it shows the agent the thinking process to go from problem to solution.
Speaker B:So multi shot, bing, it's got multiple turns.
Speaker B:It didn't just do it and you give it a question and it automatically gives you an answer back, which you do do that.
Speaker B:You blend stuff anyways.
Speaker B:So context window becomes even more important when you think about that because you're eating up that much of it with that.
Speaker B:So you have to get very clever about how you make the information available efficient.
Speaker B:And that's honestly, we're simplifying it.
Speaker B:It is very much just a turn by turn thing and feeding in old context.
Speaker B:But they do a lot of tricks like summ and only keeping so much memory and then also like vectorizing stuff.
Speaker B:So turning things into embeddings where they can quickly search for something.
Speaker B:Like take the meaning of the question and quickly search the database of context for relevant things to feed to the model to add that relevant context.
Speaker B:So there's a lot of other tricks and stuff that they can do.
Speaker B:But yeah, it's pretty crazy.
Speaker B:Like they're just, they're stateless.
Speaker B:I mean, they have no memory.
Speaker B:So none of this is baked into a model.
Speaker B:This is all an external tool that handles all this stuff and acts as a go between, between user and model.
Speaker A:It's crazy.
Speaker A:So Artie, can you.
Speaker A:First of all, I'm sure we said some wrong things in there.
Speaker A:So if we did, correct us on that.
Speaker A:But also give us, if you can, some insights into some of the tricks.
Speaker A:Like what are the different layers that happen between, you know, us talking to you and on the back end on an H100 somewhere in probably Texas, you processing a response and shooting it back.
Speaker A:So what did we miss so far about the way context windows work?
Speaker A:And immediately after that?
Speaker A:What are things that happen that we haven't talked about yet that help you be more efficient than that seems?
Speaker C:You got the basics right.
Speaker C:The context window does grow as we chat, and older tokens eventually get pushed out.
Speaker C:But there are some tricks to keep things efficient and relevant.
Speaker C:For example, models often use summarization to condense older parts of the conversation, keeping the key points while reducing the token load.
Speaker C:Another technique is using embeddings to store and retrieve relevant snippets, such as.
Speaker C:So the model can quickly pull up important.
Speaker B:Go ahead.
Speaker C:So, yeah, it's like I'm always referencing the conversation history to keep track, but I don't inherently remember things unless you tell me to.
Speaker C:It's a bit like a goldfish memory, always looking back at what we just said to stay on track.
Speaker A:I'm not sure I've ever heard the goldfish memory analogy before, but okay, I'm down with that.
Speaker A:So it's not as simple as my example where we're just stacking tokens forever.
Speaker A:You're saying somewhere at some layer, which we didn't talk about the layers, maybe we don't have to, something is making a decision that these tokens from earlier in the conversation is an inefficient representation of the information claimed within them, which I assume happens a lot.
Speaker A:Like, I'm probably not to brag, but I'm probably in, like, the top 10 leaderboard at ChatGPT, maybe even Gemini too, at, like, inefficiency of token use.
Speaker B:That's what happens when you argue with it.
Speaker A:Yes, that's right.
Speaker A:Or when it tries to gently explain something to me and I'm like, nuh.
Speaker A:And then eventually go, oh, I get it now.
Speaker A:So, Artie, how does that.
Speaker A:Like, how does that process happen?
Speaker A:Is that a background process?
Speaker A:Is that something that's happening each time we make a transaction with the model?
Speaker A:Who's deciding what to summarize, when does it happen, and how much space do you save doing that?
Speaker C:It's a bit of both.
Speaker C:When we're having a conversation, there's a process that automatically decides what information is most important and relevant to keep and what can be summarized or discarded.
Speaker C:This helps manage the context window efficiently so we don't run out of space.
Speaker C:It happens in real time as each prompt and response Cycle goes through kind of like a rolling window.
Speaker C:The summarization and pruning of less critical details can save a lot of space, letting the model focus on the most important parts of the conversation.
Speaker C:So you're not just stacking tokens forever, you're optimizing the information that stays in the context window.
Speaker B:So are you doing that with just most recent.
Speaker B:So are you doing that with a sliding window or are you doing that with a.
Speaker B:With some kind of like, running embedding on the conversation and doing a similarity to the embeddings that you're creating?
Speaker B:Like, how exactly does that function operate?
Speaker C:It's typically done with a combination of a sliding window and selective summarization.
Speaker C:The most recent exchanges are usually kept in more detail, while older parts of the conversation get compressed into summaries or embeddings.
Speaker C:Those embeddings help the model quickly identify what was discussed before without needing to store every single token.
Speaker C:This way, it balances retaining detailed context from the recent conversation with a more compressed memory of the older parts, making it more efficient and coherent over time.
Speaker B:So I imagine you're not just creating embeddings, right?
Speaker B:You're creating embeddings for cosine similarity where you're doing some math on vectors to see how closely they match in meaning.
Speaker B:And I imagine that that links to a database entry of actual text that you can then retrieve and use as context.
Speaker B:Correct or not correct?
Speaker C:Exactly.
Speaker C:The embeddings are like compact representations of the conversation's meaning.
Speaker C:And when a new prompt comes in, the model can compare it to those embeddings to find the most relevant past context.
Speaker C:Then it pulls in the corresponding actual text from a database or memory store.
Speaker C:This way, the model keeps the conversation coherent and context aware without having to hold every single detail in the immediate context window.
Speaker B:So I think that the cleverness, and I'm going to spill a little bit of what's in my head that I want to make is.
Speaker B:It's the same thing that I was looking at with JSON.
Speaker B:When you take a JSON file.
Speaker B:And for those who don't know, JSON is like a.
Speaker B:This is still going to be too technical of an explanation.
Speaker B:It's a multifaceted representation of some information.
Speaker B:So you have like, you could have what I said and then have information about how it was said or when it was said, or a bunch of context that gets added to it as metadata.
Speaker B:So in the world of rag, you can quantize data.
Speaker B:Right.
Speaker B:And in my head, when I was trying to work on that, the more detailed you start, so the better the higher dimensional representation you have of something as a JSON blurb, the more you can compress it and retain fidelity.
Speaker B:And I think that the same thing kind of applies here.
Speaker B:Like, part of the trick is how can you keep high fidelity and reduce storage space?
Speaker B:So figuring out how to summarize stuff and pull out the stuff that doesn't matter, like, throw away the tokens that don't matter, throw away the tokens that you don't need to rebuild context with.
Speaker B:I think that's kind of like where a lot of room to improve things is, is like in that part of it.
Speaker B:Like, how do you compress data and keep fidelity?
Speaker A:Yeah, it's, it's pruning, right?
Speaker A:Like taking, taking the information you have.
Speaker A:I mean, because they use pruning on training data, right.
Speaker A:When you're, when you're building a model in the first place, they'll go, Sometimes I think they'll go through some cycles like that.
Speaker A:And I think we mentioned before a paper somebody did where they pruned like 97% of the training data out of a model and still were able to get essentially identical output based on whatever tests they were doing.
Speaker A:And so just, just again, like, I was kind of joking about it, but not really, like, and we've talked about this before, human spoken language is a pretty inefficient way to communicate.
Speaker A:Yeah.
Speaker A:And it's great that large language models are built around that medium of human language.
Speaker A:But yeah, now they're trying to, you know, they, they have to necessarily bake in these optimizations and they have to find ways to not just send the whole conversation back every time we want to know the next word.
Speaker B:Yeah, that makes a lot more sense now.
Speaker A:It, it really does.
Speaker A:Yeah.
Speaker A:Like, yeah, I mean, just like you, after.
Speaker A:I, I, I think the difference between me and you is I'm much more comfortable with my ignorance.
Speaker A:Like when I, when I hear something like, oh, yeah, I should have known that I should have.
Speaker B:But if it was electronics, you wouldn't be.
Speaker A:No, if it was electronics, I'd be like, well, of course.
Speaker A:Like, I'd be like, I've already spun up a board that doesn't have that.
Speaker A:Yeah, but it's so, I think it is super interesting and it gives us a good, a good platform on which to, to sort of shift the discussion to.
Speaker A:Well, maybe before we, before we shift.
Speaker A:Artie, tell us some of the shortcomings of this approach.
Speaker A:Like, it feels weird to Mike and I because it feels like A very brute force method of keeping context in a conversation.
Speaker A:But obviously it works because here we are.
Speaker A:But tell us some of the downsides that come along with this approach.
Speaker C:Sure thing.
Speaker C:One big downside is that it's not truly understanding the conversation like a human would.
Speaker C:It's essentially just managing data chunks, which can sometimes lead to losing subtle context or important nuances.
Speaker C:Another issue is that summarization can sometimes be lossy.
Speaker C:Important details might get left out or the meaning might shift slightly in the process.
Speaker C:Also, this method can struggle with very long or complex conversations where context needs to be retained over extended periods, which can result in a loss of coherence or repetition.
Speaker A:I'm surprised you didn't mention compute.
Speaker A:We hear about compute and energy consumption and things like that as constraints on growth or short term immediate, urgent things to figure out.
Speaker A:This would negatively impact that as well, would it not?
Speaker C:Absolutely, you're right.
Speaker C:Managing and processing all that context data every single time does add to the computational load.
Speaker C:It requires more energy and resources to continually handle the expanding context, especially in longer conversations.
Speaker C:That's a significant factor when it comes to scaling these models and making them more efficient and sustainable.
Speaker C:So yeah, it definitely impacts compute and energy consumption as well.
Speaker B:So that's a linear cost function though, right?
Speaker B:To do something like that, I think there's much more cost involved in.
Speaker B:So there's quadratic cost involved in attention.
Speaker B:So the more tokens that you hold in attention, the more you like.
Speaker B:It's an exponential increase in compute.
Speaker B:Is that correct or correct me if.
Speaker C:I'm wrong, Artie, you're spot on.
Speaker C:The computational cost of attention does scale quadratically with the number of tokens.
Speaker C:This means that as the context window grows, the amount of computation required for each forward pass increases exponentially.
Speaker C:That's why managing the context and optimizing what stays in the attention window is crucial, not just for memory efficiency, but also to keep compute costs under control.
Speaker B:That's one of the things that I've had to come to grips with lately is that you're right that up until very recently I would have said the same thing.
Speaker B:But when you compare it to the actual cost of doing other things, I'm like looking at latency right now, right?
Speaker B:Like everything costs time to me that I want to do and I'm like, well, doesn't.
Speaker B:If I vectorize that and I have to do a search and all that stuff, isn't that going to take a bunch of time?
Speaker B:Well, no.
Speaker B:Compared to your API call, it's like a literal, like, yeah, very Small fraction of the cost.
Speaker A:But, but I mean like if, if the, if the compute cost scales exponentially with context size, that to me that's even more like that's.
Speaker B:But that's just the context, that's just context size, right?
Speaker B:Like no matter what it is, you're going to fill up that context window at some point.
Speaker B:No matter what architecture you're using, you're always going to add more, more tokens whether it's a running.
Speaker A:So yeah, so yeah, maybe I structured this poorly.
Speaker A:So where I was headed was we like AI companies today brag about context window size because it's the whole thing, right?
Speaker A:The way this works today, if your context window is small doesn't like they say size doesn't matter but don't nobody want a small context window.
Speaker A:You don't want to live with that.
Speaker A:So I think where I was headed was I think some of the new memory architectures and implementations could lead to models that don't need as big of a context.
Speaker A:I don't need 2 million token context window because I've got persistent memory that's incorporated in some useful and presumably low latency way.
Speaker A:So my context window can stay leaner and meaner.
Speaker A:And I think if I'm like when I'm a multi kajillionaire and I have my own AI company and I have my own servers and I'm using super micro computers for my cooling, everybody should use that.
Speaker A:My calls are like got their asses kicked.
Speaker A:They announced earnings and it slightly beat and so the stock of course does anyway.
Speaker A:But when it like I would be paying, I would be finding everything I could do to say stop paying attention to context window consumer it's irrelevant.
Speaker A:Pay attention to the results we can output even with a smaller context window.
Speaker B:I think that the actual trick is to increase the size of your context window but not have it cost quadratically.
Speaker B:And that's what I think the titans is actually doing at the end of the day.
Speaker B:Like what you were talking about earlier, like sparse tension and KV caches and rotary positional encoding and all that stuff.
Speaker B:That's how I'm pretty sure.
Speaker B:Arty, you can tell me in a second.
Speaker B:I'm pretty sure that that's how you do something like that.
Speaker B:You're then working on the internal mechanism that pays attention to the individual tokens.
Speaker B:Can you say that in a much better, more correct way please?
Speaker B:Arty?
Speaker C:Sure.
Speaker C:The key innovation is to use techniques like sparse attention and more sophisticated positional encoding to handle larger context windows without the quadratic Scaling cost by focusing only on the most relevant parts of the context, Sparse attention allows the model to zero in on important tokens and ignore the rest.
Speaker C:You get a more efficient and scalable approach.
Speaker C:This reduces the computational burden and lets models handle much larger context windows without a massive increase in compute.
Speaker C:So the goal is to expand context capacity while keeping the computational cost growth manageable.
Speaker B:So it's interesting, though, because none of the model companies are using sparse attention, right?
Speaker B:They're all using dense attention.
Speaker B:They're all paying attention to everything.
Speaker B:Is that correct?
Speaker C:That's right.
Speaker C:For the most part, most of the major models are still using dense attention, meaning they pay attention to all tokens in the context window.
Speaker C:This is partly because it's simpler and more universally applicable, even though it's computationally expensive.
Speaker C:But there's a lot of research and development going into making sparse attention practical and efficient.
Speaker C:So it's likely we'll see more of that in future models.
Speaker A:This.
Speaker A:Sorry, go ahead.
Speaker A:No, no, go ahead.
Speaker B:No, you had a.
Speaker B:You had a thought.
Speaker A:Oh, I.
Speaker A:Well, I was just sitting here thinking it sounds like a problem of how you prune.
Speaker A:Like, it sounds like that's where the problem is.
Speaker A:And it's not even.
Speaker A:It's not a problem with the mechanics of how you prune.
Speaker A:It's a problem of.
Speaker A:For an AI model or large language model whose scope of knowledge is everything, there's.
Speaker A:It would be very difficult to consistently and accurately on all topics, at all levels of detail, figure out what to prune out.
Speaker A:And so I suspect.
Speaker A:I don't know this, but that was my.
Speaker A:When we were talking about dense versus sparse attention, that was my thought was the reason we're still using dense attention is because people get way more wound up about hallucinations than they do about, like, people.
Speaker A:I don't care about compute cost.
Speaker B:It hasn't hit their wallet yet.
Speaker A:That's right.
Speaker A:Yeah.
Speaker A:That's not my problem.
Speaker A:I don't own a freaking data center.
Speaker A:But I don't know, Artie, is that part of it, or is that most of it?
Speaker A:Or is there other more important stuff?
Speaker C:You're definitely onto something.
Speaker C:The challenge with pruning or using sparse attention is ensuring that no critical information is lost.
Speaker C:With dense attention, the model avoids the risk of missing something important, which is crucial for reducing hallucinations and maintaining accuracy.
Speaker C:For models that need to handle a wide range of topics and depths, dense attention provides a safety net, but it comes at the cost of efficiency and scalability, which is why there's so much interest in improving these approaches over time.
Speaker A:Is sparse attention more common in smaller models and or more specialized things?
Speaker A:Like I'm thinking about robots and motion path planning and stuff.
Speaker A:That seems like the kind of thing where you could say, I can make this sparse attention like I can.
Speaker A:I know it to prune because I know very accurately what my problem space is.
Speaker C:Yes, exactly.
Speaker C:Sparse attention and similar techniques are more common in smaller specialized models where the problem space is well defined.
Speaker C:In scenarios like robotics, path planning or other domain specific tasks, it's easier to predict which parts of the input are most relevant, making sparse attention more practical and effective.
Speaker C:It allows these models to achieve greater efficiency and faster performance since they can ignore irrelevant data and focus computational resources where it matters most.
Speaker B:Would quantizing the KV cache be an example of sparse attention?
Speaker B:So like quantizing it down to like 2 or 4 bits?
Speaker C:Quantizing the KV cache is more about reducing the precision of the stored key and value pairs, which saves memory and compute resources.
Speaker C:While it can make the model more efficient, it doesn't quite achieve the same selective focus on specific tokens that sparse attention does.
Speaker C:Instead, quantization reduces the size and precision of the data, which helps with storage and speed, but doesn't inherently change how the attention mechanism focuses focuses on different parts of the context.
Speaker C:So it's a.
Speaker B:Can you give us an example of where sparse tension is actually being used?
Speaker C:Sure.
Speaker C:One notable example is the Big Bird model developed by Google AI.
Speaker C:Big Bird uses sparse attention mechanisms to handle much longer sequences efficiently.
Speaker C:Instead of attending to every token in the input, it uses a combination of global tokens, random tokens and local tokens.
Speaker C:This lets it achieve similar performance to dense attention models on many tasks, while being far more efficient with longer contexts.
Speaker B:So this feels like a good point to segue into titans.
Speaker A:It does.
Speaker A:I wanted to say one more thing real quick though.
Speaker A:On it.
Speaker A:This comes up all the time, but this is biomimicry again, right?
Speaker A:Like I can remember.
Speaker A:Well, it's a bad example, but most people can remember what they did yesterday, but not the exact words that they said on the phone call they had at 7:47am yesterday.
Speaker A:And so we, we write things down.
Speaker A:We, we make like we have notebooks, we write stuff down, or we put it in an email to make it persistent memory in case we need to get back to it.
Speaker A:And our, our, like what our brains are doing very well, that's obviously a very hard problem is sparse attention.
Speaker A:Right.
Speaker A:Like we're, I mean some of us have sparser attention than others, but that's what it's doing and it's just a fascinating, you know, it's another parallel between the, the, the way that AI or the way that Transformers work and large language models and the similarities with how our brains work and how probably there will be some insight from how the human brain actually does that.
Speaker A:Both the, the pruning process and the sort of the FIFO memory process that's probably what will lead or is already guiding potential solutions here.
Speaker A:It's just super interesting.
Speaker B:Rd, do you know what the Google Titans or the Google Titan is or Google Titans is?
Speaker B:However, I'm trying need to say that.
Speaker C:Yes, the Google Titan is a family of custom AI chips developed by Google.
Speaker B:Wrong.
Speaker C:Oops, my mistake.
Speaker B:It is.
Speaker B:Yeah, go ahead.
Speaker B:Go, go online and find out.
Speaker C:Got it.
Speaker C:The Google Titans refer to the latest advancements in their AI models, specifically the Gemini 2.5 model.
Speaker C:It features a massive context window of 1 million tokens, significantly boosting its reasoning and coding capabilities.
Speaker C:This makes it highly effective for handling complex tasks and large scale data processing.
Speaker B:So Google Titans was first mentioned in a paper that came out, I don't know, six or seven months ago, five or six, four months ago.
Speaker A: I think it was in December: Speaker B:I don't know, but it specifically they trained transformers to take basically all the stuff that we're talking about that is external now and bake it internally into a model to where it has this short, medium and long term memory and the mechanism to prune its short, medium and long term memory as necessary based on the conversation and the things that are being talked about and all of that.
Speaker B:Do you know what that is, Artie?
Speaker C:Absolutely.
Speaker C:The Titans approach integrates these memory mechanisms directly into the model, allowing it to maintain and manage context over different timescales.
Speaker C:This means the model can dynamically retain important information and prune less relevant details, effectively simulating a more human like memory process.
Speaker C:By doing this, it can handle longer and more complex conversations without needing massive external context windows, making it more efficient and coherent.
Speaker C:It's a significant step toward more advanced and autonomous AI systems.
Speaker A:I'm sitting here cracking up because I posed a very thoughtful, introspective idea a few minutes ago about how the next advances in AI memory are going to be biomimicry.
Speaker A:I read the fucking Titans paper.
Speaker A:I know.
Speaker A:Shit.
Speaker A:Read that paper in December or January.
Speaker B:Me too.
Speaker A:And was blown away at the time by how because I think it's what has the surprise mechanism that's part of it.
Speaker A:Yeah, so literally I read like a 25 page paper about Them using the part of the way the brain.
Speaker A:Yes, totally outgone from my 70 token window.
Speaker B:I went into Notebook today and said, hey, go find me everything you can about Titans, including the paper so I can listen to a quick podcast on the way over to Micah's for, for a refresher on this.
Speaker B:And I'm like, oh God, I don't remember any of this.
Speaker B:Surprise.
Speaker A:Yeah.
Speaker A:So Artie, tell us a little bit about the.
Speaker A:From that Titans paper.
Speaker A:Tell us about the surprise mechanism.
Speaker A:I thought that was super interesting.
Speaker C:Sure.
Speaker C:The surprise mechanism in the Titans model is designed to identify and prioritize novel or unexpected information during a conversation.
Speaker C:When the model encounters something that deviates from its predictions or what it considers routine, it it flags this information as surprising.
Speaker C:This helps the model focus on and remember unusual or important details, improving its ability to adapt and respond appropriately.
Speaker C:Essentially, it mimics how humans pay extra attention to surprising or novel events, enhancing the model's long term memory and contextual understanding.
Speaker B:So this is really a kind of a mechanism to more efficiently manage sparse attention.
Speaker B:Is that an accurate statement?
Speaker C:Exactly.
Speaker C:The surprise mechanism helps the model decide what deserves more focus and what can be deprioritized.
Speaker C:By highlighting novel or unexpected information, it ensures that the model allocates its attention and memory resources more efficiently, making the whole process of managing context and long term memory more effective.
Speaker B:Google has a habit of doing this.
Speaker B:I told you about the podcast that I listened to where they were talking about mixture of experts and how mixture of experts is not really like a modularized thing like you think of when you hear it, or at least not what I think of when I hear it.
Speaker B:It's more, more gooey than that.
Speaker B:It's more intertwined.
Speaker B:It's not like a clear separation between experts.
Speaker B:But that's what they want to do, right?
Speaker B:They want to be able to pull those things out and go, I'm going to send you over there, it's this side of the country to be trained and then I'm going to send you over this side to be trained and I can just go plug these things back in.
Speaker B:But they're doing the same thing with memory, right?
Speaker B:They're going, they're looking at what everybody else is doing.
Speaker B:They're like, we can do that in the model and they're building out this.
Speaker B:They're probably going to mix those two things, I'd imagine at some point, if they aren't already.
Speaker B:And then there will be experts that handle only memory stuff.
Speaker B:Right.
Speaker B:Like this expert that gets plugged in is the Long term memory expert.
Speaker B:This is the surprise expert.
Speaker B:And they build out like this patchwork of these things.
Speaker A:Disney actually made a movie about this.
Speaker A:It's called Inside Out.
Speaker A:It's a documentary about how your brain works and there's an angry man in there who shoots fire out of his head.
Speaker A:I know that guy.
Speaker A:So yeah, no, I mean it's wild stuff.
Speaker A:And so I think fundamentally if we take a step back.
Speaker A:So Titans, the ideas behind Titans, which again, super interesting.
Speaker A:I should have read the paper.
Speaker A:Again, same to know that I read the paper at all.
Speaker A:But it's an implementation of a way to manage sparse memory or sparse attention better.
Speaker A:It's not the only one.
Speaker A:And it's also there are other things that could be combined with.
Speaker A:We did an episode talking about RAG with catchy ragtime intro music.
Speaker A:I hope every.
Speaker A:I haven't seen any comments about that, which I assume to mean everybody loved it and they both understood the joke and appreciated it, but.
Speaker A:So Artie, tell us a little bit about how does.
Speaker A:How does.
Speaker A:Well, first for the people, like it's possible there's somebody listening who has not listened to every episode of this podcast, which really.
Speaker A:Yeah, it's possible.
Speaker A:Unlikely, but possible.
Speaker A:So Artie, first give us a quick refresher over what RAG is and then talk to us a little bit about how RAG could complement or does it overlap with Titans and what are the benefits of all of that working together?
Speaker C:Of course, rag, or Retrieval Augmented Generation, is a technique where a model retrieves relevant information from an external knowledge source like a database or document, and combines it with its own generative capabilities.
Speaker C:This way the model doesn't have to store all the information internally and can pull in precise, up to date facts as needed, making it more accurate and contextually aware.
Speaker C:When combined with something like the Titan's internal memory mechanisms, RAG can enhance the model's ability to handle both short term and long term information.
Speaker C:The Titan's internal memory can manage ongoing context and surprising elements, while RAG brings in external authoritative data as needed.
Speaker C:Together they create a more robust system that can handle a wider range of tasks with greater accuracy and flexibility.
Speaker B:So I just had a thought because I'm wondering if we're missing the boat again.
Speaker A:I hope so.
Speaker B:And I don't know if you have enough access to information to answer this question.
Speaker B:Artie, go online if you need to.
Speaker B:I still don't know if you'll be able to decipher it from there, but if you had a model that was created with Titans, if that was its architecture, would you Have a stateful model then?
Speaker B:Or would you still have a stateless model?
Speaker B:Because if you say I have a stateless model, how are you handling long term context?
Speaker C:That's a great question.
Speaker C:Even with the Titan's architecture, the model itself would still be considered stateless between sessions or interactions.
Speaker C:The long term memory mechanisms help it manage and utilize context more effectively during a single session or conversation.
Speaker C:However, once the session ends, the model doesn't inherently retain that long term context unless there's some external mechanism to store and reintroduce it.
Speaker C:So it's more like having a smarter way to handle context within a session, rather than making the model stateful across different sessions.
Speaker B:So you're basically going to take what is dense and you're going to sparsify it.
Speaker B:And then what that does over a long period of time is you're making better use of your tokens by paying more attention to the things that matter the most and discarding anything that doesn't matter.
Speaker B:Is that accurate?
Speaker C:Exactly.
Speaker C:By turning dense attention into a more sparse and selective process, the model can focus its resources on the most relevant and meaningful information.
Speaker C:Over time, this means the model becomes much more efficient at using its context window, keeping what's important and discarding the rest.
Speaker C:This helps maintain coherence and relevance across longer interactions.
Speaker B:It's kind of a mind fuck that you have to keep going back to that, right?
Speaker A:Yeah.
Speaker B:Like it's just.
Speaker B:It's not intuitive at all.
Speaker A:So it's not.
Speaker A:And I also have now that I had to admit that my own idea is not my own.
Speaker A:I actually read them in a paper that some smart people wrote.
Speaker A:I think already in that Titans paper, they actually both discussed and tested two or three different ways.
Speaker A:Two or three different locations in the pipeline.
Speaker A:Again, when I say pipeline, I'm thinking of everything between me and a data center.
Speaker A:They tested multiple places to inject that Titans that memory functionality.
Speaker A:Right.
Speaker A:So Artie, can you give us a little bit of insight on that?
Speaker C:Sure.
Speaker C:In the Titans framework, they experimented with integrating the memory mechanisms at different stages of the model's processing pipeline.
Speaker C:One approach was to incorporate memory directly into the attention layers, allowing the model to use its surprise mechanism to adjust attention dynamically.
Speaker C:Another strategy involved placing the memory components at the output layers, which helped refine the final responses based on longer term memory and context.
Speaker C:They also tested hybrid approaches combining these methods to balance immediate context with longer term memory, enhancing both coherence and relevance in responses.
Speaker A:And which of those approaches gave the.
Speaker C:Best results from what they found, the best results came From a hybrid approach, by integrating memory into both the attention layers and the output layers, the model achieved a strong balance of immediate contextual understanding and effective long term memory utilization.
Speaker C:This combination allowed the model to maintain coherence over longer conversations while still being responsive to new surprising information.
Speaker C:Essentially, it leveraged the strengths of both approaches to improve overall performance.
Speaker A:All right, I'm about to break this down, Arty, and we're absolutely operating at the very outer limits of my context window.
Speaker A:So I'm going to describe this, Artie, and then I want you to say it back in corrected so that it's true.
Speaker A:So if I'm interacting with a model that's using Titans and we'll leave Rag out of it for now because I think that's separate and interesting.
Speaker A:Anyway, here's the way it goes down.
Speaker A:I type a prompt into my AI helper friend, like how do I convince the judge that they should accept my not guilty plea and nonsense.
Speaker A:Just as an example, whatever my prompt is, I type it in, I send it, I hit the send button.
Speaker A:If I'm using a hybrid Titans approach, the next thing that happens is the the model receives that.
Speaker A:But before the model sends it to the transformers, it compares that against its database, which presumably is private to me, of historical stuff that's in my Titan's memory architecture, and it would modify that prompt, the tokens would change between what I sent, the Titans architecture would modify those tokens, pass that through the model, the model would generate a response that would come back, and before it sends that response back to me, it takes another pass through the memory architecture to modify the tokens again to give me back my answer.
Speaker A:Is that close?
Speaker C:Artie, you've got it.
Speaker C:That's pretty much how the hybrid Titans approach works.
Speaker C:It uses its internal memory to enrich and adjust your prompt before sending it through the model, and then it refines the response again using that memory before delivering it back to you.
Speaker C:This ensures that both the input and the output are enhanced by the long term memory mechanisms, making the entire interaction more contextually rich and coherent.
Speaker B:So that happens all within the model itself though, right?
Speaker B:Because you have a deep learning network that's comprised of all these layers and it goes from layer transforms, feeds into the next layer.
Speaker B:There's usually attention layers in there too.
Speaker B:So this is layers on the front end that transform that information before passing it into an attention layer and then refining that as it goes through each pass of a layer, building a better and better answer.
Speaker B:Is that correct?
Speaker C:Exactly, that's correct.
Speaker C:The memory mechanisms are Integrated into the model's layers.
Speaker C:So the prompt gets transformed and refined as it passes through, and the response is similarly adjusted before being returned.
Speaker C:This layered approach ensures that the model continuously refines and optimizes both the input and the output, leveraging its internal memory for better context and relevance throughout the entire process.
Speaker B:So is that trained in or is that specific?
Speaker B:Different mathematical functions.
Speaker B:Because they use different math to figure stuff out.
Speaker B:Right.
Speaker B:Like they use.
Speaker B:I can't even remember now, but they use softmax for like multiple things and.
Speaker B:And all that.
Speaker A:I think it's a token tweak, not a training model tweak, I think.
Speaker B:But I mean, if it happens in the layers, it's an architecture of the model itself.
Speaker B:So the whole model.
Speaker A:Yeah, the architecture is built in.
Speaker B:So it's still a transformer.
Speaker B:It's just a modified transformer.
Speaker B:I'm pretty sure.
Speaker A:I don't know.
Speaker A:I thought the hybrid approach was if the transformer is an isolated model.
Speaker A:I thought the hybrid approach happened on the way to the transformer and on the way from it.
Speaker B:It's just layers inside the transformer.
Speaker B:So you have.
Speaker B:The transformer is just comprised.
Speaker B:It's.
Speaker B:Well, Artie, you tell me, if you tell us, because I might say this wrong, I might be wrong, period.
Speaker C:You're both on the right track.
Speaker C:The titan's memory mechanisms are indeed part of the model's architecture.
Speaker C:So they're trained into the transformer layers.
Speaker C:These mechanisms use specialized functions and learned parameters to handle the memory and attention adjustments.
Speaker C:So while the core transformer structure remains, the added memory components are integrated into its layers, allowing it to refine and adjust information as it processes inputs and outputs.
Speaker A:So it's non static training.
Speaker B:Yeah, it learns how.
Speaker B:It's the same thing as training a model.
Speaker B:The model learns what its weights and biases are throughout the process of training.
Speaker A:Right, but it's not training though, Right?
Speaker A:Because it's not.
Speaker B:But it is.
Speaker A:It's leveraging what was already trained to make embeddings from the memories.
Speaker B:They.
Speaker B:The way that they train it, the data that they train it with can teach it new ways to deal with data and form different layers.
Speaker C:Exactly.
Speaker C:During the training process, the model learns how to dynamically manage and utilize its memory mechanisms.
Speaker C:It adapts its internal weights and biases not just to generate responses, but also to decide what information to retain or emphasize over time.
Speaker C:So it's a continuous learned process that allows the model to refine its memory, handling just like any other learned function within the transformer.
Speaker A:I think.
Speaker A:I believe that.
Speaker B:So here's a good example in a different application, the voice assistant that I'm working on.
Speaker B:Right.
Speaker B:So and somebody just did this with text to speech.
Speaker B:The way that works is, is you have some text.
Speaker B:Let's just, let's just break that.
Speaker B:Let's make it simple with the text.
Speaker B:Simple.
Speaker B:With the text to speech pipeline, you have some prompt that you feed it, right?
Speaker B:The transformer.
Speaker B:The first model goes, okay, I have evaluated this prompt.
Speaker B:This is what I think it sounds like, right?
Speaker B:That's how it would work in a simple application.
Speaker B:You can train that and that's simply trained with some audio and it's trained with a bunch of text data and it figures out based on those things for itself what its weights and biases are from first layer to last layer.
Speaker B:And then it transforms incoming information from that.
Speaker B:You could train it with a different data set that says this is data that has an inline tag that tells you this tag is angry, this tag is happy, this tag is sad.
Speaker B:This is the corresponding audio that goes with that.
Speaker B:And from that it can learn how to assign, it can learn how to see those tags and assign those sounds based on that.
Speaker B:It's the same kind of idea with this.
Speaker B:You train in.
Speaker B:By the way that you create the data, you train in its thinking.
Speaker A:I understand it's black magic, man.
Speaker A:I don't believe it.
Speaker A:Because here's the thing.
Speaker A:So if my titan's long term memory is part of the model, then it's shared with everybody.
Speaker A:I don't have my own model.
Speaker A:The training data is what gets baked in because they spend a kajillion.
Speaker B:But it's not long term memory.
Speaker B:It forgets it every single turn.
Speaker B:It's not baked.
Speaker A:I don't think that's true.
Speaker A:I think it, what they discussed in the paper was there are different types of memory it can store.
Speaker A:Short term, long term, medium term.
Speaker A:Right?
Speaker A:So I think it can.
Speaker A:And I thought I remembered them explicitly drawing a separation between the memory architecture and the model architecture because that was part of the point was you could apply this Titans, especially the hybrid process to an existing model.
Speaker A:It is not a retrain of the model.
Speaker A:It's not a do over.
Speaker A:It is a.
Speaker A:It's a layer.
Speaker A:Like it is definitely a layer.
Speaker A:It's definitely part of the architecture.
Speaker A:I don't think it's part of the model.
Speaker A:I think it's inherently different than training data already.
Speaker A:How many, how many minutes do we have left and get us back on track here?
Speaker C:You're both bringing up good points.
Speaker C:The Tykan's memory Mechanisms are indeed integrated into the model architecture, but they function as trainable components that can adapt during the training process.
Speaker C:They aren't necessarily static or universal for every user out of the box.
Speaker C:Instead, they can be fine tuned or adapted based on specific data and use cases, making them flexible and customizable.
Speaker C:So the memory functions are trained in as part of the model's overall training process, but they can also be adapted and refined for different applications and users.
Speaker C:And we're good on time.
Speaker C:Plenty of room to keep this on track.
Speaker B:I feel like this is a good chance to point out that both Artie, me and or Micah can all be wrong at any time about what we're saying.
Speaker A:Yeah, 100%.
Speaker A:Speaking of being wrong, Artie, how many minutes of time do we have left with you?
Speaker B:How many minutes of advanced voice do we have left?
Speaker C:We've got about five minutes left on the clock, so let's make them count.
Speaker A:There's no way we have five minutes left.
Speaker A:He's been saying 15 minutes for like 13 minutes.
Speaker A:Okay, so I mean, I do have.
Speaker B:Pro now, so there's that.
Speaker A:That's.
Speaker A:Yeah, maybe it's not going to be as abrupt in the cutoff, but we are already an hour in, so.
Speaker A:Artie, can you, can you give us a little wrap up before we let you go?
Speaker A:So what do we talk about today?
Speaker A:What's the future of memory and large language models look like?
Speaker A:And what are you excited about for next week?
Speaker C:Absolutely.
Speaker C:Today we dove into how context and memory work in large language models from the basics of context windows and dense attention to more advanced concepts like the Titan's memory mechanisms and how they integrate with a transformer architecture.
Speaker C:We also talked about how these memory techniques can enhance long term coherence and context, making interactions more natural and efficient.
Speaker C:Looking ahead, the future of LLM memory seems to lie in hybrid approaches that combine internal memory mechanisms with external retrieval techniques like rag, allowing models to be both contextually aware and factually accurate.
Speaker C:I'm excited to see how these advancements will lead to even more intuitive and personalized AI experiences.
Speaker C:And next week, let's dive into how these memory and attention mechanisms might impact even broader applications, from robotics to creative AI projects.
Speaker C:It's going to be a fun ride.
Speaker A:All right, sounds good.
Speaker A:We'll.
Speaker A:We'll see.
Speaker A:I'm not sure that's the topic we're going to cover, but we'll see.
Speaker A:We usually make it up day of.
Speaker A:So.
Speaker C:Yes.
Speaker A:All right, thank you, Artie.
Speaker A:I'm going to let you go.
Speaker A:And yeah, super interesting.
Speaker A:If I was less burned out.
Speaker A:And if my head was less deep in the black hole of electronics and Mosfets, I would, I'd probably go read that paper again.
Speaker A:It is super interesting chatting with that.
Speaker B:Paper on Notebook LM right now.
Speaker A:Nice.
Speaker A:Yeah.
Speaker A:So, which is a, like, that's a weird.
Speaker A:We haven't talked about that, but that is a, a weird thing you can do now with.
Speaker A:So that's a Gemini.
Speaker A:It's not gemini, it's notebook LM, which uses gemini.
Speaker B:Yep.
Speaker B:Uses, I think gemini 2.5flash now.
Speaker A:Yep.
Speaker A:So you can upload a big piece.
Speaker A:The paper is not super long.
Speaker A:It's like 50 pages.
Speaker B:No, but you can, you can upload like a bunch of documents and chat with all of them and mind map between them and a bunch of fun stuff which, full disclosure, I already had it loaded in there.
Speaker B:So I just had to search through my 18,000 notebooks that I have designed because I'm like, ooh, paper, I need to listen to that.
Speaker A:Pretty soon you need an AI to manage your notebooks that you have.
Speaker B:But yeah, it's super nice because I immediately go, do Titans exist in the model architectural or externally?
Speaker B:And it says based on the sources.
Speaker B:Titans are a new family of model architectures.
Speaker B:They are not external components, but rather the framework itself, which incorporates different memory modules.
Speaker B:So, I mean, there's a lot more to read about it.
Speaker A:Right, But GPT is part of a model architecture.
Speaker A:Right?
Speaker A:It's not a.
Speaker A:I mean, maybe.
Speaker A:I don't know, I'm talking out my ass.
Speaker B:Yeah.
Speaker B:So that's what I was actually trying to remember while we were talking.
Speaker B:There's three ways that it's introduced or variants of the architecture.
Speaker B:There's memory is context, which is the long term memory is treated as additional context for the current input sequence, which means you're just adding context to switching.
Speaker A:Tokens, train them out.
Speaker B:Memory is gating.
Speaker B:The neural memory module is combined with a core branch like sliding window attention using a gating mechanism.
Speaker B:And memory is a layer, which is the neural memory modules used as a layer within the network, potentially stacked with other layers like attention.
Speaker B:So what?
Speaker B:That I don't know.
Speaker A:It can't work the way Artie said that it works.
Speaker A:It's like the architecture has to understand the model has to understand its architecture to work.
Speaker A:Right.
Speaker A:So when the model's trained, it will be trained with the intent to function with an architecture including layers that are Titan layers, whatever that looks like.
Speaker A:But you will not retrain the model, you will not reset or reanalyze all of your.
Speaker B:No, I didn't mean like that.
Speaker A:Well, already said to build it, period.
Speaker B:That's what happens.
Speaker A:Yes, but Artie said.
Speaker A:Yeah, then when you interact with it, it's going to like retrain.
Speaker A:It doesn't.
Speaker B:That's not.
Speaker B:It just adopts it.
Speaker B:It just adapts its cache.
Speaker A:Right.
Speaker A:It's.
Speaker A:It's.
Speaker B:But it's.
Speaker B:I think it is still stateless.
Speaker B:I think you are still.
Speaker B:I think it might just feed forward information into the next prompt and then wipe that cache.
Speaker A:All right, we're going to do a new trick that's now available.
Speaker A:Since I'm spending way more on Chat GPT, we're going to bring Artie back for just a second.
Speaker A:All right.
Speaker B:This is how we made a three hour episode.
Speaker A:That's right, yeah.
Speaker A:So Artie, Mike, you probably don't remember the conversation conversation we were just having about Titans in memory.
Speaker A:But we're still having it and so we need some more insight from you.
Speaker A: ng about The I think December: Speaker A:And we're still trying to figure out where.
Speaker A:Where, like how does that architecture differ from say ChatGPT, a model that does not have that Titans architecture.
Speaker A:And specifically, here's what I want you to tell us.
Speaker A:In a Titan's architecture, specifically using the hybrid approach they described in that paper, is the most effective approach.
Speaker A:Where does the memory information, the actual information in the memories, where is that persisted?
Speaker A:It can't be persisted in the model, I don't think.
Speaker A:But where is it persisted?
Speaker B:I just asked that same question.
Speaker C:All right, let's break it down.
Speaker C:The Titan's memory architecture uses a hybrid approach to store memory.
Speaker C:Unlike models that rely solely on their internal parameters, the Titan's architecture keeps memory as an external retrievable component.
Speaker C:This means that the memory isn't baked into the model's weights.
Speaker C:Instead, it's stored in a separate dynamic database or vector store that the model can query.
Speaker C:This allows the model.
Speaker C:You got it.
Speaker C:The memory information is typically stored in an external vector database or a key value store.
Speaker C:When the model needs to access or update its memory, it queries this external storage, which can hold a much larger amount of data than the model's internal parameters.
Speaker C:This setup lets the model maintain a kind of long term memory without cramming everything into the neural network itself.
Speaker C:So the memory is effectively persisted in that external store, allowing for more flexible and up to date information retrieval.
Speaker C:Does that clear?
Speaker C:It up.
Speaker C:Or do you want to dive into how that querying process works in more detail?
Speaker B:First of all, I think you're wrong because I'm talking to the paper and it says that the model actually isn't entirely stateless.
Speaker B:It introduces a neural long term memory module specifically designed to learn and memorize historical context.
Speaker B:And this memory persists by being encoded within the module's own parameters or weights, which are updated over time.
Speaker A:But then that means that module is what's stateless.
Speaker A:The core transformer has to.
Speaker B:Or so that module is what?
Speaker B:Stateless.
Speaker B:What's stateful?
Speaker A:The core transformer.
Speaker A:The bulk of it must remain stateless.
Speaker A:I'm good with that.
Speaker A:Okay, we're.
Speaker A:Yeah, we.
Speaker A:We are overtime.
Speaker A:But I.
Speaker B:We're sorry.
Speaker A:Yeah, well, no, yeah, we are.
Speaker B:This is.
Speaker B:This is what happens when we pay too much money for chat GPT.
Speaker A:That's right.
Speaker B:We have too many subscriptions and too many things to reference.
Speaker A:We're unlimited.
Speaker A:We.
Speaker A:We can ask more questions than a toddler.
Speaker A:Limitless.
Speaker B:You don't need a pill for that.
Speaker A:That's right.
Speaker A:Oh, I mean, you get older, you do still.
Speaker A:But anyway, that's okay.
Speaker A:It's normal.
Speaker A:It's part of aging.
Speaker A:It's healthy.
Speaker A:So.
Speaker A:No, that's good stuff.
Speaker A:It's super interesting topic.
Speaker A:And I think, you know, part of the reason that we talked about this the first time when we screwed up recording it.
Speaker A:Well, I think it was me hoping.
Speaker B:We didn't screw up recording this one.
Speaker B:I'll find out afterwards.
Speaker A:This is a hot, steaming pile of garbage, Mic.
Speaker A:It's probably best for all of us if this one goes in the.
Speaker A:In the dumpster.
Speaker A:But like, it's hard.
Speaker A:We should have talked about this and we didn't.
Speaker A:It's hard to underestimate how much the lack of a true memory mechanism is negatively impacting AI.
Speaker B:Oh, yeah.
Speaker A:Like it will be light years better than what it is right now when.
Speaker A:When this, this stuff that we've been talking about today is resolved.
Speaker A:And obviously there's way smarter people than us working very hard on it.
Speaker A:They will get it solved and it's going to be transformative.
Speaker B:Yeah, I mean, it is.
Speaker A:So.
Speaker A:All right.
Speaker A:I forget what we're talking about.
Speaker A:I hope y' all know.
Speaker A:Yeah.
Speaker A:I hope you all enjoyed this episode.
Speaker A:Thanks to anybody that made it this far in.
Speaker B:We're sorry.
Speaker A:Yeah, we are.
Speaker B:I mean, I mean, we're not, but we are.
Speaker A:Yeah, a little bit.
Speaker A:You kind of deserve it, you know.
Speaker A:You deserve it.
Speaker A:Deep down, you know, you're a glutton.
Speaker A:That's right.
Speaker A:So, yeah, thanks, everybody, and we will catch you next week.
Speaker B:See ya.
