The kids who used AI and studied for a similar amount of time as the non-AI high performers apparently had similar (slightly higher) performance. The kids who used AI and did poorly used the AI to do the homework for them (this is what the article says).
I believe AI is basically an amplifier of bad and good. I’m cynical about the world and assume it will be used more for bad than good, but I don’t doubt some of the best people in every field will be using AI to amplify their work in good ways.
It's not so clear that AI is an amplifier.. The paper has some fascinating analysis on this topic:
"At the other end of the distribution, AI students who spend more than 65 minutes on their homework receive homework and exam scores similar to those of non-AI students, suggesting that these students do not use generative AI for homework assignments. However, this group consists entirely of students who adopted generative AI no more than Öve months. Six months after adoption, no AI student spends more than 65 minutes completing their homework (see Figure A5). This is consistent with the gradual process of learning how to use AI tools. It also suggests that AI crowds out the highest level of e§ort."
"Interestingly, in the range of 50-65 minutes, the median and the interquartile range
of exam scores of AI and non-AI students are similar. This implies that, in the range
where AI students and non-AI students have overlapping homework times, students
who spend the same amount of time completing homework on average receive similar
exam scores."
"This pattern shows that students who spend the same amount of time on homework
learn similarly, with or without generative AI. In other words, generative AI reduces
time spent learning for the majority of AI students but not learning efficiency for those who spend the same time studying as the non-AI students."
I wondered the same. ff and fi are common ligature letter pairs, maybe the original is in an encoding where those are represented with separate characters, which then correspond to § and Ö in whatever encoding we find here?
It was copied from a pdf, maybe LaTeX generated. All the ligatures are wrong -- not sure where the problem lies, the pdf itself, pdf reader, or the clipboard on Linux.
Depends who's using it. Like many tools the force multiplier depends on the operator.
I'm confident it's an amplifier for people who know how learning works and already do a lot of it, successfully. However the level of "learning fluency" I'm talking about isn't reached for many until late college or grad school, and sometimes not at all. So I'm not surprised by the quoted results for 12-18 year olds.
Fair question. In theory you could do an experiment where the subjects were grad students or professors, or top performing college students. Give them some fixed amount of time to understand some new subject on which they'll be tested, only 1 group has access to an LLM with appropriate context for the learning task, etc. That's just one idea...
This experimental setup would only tell you whether this demographic (grad students or professors, or top performing college students) benefits from access to LLMs for acquiring a new knowledge or skill.
It would not prove that "the results depend on the operator". To prove this (and figure out the traits that make a proficient operator) you would either need a massive dataset to which you'd apply some machine learning to figure out the correlations, or at least you'd need a hypothesis on the traits you want to test.
Do left-handed people perform better with LLMs? Analytical thinkers? Dyslexic people?
It's not enough to just claim "some individuals will perform better with these tools" without any notion of who does, and whether it's possible to become one of these people, and what is the expected gain in this group.
Otherwise, sure, we could sell unsecured chainsaws as a tool for making ice sculptures and observe that "some people" are indeed more proficient at not getting their face cut off ; but if the tool is being marketed to schools, such a vague statement is not helpful in making a case for it.
Yes, I'm using "high-performing college students" or "phd students" or whatever as a proxy for "people skilled at learning". It's not perfect but I think it'd be good enough. You could do lots of refinements on the idea. You could try to do some kind of "pre-test" that more directly tested "meta-learning skills".
My only point was I think it's possible to do. I personally think it's pretty clear. And there's nothing special about AI. In rural places, you could replace it with "access to books" or "access to a good tutor" and so on.
I also think you'd see the same effect with "motivation" (if you could test it), with performance on standardized tests, and with grades. AI will boost the better (by these metrics) students more.
And people who want to learn. Most teenagers lack agency in their studies. They aren't in high school because they love it but because they have no choice.
> Sometimes things are just common sense pure and simple.
The discussion is about "AI", so common sense is out the window. These people's professional reputations depend on addict-level "AI" usage remaining socially acceptable.
well most university exams are designed to measure how much you study. so we didn't really need a study to tell us, "Exams continue to measure what they are designed to measure."
they're not designed to measure general aptitude, or function as admissions criteria, or screen for job applications, or any other numerous things they are used for.
there can be many questions of pedagogy. one of them is, what do our exams measure and how do we use them? professors who say, "My exam is designed to measure who studies, not be used for all these other purposes that they are actually used for" - I don't buy it. It's the same as late night comedians saying they are not responsible for solutions, even when spending 90% of their air time making political jokes.
THIS is the pedagogical issue, that pedagogy has NEVER caught up with the scope of responsibilities. This is acute in STEM - I mean, the humanities departments are generally pretty well run, all things considered, in this regard. Generative AI is accelerating that pre-existing crisis.
The fact that exam scores are correlated with how much you study is not the same as exams only reflect how much you study. Two students who study the same amount could have very different exam scores. The reason that there still is a strong correlation between exam score and time of study is because if all other things being equal, students who studies more have higher exam scores.
i'm not saying they reflect how much you study. they reflect a lot of things, including that. but you ask the people who write the tests, they're going to say, how much you know or how much you study, but nonetheless, they are limited. i agree with you. that's part of my point.
let's imagine a different study. we instead compare AI-users and non-users on a Wechsler (IQ-adjacent) test.
overall, it would be surprising if AI usage impacted your Wechsler scores. someone has done this study and the impact is quite quite small. BUT. do we care? We don't use Wechsler scores for admissions, we don't use them for jobs, we don't use them for... are you getting it now? A Wechsler family test is measuring something real, just like a university exam measures something. But what do we USE them for? Wechsler and a typical university exam are, in some senses, EQUALLY vague in terms of their fitness for purpose for answering a question like, "should we hire this guy?"
Like there is an association between IQ and earnings but it is actually surprisingly small! There is an association with math education and earnings and it is also surprisingly small. And consider how many people get by just fine without using a single piece of math education once they have finished school - like what if maximizing your earnings isn't all that it is about? Are you getting it now?
The issue isn't the AI usage. I can find tests that are immune to AI usage. The issue is using tests for things that they are not designed for. We pick and choose, for some subtle but nonetheless pervasive cultural reasons, which tests we use for which purpose, and very frequently, not because they are calibrated for the chosen purpose. This is coming from someone who scores very well on all these tests, and have kids, so I have a very strong incentive to buy into the status quo, and I'm telling you: academic testing has been fucked up for a long, long time.
Education is a complex topic. I don't claim to understand it, nor do I think we can sort this out in a hacker news thread. What I reject is the simplistic claims like "exams don't mean anything". There are certainly exams that are badly designed, but there are also exams that are well designed. A well designed exam can look very badly to different people, based on what they know and where they come from. It's a bad idea to think exam scores as the single metric of education quality, it is equally bad to reject them completely, because chances are any alternative measurement people come up with are going to be worse.
I think this deviates from the original topic. The article's intent is to argue that AI helps with learning itself, while you question whether this sample can serve as a reference. But what if the target of this sample is a group of serious college students? The article questions AI's enhancement of learning ability, not whether that exam has a real purpose.
You're right that academic exams also measure something along the lines of instruction following / obedience / willingness to jump over hoops for no good reason etc. and that's often a good signal for most kinds of jobs.
I just started teaching undergrad CS courses after ~20 years of various non-academic jobs and it took me about two semesters to realize that almost nothing matters except how I assess students.
Two weeks of ADHD-fueled research later, I concluded that academia is actively resistant to implementing assessment reform because it would expose the utter pointlessness of most of what happens in university classrooms.
The reality is that we have no idea what most university exams measure because they are ad hoc, written by amateurs (yes, most professors are untrained in pedagogical methods) with zero psychometric validity analysis.
> almost nothing matters except how I assess students
This is a reasonable opinion after two semesters.
IMO, with more experience you should have modified your stance.
If all classes were pointless, students would leave at the same point they entered. Since we observe that is not the case, something is happening in class. Do more of that.
The explicitly stated goal of most courses is mastery of a well-defined set of domain-specific knowledge and skills. The only way to objectively determine if the course achieved that goal for a given student is comprehensive assessment. Bad assessment design, much like bad experiment design in science, is worse than useless—it actively misleads and confuses.
A well-designed, repeatable, reliable assessment is a prerequisite before one can even consider the effects of particular pedagogical methods inside or outside of the classroom itself.
at least in my experience in university - i didn't really ask this question, since it is obvious to me, but some students have asked it during lecture, or some instructors have volunteered the answer ahead of time - if you ask how to perform better on the exam, usually the instructors say, "here's what you should study." they never say, "know more." the thing i am talking about is consistent with the paper. really, your takeaway should be, exams can't see how much you know!
Because "know more" isn't actionable. Knowing more is achieved by studying but not necessarily more time spent studying, but well spent effort. Staring at the page for hours and saying "I don't understand" doesn't help. Solve exercise problems, explain the material to fellow students, discuss it with them, make mind maps, bullet point summaries, work through derivations step by step, etc. There are many techniques.
At the end of the day though what matters is what you know. Furthermore, if it's a serious subject, it shouldn't matter whether you learned it from this teacher or from another school and teacher, as long as your knowledge is correct. Knowing the idiosyncracies of this particular teacher should not factor into the grade. A serious subject can be learned on one continent and examined on another. Bullshit courses are all about learning pet peeves and hobby horses of a particular teacher.
I would usually say something along the lines of “everything we covered is on the table” or “everything we covered since the last exam is on the table” depending on the nature of the test. That’s the same message as “know more” but I think it sounds politer.
I never studied in university and yet I acheived good exam scores. If you understand a topic and ave a reasonable memory and ability to apply my our understanding not much studying is required in my experience. If you don't understand the big picture then you got to laboriously keep track of and manipulate a bunch of disparate pieces.
Social stigma against just juices people to stick to socially desirable answers; no doubt a whole bunch who self reported as non-AI users actually used AI
Sounds like the opposite of the social stigma in tech companies, where most people are under pressure to self-report as AI users when they are actually just using normal methods to get the same result (and spending the time saved however they like) (nothing against genAI for making work take more time, it's still good for business as long as the extra work can be moved around)
It would be good to see the effect of access to AI during preparation on those who previously achieved top 10% points in exams of similar topics before. I suspect they would benefit further.
My apologies for coming off as over-enthusiastic, I am currently obsessed with this study. Here is another quote:
"The negative learning effects are larger for students with higher initial achievement. The differences in the estimated full (6-10 month average) effects are substantial, with a 50% gap between the most negative effect (-24 percent) for the highest tercile and the least negative (-16 percent) for the lowest tercile. "
Not top 10% as you asked, but the closest to what you asked. My working hypothesis is that top performance is highly correlated with willingness to work hard, and AI decreases the motivation to work hard.
Interesting. Top academic performance is mostly correlated with conscientiousness (willingness to work hard and keeping track of things) and intelligence. And I'd add motivation and interest to that too.
I think if you take a physics class where the student is intelligent and intrinsically motivated through their own interest (I admit this is rare) then AI probably helps.
I think if they are intelligent, intrinsically motivated, and willing to pursue knowledge beyond what the class requires, it probably helps.
That last part is key. Intrinsic motivation doesn't mean you pursue it outside normal bounds. My kid loves soccer, its her second favorite thing in the world, she has an absurdly high tolerance for physical discomfort while playing, but she doesn't play it at home. There's other things she's rather do, such as play with her toys.
When you move the bar to something even less interesting to most kids like science, you're going to have a pretty huge falloff. You're basically selecting for kids who choose to do it in their spare time. I know a lot of smart kids (I run a boyscout troop, my wife a girlscout troop, both with lots of high achievers), and none of them do this.
I think a substantial minority of students like subjects beyond what the class requires. They go to extracurriculars to learn more interesting math and science, or are history buffs who read about it on their own time etc. These are the people who as adults move things forward and solve novel challenges so it is important to equip them well.
Personally in my own work it's pretty obvious how it's an amplifier. You waste far less time trying to find an answer to something that confuses you.
You might say "it's good to learn research skills" and that's true to an extent, but tutors have always made people better students. And AI is a tutor you can message at any time, day or night, for free.
"AI is a tutor you can message at any time, day or night, for free"
which makes it not a tutor. if it is true that tutors have always made people better students, then those tutors are definitely imposing some limits on how many answers they give you and requiring you to do some thinking. I think you could have premised your same argument by, "copying a smart kid's answers has always made people better students...."
That's why you see the split in outcomes. The fraction of kids who value self improvement are going to get smarter, and the ones who just want to slide by will fall behind.
Nothing stopping kids from asking AI to give you responses like this, it is more than capable. There is always going to be the temptation to take short cuts though.
I wouldn’t be surprised if you did and still had the same issue, but if you relied on the training data recall alone, you definitely shortchanged yourself.
I’ve had decent results with tasking a model to run pre-research and summarization as a checklist, then have a second instance of the model(s) review and then develop my learning plan learn, versus the times I just asked Claude or ChatGPT to explain something to me “from memory.”
The knowledge AI has is vast. Communicating it to you is slow and narrow baud. The chat interface itself as a medium is lacking and will need to be replaced without another medium.
Finding an answer isn't always an objective, though. I agree it often has been too much in baseline schooling.
First, there's a certain amount of baseline knowledge we'd like students to possess. Without a certain prerequisite amount of underlying information committed to memory, it gets far more difficult to achieve fluency in a topic.
But more than that, there's other skills we're trying to build: frustration tolerance, processing contradictory information, disciplined problem solving. You only really develop these skills through productive struggle. If you find a way to shortcut the productive struggle, students truggle.
> but tutors have always made people better students.
Sure. Bloom showed us that students taught with a combination of tutorial and mastery methods, one-on-one, outperform students in a normal classroom by roughly 2 sigma.
The paradox has always been-- why hasn't technology unlocked these gains for students in normal classrooms? If we could boost everyone's performance by this amount, it would be huge for society-- but society can't afford to teach everyone with tutorial methods.
Since the 1970s, we've invested in edtech towards trying to make this happen, but most of it has actually had net-negative effects as best as we can measure. AI, so far, looks to be much worse.
I think part of the answer is that a big part of what makes a conventional classroom work are social pressures. So far, it looks like AI (and edtech in general) does more to dismantle conventional pedagogy and to break down the social fabric of the classroom, than it has improved differentiation or unlocked this tutorial effect more broadly.
the difference in the vast majority of cases is that people go to a tutor with an intentional stance of cultivating a particular kind of practice, not merely to get an answer to a question. It's not technically impossible to do this with a chatbot but far less likely given that, unlike an even half decent tutor, a chatbot will never cultivate that attitude in you. The elimination of that friction is exactly why people talk to bots.
The good thing about a tutor is that he or she isn't always available and knows they won't be in the future, so they instill good habits and independence in you.
> a chatbot will never cultivate that attitude in you
Hence the whole "AI is basically an amplifier of bad and good" argument made earlier. The ones who do this because they want to understand, don't need to cultivate this at all, it naturally happens with chatbots. They don't just ask for the answer to a question, but then dig into why it's like that and what not. But the ones that don't care, now have to do even less to get the fast answer without understanding.
The scary thing is that 81% of AI users in this study were determined to be "outsourcing" their homework to the LLMs - and the rate increases the more exposure they had to LLMs.
I thought that it was generally accepted by now that homework in the volumes that it is being assigned in the modern day was not found to be beneficial in any significant way in the first place? Maybe once all students start outsourcing it to AI it might finally die like it deserves to. People these days grow up with almost no free time for themselves, it's all school, sports/extracurriculars or homework nonstop. All worker drone and no play makes for an increasingly dysfunctional society.
What I remember hearing is that exercises and practice are needed to transform words on the page to something you know and master.
What I remember being questioned is does it make sense to do those exercises as homework or would they be better in school.
Or on the flip side should school get out early like 11 or noon, like I think the german gymnasium does and have all the exercises as homework.
The US system where children get out of school at 15:30 and still have a bunch of homework seems a little lopsided someplace.
The 3 month break in-between school years is definitely questionable.
I think the answer is a complex one because it intersects with personality and neurodivergence.
Depending on who you are and your family situation any form of homework can be a real challenge. Not because of what you are studying but because of how difficult it is to sit down and do anything you aren't passionate about. Certainly anyone with an executive function disability will have that challenge.
The problem was never the volume of homework, it is the targeting and specificity. Learning happens through repetitive deliberate practice, full stop. You can’t learn through osmosis; effort is required.
Industrialized mass education has always suffered from a unit economics problem: the labor required to assign individually-tailored problem sets and manually grade them in the volume needed for most students to actually learn the material is prohibitively expensive.
> I thought that it was generally accepted by now that homework in the volumes that it is being assigned in the modern day was not found to be beneficial in any significant way in the first place?
Citation needed? I have no clue where you got this from. I hadn't even heard of it as a conjecture, let alone as something anyone accepted, let alone as generally accepted...
Before I dive into the paper: the claim was that some effect was generally accepted, not that a study was performed on it and that it drew a particular conclusion. Is this actually going to establish the former or are we just assuming that the existence of a study implies general acceptance of its conclusion?
All play and no work makes for an even more dysfunctional society. Kids need to do their homework, both to train their minds but also to develop integrity and work ethic. Being able to diligently work toward a distant goal is not something you're born with.
But I keep reading here on HN about the brave future where everyone carries around a little box that you put coins in whenever you need to know how to compose a song, solve a problem, write some code, etc, and it will give you your answer you can take credit for without actually wasting time on learning or work ethic.
I think there are better ways to teach this particular skill than homework. But I agree that practicing is very important. In hindsight I think that the concept of TD (classes where you only do exercises and have a teacher guide you if needed) in France is really good. (This is not the same thing as lab classes)
How do you expect students to learn anything when:
1. They don't do any homework.
2. All the in-class time is split between the teacher babysitting and playing social worker to problem students, and lecturing, with little to no opportunity to actually practice what they've learned?
I understand that some students don't have home environments that are conductive to doing homework well. I understand that some students are enrolled in five hours a day of extracurricular university-application-padding activities. I understand that some students have incredibly poor screen discipline and impulse control.
But I don't understand that anyone has magically figured out how to teach complicated things to students, and have it stick without them spending a lot of time practicing what they are learning.
As anyone who has tried to do something hard knows, the first step to being good at something is to spend a lot of time being pretty shit at it.
A student who has written and received feedback on 500,000 written words is going to be way better at writing than that same student who wrote 50,000, just like someone who has put 5,000 hours of focused practice into playing the piano is going to be better than my dumb ass, who has only put 100 hours in.
(If you found the solution to get good at stuff without practicing it, I'd love to get good at piano without putting any homework in on it.)
I suppose I could always place my phone that's playing Rachmaninoff somewhere under a pile of sheet music, while I mash the keys on an unplugged electric piano.
You don't have to do a lot of repetitive, boring exercises to learn piano. Once you learn how to hold your hands a little, read music, and learn where the notes are, you can simply try songs you like, and switch songs when you please. On other instruments, you might not need to read music (I've met many guitarists and drummers that cannot read music, nor do they use printed music to play - and blind folks can play instruments too)
Art is similar. You can look up skills as necessary to try to make the piece you want. Adult art learners often just start with the type of art they want to make.
In both cases, the finished piece might not be the quality you want, but in neither case are you necessarily doing repetitive stuff to learn (playing scales over and over or sketching the same bits over and over). You can retry a piece or move on - the skills will still carry over.
None of these reflect homework in school. Now, I know I graduated school decades ago, but math homework was repetitive with seemingly no real-world application and absolutely no help if I needed it. I took a math course online some years later and it was much better: Instant feedback if I got the problem wrong and instant help to walk through the problem if I needed it - then a different but similar problem was given for the homework. This was actual practice in ways traditional homework wasn't. Homework in the traditional sense isn't practice - they are all miniature tests that affect your grade.
I didn't have to practice writing - I just did the papers assigned and rushed through them. You can grade papers for other subjects on prose and grammar instead of just doing it for language courses. You don't have to read classics to read better if you just read a variety of things you are interested in. You'll read plenty of boring things for other subjects and get that sort of practice.
With most of this stuff, having actual homework isn't necessary as long as students are given time for practice. Practice is what makes you good at something, not homework, and practice can take many forms.
This is kind of damning, isn't it? "Similar or slightly higher" for spending the _same time_ means it's not, in fact, helping. At the same time as being extremely damaging to a significant number of students who, of course, use the AI to cheat. Because everything about it makes cheating easy.
Yes, it strongly suggests that AI as a learning tool is at best pretty much useless, but on average its strongly detrimental to most people exposed to it as it makes it too easy to cheat
What AI definitely is as an amplifier (in my interpretation of the article) is an amplifier of "laziness": students spending less time studying, getting by with the homework with the less possible effort, and finally achieving worse results. And the longer one uses AI for, the "lazier" they get (the more they rely on it for homework, and study less).
If it is an amplifier of "good" vs "bad", I don't know, and I do not know what that means exactly. I do not think some students are just "bad" or "lazy" by character and that explains all. I think that their environment has a big role in shaping their actions. I don't think that AI cannot be used beneficially (and the articles talks about some attempts that maybe worked well), but it requires careful considerations, guidance and intentionality from educational institutions themselves. The same goes with all kinds of technology, and AI is nothing new in that sense, but often new technologies are introduced in education without careful considerations or evidence based policies. Which is why imo articles like this are very important.
That’s why I said it will be used more for bad than good. Maybe I should have said, “it will be used for far more bad than good for the purpose of human intellectual flourishing”.
It can be used for good for this purpose. You can use it to find primary source information. You can use it to generate challenges for you. You can use it to critique your work when there’s nobody else available. Etc etc.
Intelligence and perseverance are two different traits. Smart + hard working kids will be fine. The problem is smart and lazy kids will find the easiest path, which in a world of AI leads to little actual learning. Maybe it was ever thus, but I fear the currently dwindling endangered book + teacher approach was better at getting slack asses to learn and eventually gain skills and knowledge for themselves.
The dangerous thing about amplifying is that bad people are often more willing to amplify their activities, because they don't care about the negative effects. We are seeing this now as AI companies (and large companies of all kinds) rush to secure whatever advantages they can, regardless of the negative externalities. Meanwhile people who actually care about doing the right thing get trampled.
We need to shift the incentives by adding ruinous penalties for things that are currently quite commonplace if they are done by large players. Some dude training his own AI on his own computer can scrape and train. The fine for OpenAI or Meta using a single copyrighted book without permission should be in the tens or hundreds of millions.
What we're seeing currently in our society is a "loophole inversion" where the rules have an effect mainly via their loopholes. The most profitable activity is to find loopholes and exploit them as frenetically as possible to gain as much advantage as you can before the loophole is closed, or get people hooked on the loophole so it's retroactively legalized. Entities that are big enough to do this are big enough because they have lots of money behind them. Entities doing the same kinds of things without lots of money are not really doing much harm. So the best approach is to adopt a "sliding scale" in which even tiny violations by wealthy actors result in penalties enormously greater than fairly large violations by small players.
> they don't care about the negative effects [...] rush to secure whatever advantages they can, regardless of the negative externalities
I think we don't yet know the long term view - what is harmful or damaging in the long run. Too early to tell and pass judgements. Every new technology from writing to internet had its detractors and they all pointed out negative externalities, as they seemed to appear at the moment.
You aren’t being cynical, you’re arguing with disingenuous entities with money on the line. We already know it’s used primarily for the negative case. Everyone who ever intended High School or College knows this.
Definitely had a sheltered public school experience, took AP stats my senior year and realized all the top students were sharing answers from an earlier period through text messages. After figuring this out I too joined in on the action, then shortly after I quickly deskilled.
Suppose it's good to learn how elastic the brain is, in both directions, at a young age where it doesn't matter.
Not really. I would rather not find out later in life, with way more responsibilities, that when you purposely neglect mental skills they atrophy quite fast. People already have a hard time noticing that they age, being able to truly self asses is a skill that few, including myself here, possess.
It's a familiar story. If you copy/paste Wikipedia and turn it in, you learn nothing. If you skim Wikipedia, hit up the references, the consult other sources, then synthesize your own thought, you probably get farther faster than you would without it.
We could probably cook up thousands of examples of the same problem (YouTube DIY tutorials, GPS navigation, etc.)
LLMs make significant mistakes frequently and smart people have no way of judging those mistakes outside their domain expertise. They are also sycophantic and great at being an echo chamber which makes people feel smart even if they are not.
So I think the burden of proof is on you to prove that they somehow amplify intelligence, it seems highly unlikely.
Most domains have some kind of internal consistency/theory building you can do. A smart person can certainly notice inconsistencies when trying to learn something. In fact they're likely to be points of confusion that the smart person will dive into just to try to make sense of things, even if they don't suspect the LLM is at fault.
People who think they are smart often think they can get by in any domain without knowledge and context, but at a certain point this fails and knowledge is required, not just intelligence.
We're talking about learning here, i.e., building that knowledge and context. People also refer to more than one source, and I think you overstate the degree to which LLMs make mistakes in current year. Human teachers also make mistakes, and it's typical for a bright student who's paying attention to correct even small details.
I don't think anyone is claiming they're a shortcut to mastery. Personal tutors aren't a shortcut to mastery either. Both simply have the advantage of being able to deeply know where the student's knowledge frontier is, so they can tailor their answers to that, make connections and draw analogies to things the student has mastered (while pointing out distinctions), etc. The student still needs to put in the work.
In the sense that they are not programmers and do not deeply know e.g. performance engineering, sure. But I can still give one a program and ask it to optimize it, and it will turn it into a zero allocation streaming SIMD implementation that's orders of magnitude faster than where it started. It may not "know" how to do any of that, but it does it anyway.
Pretty much this. With the internet and my websearch-fu, I can find some facts on a lots of things. And while I do for some questions I have, I try to never interject answers when I don’t have a modicum of mastery in the domain. Mastery is the formation of an internal model that guide your actions and thinking. It’s not merely knowledge and intelligence.
Smart people know LLMs confabulate and tell them they’re Absolutely Right! Smart people don’t want to be embarrassed by trusting the hallucination machine and revealing their gullibility to others.
Smart people are handsome, charming, and always make the right decisions even on things they don’t know about. And if they ever are deceived, well I guess they were no true smart person.
Really widely reported in normie news and on reddit, hallucinations and sycophancy. Stand by that whether you’re a petroleum engineer or a bartender, if you’re clever you were probably either well informed or curious enough not to blindly trust the plagiarism machines.
Why would a smart person go to an LLM for an answer they cannot judge or test, be succeptible to flattery and sycophancy rather than picking up on the emotional manipulation and being suspicious/sceptical of the interaction, or looking for support from an echo chamber rather than a Socratic opponent?
All of those sound like flaws and defects of dumb people?
> Why would a smart person go to an LLM for an answer they cannot judge or test, be succeptible to flattery and sycophancy rather than picking up on the emotional manipulation and being suspicious/sceptical of the interaction, or looking for support from an echo chamber rather than a Socratic opponent?
Great summary of the flaws with LLM "research"/"reasoning". It's always trying to con you, and I question the literacy and intelligence of the people who can't see this.
Many people think they are smart and are taken in by LLMs. Also many genuinely smart people are unable to judge their competence in other areas (eg Nobel Laureates making pronouncements in fields they know nothing about).
Just one 20 GW AI data center which is being built with gas powered turbine generator power plant, will be the largest fossil fuel power plant in the world... How exactly is that the same as one coal mine
I dunno, concentrating the environmental impact of 20GW of gas based power generation would be the equivalent of 6-9 Million cars running... All in the space of a corner of a small city
It helps smart people be barely more effective and helps dumb (more importantly, people who do not value effort, people who are lazy, people who are self absorbed) people shit out endless streams of worthless tokens that can swamp out everything.
Raising the noise floor like this only makes it that much harder to find "Smart" people, which we were already doing terrible at.
I use Claude every single day, but this is such a bad tradeoff. Maybe it will help me standup a quick fix when that is needed. Maybe it can help me dig through documentation to find relevant bits and figure out the unstated assumptions underlying it. Maybe it helps me generate test cases.
Meanwhile, my day to day life is now noise. All social media is noise. All content is noise. Slop pours onto me from all directions. Writing more test cases isn't helping me.
Am I smart? Am I dumb? I don't care, right now I'm deafened
All I've noticed is AI creating a perverse incentive to make everything as complicated and bureaucratic as possible, so only the people that know how to leverage AI to cut through it ever succeed.
I'm using Claude at work myself and am impressed with the product, but notice that this is the only reason I need to use it at all. Our product pages were shit to begin with, now they're AI-generated and somehow even worse. Our procedures are incomprehensible spaghetti with enough arbitrary context switching to give a sadistic Soviet municipal administrator an erection at the thought of watching anyone try to actually follow them.
Use AI to create inefficiencies, then use AI to bypass them. Those who can't do the latter will struggle to survive.
> It helps smart people be barely more effective and helps dumb (more importantly, people who do not value effort, people who are lazy, people who are self absorbed) people shit out endless streams of worthless tokens that can swamp out everything.
... including ultimately stuff needed to make the smart people.
> Raising the noise floor like this only makes it that much harder to find "Smart" people,
It does give us a new heuristic, though: people who are willing to completely cut generative AI out of their lives (cold-turkey, if you ever started using it) are a much smaller group of, predominantly thoughtful, people. You do have to give up Claude to be part of this group, but from what you say, that's no great loss, and no longer being deafened is worth it.
This has considerable advantages over conventional elitism, because the barrier-to-entry is negative in almost all cases.
The one exception I've found is assistive tech, where the state-of-the-art is so poor that vibecoded slop is genuinely an improvement over the state-of-the-art, and in many cases the tooling simply isn't available to make your own assistive tech (unless you want to bootstrap an entire networked computing environment, which isn't very helpful when you want to do your online banking and do not, in fact, work at your bank).
But there are not many principled exceptions where you could seriously argue that the trade-off is worth it. Take mathematics, for example, which we often see touted as a "good use-case" of generative AI. The primary advantage of generative AI in mathematics is being able to search though a vast corpus of ivory towers and inconsistent terminology (without proper attribution) to locate and connect ideas that can help solve problems. The deficiency this is addressing is elitism, inadequate communication, and inadequate indexing within academic mathematics. This problem is entirely created by the academic mathematicians, and has been known for nearly a century (per https://en.wikipedia.org/w/index.php?title=Nicolas_Bourbaki&...):
> Bourbaki was founded in response to the effects of the First World War which caused the death of a generation of French mathematicians; as a result, young university instructors were forced to use dated texts. While teaching at the University of Strasbourg, Henri Cartan complained to his colleague André Weil of the inadequacy of available course material, which prompted Weil to propose a meeting with others in Paris to collectively write a modern analysis textbook.
To my knowledge, this is the only organised project to clean up and improve mathematical communication. Everything else (Metamath, Mizar, AFP, Lean) is yet another ivory tower. The Wikipedia article on this topic (https://en.wikipedia.org/wiki/Mathematical_knowledge_managem...) risks deletion as non-notable, that's how little anyone's actually trying. They made their own bed, and generative AI will only provide a brief respite from having to lie in it. (I was surprised how many other "compelling" use-cases evaporated when I applied this razor to them: the sibling comment https://news.ycombinator.com/item?id=49392265 points out one such.)
Vibe-coding assistive tech which doesn't yet exist, as a temporary scaffold to improve the quality-of-life of yourself and others in a social world dominated by non-essential access barriers is, to my knowledge, the only exception to this principle that can be justified. If you treat people who make other excuses, or who don't even bother with excuses, as not worth listening to, you lose little – and doubly-so, if you make your stance clear, so that others know the "cost" of gaining your attention.
AI does not make me "smarter" in the important thing I am working on.
In software particularly, what AI does do is give me back my time from the drudge work that I don't care about. Keeping my build system configured and my tests up to date and my documentation synchronized is a good use of AI because I mostly don't care about how crummy the result is as long as it "works".
All AI did here is exposed an old problem in education. Kids are expected to get everything perfectly for the first time but a lot of them don't so the class gets easier next year to keep up pass rates.
We need to restructure the system to treat failure as a signal instead of a disaster. Grades should come from hard randomized exams with unlimited retakes so one bad day won't hurt you. Homework should be optional material for self study, evaluated by teachers if you choose to do it but never forced.
Hold back students for individual classes instead of a whole grade so failing one can't ruin your social life and teachers are more willing to do it. F students will realize they have to study, start actually learning and then pass on the second time. No big deal. It happened to my friends in college, no reason they can't do it in high schools.
Discipline is a skill and it's one you have to get from experience. If you try and force kids to study when they don't want to "for their own good" you're not actually helping them. Everyone needs to find their own path. Let people fail.
> Homework should be optional material for self study
At Caltech, homework was assigned but had no bearing on your grade. The grades were based on the midterm and final exams.
But not mastering the homework usually resulted in flunking the exams. There were "retch" sessions after each homework assignment that was staffed by a grad student, and the purpose was to help the students understand the homework problems. I knew only one person (Hal Finney) who was so smart he didn't need to do the homework.
I learned the hard way that the path to success was:
1. never miss a lecture, no matter what
2. take notes by hand during lecture
3. do the homework on time, and make sure you understand every problem. Take advantage of the retch sessions.
Likewise at Oxford - only the final exams counted (though you had to pass first year exams to continue to the second year).
But you had tutorials each week, and if your tutors thought you weren't doing enough work they could set you exams mid-course called 'penal collections' and if you failed them you could be thrown out. They were rare but definitely not unknown.
Naval Nuclear Power School had (maybe still does) this model 20 years ago as well. Pass rate for my class cohort was 28% over the entire 2 year pipeline.
We didn't call them retch sessions, and they were taught by the instructors though. We also were encouraged to peer tutor and since we were all restricted to one building the homework was always group work allowed.
Also had badges to track time spent in the building for required study hours, though some people gave up and just slept at their desks when they started sliding down the grade scale and the hours racked up.
Generally I think I did 30 hours of studying/homework (went up and down depending on what was being taught, but was around that) a week (for 12-15 hours of actual lecturing), with some of my friends putting in 50% more. Generally the only day we weren't there was Saturdays. Most of the day Sunday was usually spent in class preparing for the next week.
I think only a third or so of my class wound up graduating. Caltech doesn't flunk people out. What happens is students will transfer to easier schools, or drop out, or decide driving a truck is more to their liking.
I'll add one more thing to this, given that my experience as a math major at SIU was that each hour of lecture was expected to carry four hours of studying: read and summarize the course's texts (additionally, for math courses, attempt the problems) before lecture.
I found that (1) I didn't need to take anywhere near as many notes during lecture and (2) I could ask way, way, way more relevant questions.
This also helped tremendously when it came to studying for the actuarial exams.
As a student of a top-tier French Master's degree, I consulted with a teacher to deal with exhaustion and to ask for a class rescheduling for my case. Explaining my situation, the teacher looks at me and interjected:
—Waitwaitwait. You... you went to all lectures!?
And he was right. Rather than following the curriculum, I should have developed my taste for various engineering topics and only used the classes as entertainment.
I think the first semester, or at least half of that, you should go to every lecture until you develop taste for what's useful. By the time you do your master's you should have a good sense for what classes are serious and have useful lectures and where you can just cram last minute because it's a bullshit class and where you're better off studying from a textbook.
That only works with competent lecturers. I can speak for my experience at a more mid university, where that would apply to about two thirds of the classes. For the remaining third, going to the lectures felt genuinely counterproductive and you could actually feel yourself losing braincells listening to the confused nonsense or classes held in English for Erasmus students by someone who could barely speak it coherently. It was just a complete waste of already little available time.
So the endgame was figuring out what the tests in previous years looked like (cause it was likely gonna be a copy paste affair), do a targeted study run for those exercises and 9/10 you would pass.
When I was there, a long time ago, exams were timed and were usually open book open note. Blue books were filled in. You were trusted to adhere by those rules, and most students did their exams in their dorm rooms.
The evidence that the students honored the rules was some exams resulted in a 50% failure rate.
As for me, I went there because I wanted to learn the material. I did not care about getting a diploma. (Mine is in the basement somewhere.) I did not take any "easy A" classes, because I wanted a return on my time and tuition investment. (Though, easy A classes were hard to find at Caltech.) I wasn't even going to attend graduation, but my parents showed up and I attended to please them.
The classes, year by year, were dependent on mastering the previous year's classes. So if you cheat with AI, you're digging yourself into a bigger and bigger hole. Caltech rewires your brain. If you don't learn the stuff, you're going to be one of those EEs who carries around a card with V=A*R, V/A=R, V/R=A printed on it.
Unlimited retakes would be an enormous amount of work for professors/TAs/teachers.
Optional homework is often a disaster. At best, students would do it right before an exam and the goal of education is not to just pass exams. They’d probably still get a lower score than if they did the homework when they were supposed to.
What I think is better is to have a due date, but just make the maximum 10% each day it is late. So after 2 days, the highest score you could receive would be 80%.
I liked that system because it gave some flexibility with deadlines while still encouraging you to turn things in on time.
Late work policies like this can be a logistical nightmare at scale. Rather than play the penalty-over-time game, have hard deadlines that are inflexible. Students don't have to engage in mental gymnastics with their scores and instructors don't need to track this stuff.
My view is this: a student can turn in an incomplete assignment. Work they completed get evaluated and graded. Anything not completed gets zero credit. Homework should be an investment in effort over time anyway. Unfortunately, procrastination is a deeply rooted problem that usually kills this.
> If you try and force kids to study when they don't want to "for their own good" you're not actually helping them. Everyone needs to find their own path. Let people fail.
This simply does not work for most, as pupils are too immature. You put sunscreen on children, keep them away from the street and brush their teeth. Having everyone re-learn the same stupid lesson is wasteful, it's better to institutionalize this knowledge in the form of mandatory participation.
What you described about exams is like how I approach most certification exams: go through a test bank of questions until I get 80% then take the real test.
The study was done in China, where making the class easier to improve pass rates isn't really a thing, as the Gaokao operates more like a stack ranking where it doesn't matter much how well you did in an absolute sense, only that not too many others who did better than you are competing for the same spot. Unlimited retakes are possible in theory, except each costs you a year of your life and except for some extreme cases of repeat test-takers, most people are going to give up after just one or two bad results. Homework is definitely not optional, but going home from school is. (Due to the hukou system, children often have to go to school far from where their parents work, so boarding schools with teacher-supervised self-study are common.) I don't see how holding back for individual classes is supposed to work considering scheduling conflicts.
You put way too much faith in kids' desire to learn and their ability to make responsible choices.
> Homework should be optional material for self study, evaluated by teachers if you choose to do it but never forced.
My wife is a teacher and her school does this. The result is that nobody does homework. The kids who need the extra practice don't get it and they fall into a spiral of failure and apathy. This is especially bad for subjects like math that build on themselves.
IMO the better solution is to require and grade shorter homework and provide students with optional, supplementary assignments that can be used to make up for missed credit on homework for the same material. That way students who grasp the material quickly can demonstrate and move on while others who need practice are naturally encouraged to get it without being permanently punished for struggling initially.
Homework is no longer useful in the era of AI. Options are: treat kids like they are able to make responsible choices, and accept that much more of them will fail; or, keep the charade of grading AI-"assisted" homework and push kids farther along the track, even though they definitely don't know enough.
I much prefer the first.
School shouldn't be impossible, but you shouldn't have a charade of schooling.
It's not necessarily useless. If you make assignments short, provide some time in school to work on it, and do not allow students to use devices with access to general-purpose "AI", then there is a good chance most students will do the work like they're supposed to. Some will still cheat, but it's foolish to reorganize everything around the few who stubbornly refuse to learn.
> If you try and force kids to study when they don't want to "for their own good" you're not actually helping them. Everyone needs to find their own path. Let people fail.
this rhetoric is pretending to be an alternative to coercion. IMO the ideas you are talking about are well trodden and are still coercion nonetheless.
I could be mistaken but my interpretation is that "studying for your own good" is being contrasted with "studying so you don't fail". If you actually fail students, they'll have a second motivation beyond trusting some authority figure's advice. Namely consequences.
You are missing the point. Getting a certificate for all of the programs you passed and still graduating could completely change the course of a person’s life. Community college has certificates and associates degrees to let people start working when they reach a practical limit in their general education.
It's all cute but naively glances over what this all boils to - competition.
Education system, contrary to popular belief/name, isn't tailored to educate but to select winners and losers which then will be picked on the job market.
That's why it seems absurd when you think about it as an institution that aims to educate. That's because that isn't the real purpose of it. The purpose is to stratify and classify early.
Schools do not operate in a vacuum; they serve as credentialing gatekeepers for a hyper-competitive capitalist job market. If everyone could easily retake exams until they got an A, grades would lose their primary utility for employers and universities: differentiation. Society relies on schools to provide a neat hierarchy of candidates that for one reason or another thrived in difficult environment of adolescent schooling.
The system often prioritizes compliance, endurance of boredom, and social maneuvering over actual critical thinking precisely because those traits align with corporate hierarchies.
> compliance, endurance of boredom, and social maneuvering
Compliance and endurance of boredom I can somewhat agree with (thought not to a great extent), but social manoeuvring? As long as a student completes their classwork and homework, and does well in their exams (none of which mandate any social interaction apart from team projects), they easily pass through the school system. In fact, I used to hear about how the school system disproportionately benefited studious, shut-in students over street-wise, socially-savvy ones.
Moneyshot: "The results are eye-opening. After six months, pupils using ai saw their average homework score rise by 18% across all subjects. The time they took to complete each assignment fell from an average of 64 minutes to 45. But come exam time, the same students scored 20% below their classmates who had not called on ai’s help. Homework scores once predicted exam performance; now those who score highest are, perversely, more likely to do worse in exams."
Study design: "David Stromberg of Stockholm University and Victor Lei and Wu Yanhui of the University of Hong Kong set out to fill the gap. They tracked 27,000 pupils aged 12-18 in China, where ai adoption has been fast. Around 80% reported using models such as Doubao and DeepSeek; the other 20% formed the control group."
As with most training the journey is the point, not the destination.
That said, I think smart use of AI could help. It could explain concepts in a way that might help you understand better, it could probe your knowledge in a more dynamic way by tailoring questions, and so on. This requires the AI be restrained by some harness, not free to write down the answers for you.
When I was in college, we had office hours with the prof and the TAs, we had group study sessions, and private tutors. Or you could ask your friends. All of those required going somewhere at a specific place and time and asking someone else to give up their time for you.
AI gives you a way to get the same help without asking another human. Say of that what you will, but not everyone was comfortable asking other humans for help even back then.
You also have your book and a wealth of online resources, which also don't require interacting with someone, and which have the same information the AI is trained on. Fundamentally nothing is different. We don't see positive outcomes from students using AI as an aid even if hypothetically it could be possible.
Back when I was studying there were topics which I had a really hard time grasping. Having a personal tutor that I could ask specific questions and have a back and forth with would have helped a lot I think. At least the few times I did have such an opportunity that was definitely the case.
Kids are, by and large, lazy however. Hell, adults are lazy, this isn't even an indictment on kids or even people in general, I think our bodies & brains are simply wired to seek the path of least resistance from an evolutionary POV.
Kids won't be using LLM tooling as a personal tutor following some sort of Socratic method, they'll ask it to solve their home/coursework for them and blindly copy/paste the answer. Hell, they'll just manually copy down what's on their screen if copy/pasting isn't possible for whatever reason.
Obviously exceptions exist, but I'd wager from being an ex-kid myself the type of kid who would genuinely use these tools for actual proper self-tutoring would be an extreme rarity.
I'm way beyond homework, but I do appreciate the compliment on seeming young!
I meant: what's going to stop the average lazy student from ignoring whatever harness the university recommends and instead use an unrestricted LLM, thus learning nothing?
> what's going to stop the average lazy student from ignoring whatever harness the university recommends and instead use an unrestricted LLM, thus learning nothing?
Nothing until they start failing exams and maybe seeing the error of their ways. My suggestions wasn't for you, it was for the smart student who wants to use AI to enhance their learning but not have it do all the work.
But cheating isn't new. People have been cheating in school for centuries. It's easier now, but the consequences were always the same. At some point the chickens come home to roost and you pay the piper.
> But cheating isn't new. People have been cheating in school for centuries. It's easier now, but the consequences were always the same. At some point the chickens come home to roost and you pay the piper.
My concern is that with LLMs and above average bullshitting abilities, the chickens might never come home to roost.
I think that is fine, certain things are becoming less important to learn and certain things more important. With the pace of change it is likely to continue shifting.
I really don't see how anything but social skills are rising in prominence. Perhaps twenty years from now everything will depend on your connections as any "hard" problems will be readily solved by AI (seeing as few problems are ever really novel).
College stopped being about getting an education a long time ago.
It's just a low pass filter for the job market. Students aren't actually interested in learning anything, they have been condition for a long time to jump through the hoops.
It would be much better if we actually returned to exam and performance based evaluation of students, and stopped giving everyone As just for trying hard. But parents look at college fees as buying their kid a job opportunity, and so grade inflation has basically ruined the SNR of academic performance for all but the lowest performers.
It depends on college attendance. In developed countries, it is a low-excluding - or the inverse of a low-pass - filter as most graduate, while in developing countries it is a high-pass filter as few have the resources to reach an undergraduate level of education.
For some of the classes my kids were in, the lessons and homework time were reversed.
Learning the material (using the text book and videos) was what they had to do at night and class time was spent working through problems or applying the material in some way.
Flipped classroom was a popular method when I was teaching math. Some students found it very helpful. The trouble was getting the students to do the reading before class. If students came in unprepared they struggled to keep up.
I did this for the last college programming course I taught (2025) as an anti-LLM measure. Every class started with a very short closed-book auto-graded multiple-choice test on the reading. I was careful to specify which was the core reading needed to learn the material (on the quiz) and which was extra (not on the quiz - I think many professors assign this extra reading to everyone, thinking of their strongest students). I think this developed an expectation in the students that doing the reading was easier than finding a way not to.
It's especially hard when multiple subjects try to do it, because it can add enormous amounts of home-time expectations. And many families can't dedicate that much time.
I do like it though, and I like that college generally leaned that way more (and was better balanced, as it had substantially less time in classrooms, so you could study during the day).
Popular yes; based on good quality research… no, not really. It’s really pretty unfortunate timing with the AI boom, because the field of education research is not very high on the science quality totem pole. And the field as a whole was and is also slow to incorporate neuroscience findings, too.
This is typical at the college/university level - when I attended 20-25 years ago, we were expected to learn the material through readings outside of class, and then class time was spent on discussion, application, questions, and working through problems.
IMHO there's a lot to recommend this style. A lot of students learn better when they can wrestle with the material at their own pace, on their own time, in their own setting. It quells a lot of anxiety about "I'm not following what the teacher is saying, am I stuped, will I look bad in front of all my peers?" And then class time can be spent identifying holes in your knowledge and getting instant feedback from an expert, which is where they are most useful.
Although obvious, I think it's worth stating that this method mostly (only?) makes sense for higher education, where students are conscious, involved and engaged in their curriculum.
I think it makes sense as soon as students have the fundamentals down. You have to be able to read, listen, make connections, think critically, devote sustained attention to a task, and manage your time effectively. Some students develop these skills in elementary school, and then starting in middle school, the flipped classroom approach can work. Some adults never develop these skills. Although arguably, people that never develop these skills will likely be sitting there zoning out in a traditional classroom lecture and not actually learn anything.
What do you think about it? I think that the ability to study by yourself is probably the most valuable thing one could get from school and most kids usually don't. Your setup would suggest that demanding that a kids know how to study by themselves is going mainstream...?
Your answer - middle school - I find it extremely young for this, which is extra interesting.
IMHO, my kid was assigned way too much homework. One year I eventually told her to do some amount per class (I think it was 45 minutes) and then be done with it so that she would have time for sports and family stuff. She was spending 4 or 5 hours every night on homework and this was in 4th grade.
That year I sent an email to her teacher to let them know that I'm the one responsible for her not doing all the work and if that's a problem, we need to talk. It wasn't a problem.
The one class that was flipped was a relief because she could breeze through the lesson much faster than would normally be spent on it in class and the amount of time on exercises was limited to the class time slot.
Here's research showing the same exact effect 15 years ago, except it's about homework copying: https://news.mit.edu/2010/homework-copying-0318
Students who copied their homework more saw their grades go down more throughout the semester. A/B students became C/D students.
A name people use for this phenomenon now with AI is cognitive offloading.
Education sustem is not about the actual knowledge, but about training various aspects of the brain during kids development.
So even memorizing literature have benefits, even though it doesn't bring any career use.
If you skip this, well your brain won't develop as much at period of life it's able to do so.
I see this as a memorize-vs-understand fallacy many repeat.
It is really a combination: you need to memorize a large corpus of information to operationalize knowledge — understanding a bunch of theorems in mathematics does not help much (even if you are able to prove them when you see them) if you can't remember the boundary conditions they hold under.
Or having good understanding of foreign language grammar won't help you if you do not memorize words that you need to express your thoughts.
Good literature will show you some of life's challenges and potentially let you think through them in a non-stressful situation (other than "I've got to finish the last 200 pages by Monday" ;)).
> if you can't remember the boundary conditions they hold under.
At the outskirts of one's knowledge this is inevitably the case. My impression it is still useful to know that a certain implication is possible - and one can look up the exact conditions.
This is why I said operationalizing: you need to memorize enough to either be able to know what to look up, or know it outright if you need to be fast(er).
Maybe. As the nature of assistive tools changes, so does pedagogy. Calculators are the easy example since most of us are (still) familiar. But there are others that were just as pronounced -- e.g. log tables and slide rules were transformational and instrumental at one time, and you've (probably) never used them. If you had used slide rules to solve your homework problems then were excluded from using them on exams, you'd see a similar effect.
AI is a big change; pedagogy is going to change too.
I mean, does it? The way I see it, the best LLMs have to offer is infinite patience (until they inevitably and unpredictably start to confabulate, which should be a gigantic red flag, anyhow), I don't see how LLMs can be used to explore teaching paradigms that haven't been explored before. We know pretty well from centuries of empirical experimentation how children's brains develop under different stimuli. It's not exactly something the software industry needed to "hack".
Using the forklift as a spotter and to assist in loading weights increased gains. Then again, a human can do all those things, and provide real human connection.
If you structure learning in such a way that makes learning just a means to some end, and overindex on that end being the ultimate goal, that's what you get.
This is a pedagogical problem that AI merely exposed. Educators need to figure out How to make students choose the scenic route instead of having them optimize for the most efficient completion of a task.
School has always been about creating compliant workers, not educating people. They will double down on "performing the right things" and "morals" while the economy will continue on its K shaped path as AI gets more capable.
Yea, also the programs are too different. My psychology bachelor felt like a tea party. My computer science master was 60 hours per week during the hardest courses (which were supposed to go for 20 hours per week - so I could only do one course if it was at this level). My game design master felt more like we were dumped into an art school masquerading as a psychology/CS master but it really was art school. My information science bachelor felt like the only "normal" study program.
All of this was at university of which 3 of them were at the same university.
Especially the artsy game design program definitely did not feel like it was preparing me to be a cog in some giant corporate wheel.
I was thinking how tricky it would be to learn to program now, honestly a sense of laziness is one of the reasons I learned programming - so that I could automate away repetitive tasks. Our generation might be the last people who can actually program properly, everyone will just learn some basic architecture and point the AI in vaguely the right direction.
Seems if any assignments are to be handed out they should be structured in such a way that takes into account the potential of LLM use. Additionally to make the output (hardcopy? handwritten?oral presentation?) in such a way that even if LLms are used at the very least some part will be forcibly undertaken by the students in preparation of finals/real life.
Open book exams existed for a long time and even take home assignments where the net was usable to complete the tasks.
Difference was one actually had to do the reseach and learn something along the way regarding research skills or the topic itself.
LLms just make the automation and laziness with a lower barrier to polished output.
I full heartedly recommend anyone who wants to learn, to use the Socrates skill, and then build a daily 1 hour curriculum that you will tackle. It's painful and not pleasant, but it works quite well when to don't have a teacher and want to learn something really hard and complicated
I am waiting for AI Viva Voce (“AVV”). I am really surprised we aren’t hearing more about this idea?
Try it for yourself by making a prompt like this (adapt as needed):
Pretend I am an undergraduate student of Computer Science. I am learning about early microprocessors from the 1970s. I want you to ask me an examination question as if you were doing a viva voce exam with me, to test my understanding of concepts. I want you to receive my answer and then based on what I said I want you to ask me a more specific question to probe my understanding. Repeat this interaction up to 5 times. Then grade my understanding so far, by giving me a pass, merit, credit, or distinction. Can you explain how you arrive at the grade based on my answers and your expectation of undergraduate knowledge of microprocessor theory?
I've tried stuff like this, but the problem is that you don't know if it's answers are correct, and more importantly, you don't know if its questions are on base or not. Plus its too easy to go off on tangents, especially when you ask it questions to clarify.
You can get some more success by writing down a framework beforehand, but openended dialog is still not a good way to learn with AI, from my experience.
Good points.
I’ve tried this a few times and I’ve been impressed by the dialogue. Given this was a very simple prompt, I am convinced a more precise version could be developed, including a rubric, a library of course content, starter questions and other guidance to the AI. I haven’t seen anything like this and schools / universities seem to be “struggling” with AI, which has me wondering why not embrace LLMs for learning and assessment, like this?
I mean, everyone I know has been using socratic dialogue for months to aid understanding, this seems like a variation of that. The issue is that if the answer is too easy to obtain it's hard to have the discipline not to cheat.
Meetings are still like exams, if you can't craft a line of thinking on the spot it doesn't come off well.
You don't need a final answer but you can't just answer "I don't know" otherwise you risk being slimed (shout out to anyone who still remembers "You Can't Do That on Television")
I find people’s on the spot reasoning abilities considerable worse than even cheap LLMs. This “fast” mentality is what causes stupid decisions because we need to “keep moving” (To where? Wrong meeting!) Answers, answers, decisions, decisions, quick, quick! next quarter In hindsight we need to do things differently. Answers, answer, quick quick! Time is of the essence! If you don’t start producing sound within 5 seconds we will all spontaneously combust!
People can also ask questions that it is unreasonable to expect people to know on the spot too and after a while the role of the questioner in the meeting can be questioned. :)
I prefer to share a video, ask people to watch it, and have them send me questions. That way I can spend time preparing answers to those questions, and then then meeting can be about discussion, not information delivery or recall.
Asking people to be conclusive on the spot is a recipe for acting on bad data. There are too many biases related to stature, they interfere with accuracy.
I can see that. Personally, I use that for people who like to go down tedious rabbit holes and want to free up the rest of the attendees to go back to work.
True, but only for things that that your colleagues don't expect you to know already. Remember the "I'll circle back" meme about Jen Psaki? This response can only be used sparingly before it's seen as a sign of underpreparedness/inability to do back-of-the-envelope thinking.
The answer is "accepted", in the sense that it won't get you fired. But the meeting plows on, and decisions are made in absence of the answer to the question! — and never revisited once the answer is known.
And like 90% of the time it comes up, it's because the data contradicts the decision.
Every conversation you have with a person is an "exam." Not being able to extemporaneously express yourself or reason analytically without running to Claude seems undesirable.
In my professional career, my clients value the depth of ready knowledge that helps me use my tools with particular insight and finesse -- a kind of knowledge that exams demonstrated far more effectively than homework when I was back in school.
Stand-alone bottom-up observations (even if valid) obscure the fundamental problem that AI (or "AI" if you like) is inducing systemic change across human civilization.
If cohorts are graduating into the workforce with decreasing stand-alone skills, that implies the economic value-add of labour is shifting to AI, which implies broadly falling standards of living, falling political power, and an ongoing transfer of power and wealth to capital.
If students are taking shortcuts with AI when their brains have the highest capacity for learning they may be permanently weakening their prospects. It doesn't seem likely that whatever "AI management skills" they incidentally learn will make up for, over their lifetime, the economic value loss they suffer due to reduced cognitive skills.
On an individual basis, in the next few years, this might not matter. Across decades, economies and polities, it will.
> Stand-alone bottom-up observations (even if valid) obscure the fundamental problem that AI (or "AI" if you like) is inducing systemic change across human civilization.
Students have been cheating and taking shortcuts on their work long before the current generation of AI tools existed (and probably for as long as graded exams have existed).
When you lower the barrier to taking the shortcut, you get more people to take it who wouldn’t otherwise have done so. If cheating becomes incredibly easy, lots of students who would have otherwise been honest take the easy way out. Moral conviction gets harder to keep as doing the right thing becomes less rational.
Doing the right thing never becomes less rational. It may suck compared to the temptation towards the alternative, but part of growing up and being responsible is realizing there is a reason the shortcut isn't supposed to be normalized.
And civilizations do fall. The bronze age civilizations did collapse. There was something dark about the dark ages. There is no law of nature which says that things will be okay, and plenty of examples where things have fallen apart.
Just to add to this, if AI is as epochal a change as its boosters say (and I am open to that argument), then surely it must have the potential for awful consequences, in proportion to its potential for good.
> The social aspects are more important than the exam sitting most of the time.
To spell it out more clearly: every single aspect of your career is an exam. The interview is an exam, quite literally. The day-to-day responsibilities (meetings, planning, problem-solving, collaborating) are parts of the exam. If you're failing at those or automating them away, then what is your role? The assignments (solve X bug or add Y feature) are parts of the exam too. Failing to do these things will mean that, yes, you are failing the exam.
And yet if nobody likes you your career is going to stall. If you’re annoying, if you smell bad, if you dress poorly, if you’re fat and ugly, just like in school you’re going to have worse career outcomes.
And I could also call most of those things assignments instead of exams and nobody would bat an eye.
Your analogy was both thoughtless and stands up to no scrutiny
Even if we grant that education is to prepare for employment, which a lot of people disagree with, it does not follow education should do the same thing as what's done in employment.
Your employers want those assignments done. That’s why they pay you. School assignments belong in the trash can and you pay for someone to look at them before throwing them away.
What if learners used teacher AIs such that the AIs would keep track (and submit to class teacher) of the scoring for different "mini exams" it put the learner through, so in effect better preparing for exams.
I know this study is focused in China but one thing I'd like to understand is the nuance in what kind of student is likely to always reach for ai for homework help as I think (especially in the US, can't speak for other countries) the varying powers that be that can determine the quality and type of education you can receive (does your country have means to buy textbooks per student or shared, do you have funds to give students ipads to take home or no, what sets the bar on how far curriculums can go, are teacher and faculty pay and incentives simply tied to student pass rates, etc.) and those do more in shaping what "use AI responsibly" will look like and why it's so different.
AI itself is not the problem imo. The core issue is basically outsourcing your entire brain to the LLM and never trying to solve a problem yourself
When I was in school there were multiple instances where I would be stuck on a problem for nearly an hr but I always learnt something from it. They key I think is to accurately identify when to and when not to use AI
Glad to have scientific results on this, though in my view you could get this from first principles. Homework is onerous but it forces you to learn the material and get it into a configuration that works in your head, which you then validate with the exam.
as a fun aside here from the abstract it does come down to how you use it:
> AI users who maintain similar homework completion time as non-AI users experience small learning losses.
If the pure completion time of homework goes drops significantly with an LLM tool, I suspect these students are spending the extra time turning the material over in different ways to internalize, which is interesting.
If others studies tend toward this result, a question comes to my mind: it is a problem with AI or our educational system? or both? I would like to have some comments...
Now, what this tells me is that LLM AIs can supply the knowledge of the moment for the prepared human person, and the person does not need to learn that knowledge, for a extremely large class of problems. Which is fantastic for capitalism and business in general, and very very bad for humanity in general.
The article does not relate AI usage for homework to exam score drops.
It tries to link them together with no obvious cause and effect.
The score could have dropped due to worse environment at schools or kids who were stuck at home studying online and now in physical locations could not adapt fast enough (meaning, possibly just temporary drops). We do not yet have enough metrics gathered yet.
This type of article bashing AI for the root of any problems, I find it appalling without any evidence.
Honestly I wouldn't demonize AI in education. When we were students we were forbidden from using calculators, then the internet and now it's become the norm. It's the same with AI, I think. AI has already become a part of our lives. The important thing is just teaching children how to use it correctly.
Unlike AI, calculator use has been studied and shown to be useful for math education. Our understanding of these things evolved over time, however the results of this study are concerning.
It's an ad hominem attack, I know, but why am I not surprised that the Economist loves this?
True or not, it supports a narrative of "kids these days...", of a morally bankrupt educational system, of indulgent adults spoiling children, of just desserts to the wicked and lazy.
Again, I'm not saying the article is right or wrong. I'm saying I'm skeptical because it so tidily aligns with the Economist's editorial world view.
I used to skip homework in high school because it was boring and cut into my personal life outside of school (including my job) but I would always do very well on tests.
Then I stopped caring about tests too.
I did the same thing through college. The only reason they passed me and I got a degree is that I built the school’s website and I built personal ecommerce sites for the head of art and his wife to sell their paintings.
I haven’t done shit since like 7th grade.
Worked at Facebook, Apple, Microsoft, same behavior there - did basically nothing for them while making thousands off my games on App Store.
I think we need to accept that AI causes cognitive decline, like Alzheimer's. For an Adult this may be manageable, but in a young brain, this is detrimental.
People using books did in fact lead to cognitive decline since remembering could be offloaded. But that didn't stop the spread of books, was the point I tried to make.
Same for social media. The negative effects have been apparent for years.
I don't think we can go back to a world without books, without social media and without AI.
Comparing it to Alzheimer's is an allegory. The cognitive loss is already well documented. I might also compare it to Chronic traumatic encephalopathy. But I think more people understand what Alzheimer's is.
> I think we need to accept that AI causes cognitive decline
That is catastrophic because I heard several CEOs that said that we cannot do anything about it and that the technology is here to stay. So, I suppose that we will just need to adapt to a world with declining mental cognition for millions of children and a future with diminishing production capacity because ... checks notes ... this quarterly profits.
I got graded homework even in a few college courses and thought it was dumb, but then I realized it's always in the low-tier classes where they can't trust the students to study on their own. Those who skip homework hurt themselves by doing poorly on exams, but that's not enough to stop them. This ruins the curve in ways the instructor can't really fix unless they're prepared to fail half the class, which they aren't.
BTW / Offtopic:
I have read some weeks ago, today if you are hosting a party for your kids birthday, parents are preparing "give away bags" for handing over when leaving the party
The often underworked (most are autograding these days and reading old powerpoints rather than do real curriculum development) and (for the quality) overpaid teachers/professors will mass riot against this because it means they might have to actually teach. We might even get yet more anti-boy treatment than already exists among them as a social class in retaliation.
Please also abolish PowerPoint and force oral exams while you are at it.
Teachers are also luddites as a social class. Math teachers cried about calculators yet everyone of them who refused to adapt did disservice to their students (especially anyone doing statistics). My state's education system is utter trash despite being well off economically because of strong teachers unions keeping them from undergoing even a tiny bit of scrutiny.
They do not deserve the social credit/grace that they get. They are ultimately gatekeepers, whose extremely biased decisions in grading/treatment of their flocks decide which kid grows up on the streets and which kid grows up to eat lobster thermador every day.
Teachers also should not be disciplinarians. Children with discipline problems bad enough to disrupt a whole class should be thrown out of the class room as basically the only thing a teacher does for "discipline". The "right to education for the shit kids means we can't do that" framing is such trash for those who want an education and will inevitably be disrupted by a small minority of terrible students.
Also it's interesting that one of the only other things Max Stirner wrote about besides philosophy was education and his myriad problems with it (given his history as a school teacher).
I like school and think teachers are good and that your comment is the nonsensical ranting of a lunatic and as a result don't feel compelled in the slightest to read whatever it is that you linked.
This is the answer. Many (all?) European schools don’t give homework (this is partly because of the way the grad programs work). It’s on the students to study in order to pass the (difficult) exams.
There is no such thing as a European school. Each country is different. And if the country is a federation, each state might be different. And we have homework in most European countries I’m aware of.
Ah you’re talking about university. There isn’t a European-wide rule or custom for homework, it will very likely be up to each university and/or teacher to decide if they want to give homework or not
Seems this did effectively abolish homework if AI is now doing it all for them. The result seems to be that fewer students pass the exams, not more.
As with any skill, practice is necessary to achieve competence. If homework is no longer a viable way to force students to practice, you'll need to find some replacement, you can't just abolish it and expect there to be no negative consequences.
It is what all my European peers reported to me during grad school, and you should care because it is an example of a system which solves this problem successfully (at the university level).
Oh, university level? The article is about 12-18yo. American grad school already doesn't have graded homework, and undergrad sometimes does or sometimes not.
I know someone who pasted all their six sigma certification questions directly into an AI tool and they got a 98% on their exam. Seems like it would help their exam score too if they were allowed to use it on both the homework and exams /s
Should have let the kids use AI during exam too, later in life they will be solving tasks with AI not manually. Just make the test questions difficult enough even with AI.
I see our current approach like a driving school training its students at fast running.
This feels like the calculator debate all over again.
Yes, using a calculator leads to worse outcomes in your learning.
But also, yes, everyone always carries a calculator on their person 24/7 these days and you'll never be as fast and as accurate as a calculator.
I think it would have been kinda cool if you could have unlocked the privilege of using a calculator in school through good grades in math class. Prove you can do what the tool can, then you get to use it. Would have also provided at least one incentive for students to get good grades in at least one class.
Offloading problems makes it so you do not have to solve them.
If you don't have to solve a problem, your brain does not have that work to do.
Your brain working less means it has less opportunities to train.
The consequence is worse cognitive abilities which is tied to worse learning outcomes.
-
From your link:
> Sustained calculator use in Grade 4 appears to hinder the development of basic skills in average students.
It noted the following positive outcome:
> apparently improves the average student's basic skills with paper and pencil, both in working exercises and in problem solving
But this only measured the outcome for solving math. Yes, offloading problems will make it easier to solve problems, at the cost of not training your brain, which leads to worse learning outcomes across all subjects.
When you get out into the real world, what will matter is how effectively you use AI. When you are done with school, you can take the "exam" using AI. The problem is that teaching is trying to teach skills that are no longer relevant and is always behind what the latest going on in the industry. Every time without fail you got interns, who after two years at stanford would learn more in the 3 months on a google internship than in those two years at stanford. They learn useless shit.
My own experience is that college is fun. You can spend the time wisely, but school stuff, is not what makes you money.
Make student loans dischargeable in bankruptcy and make colleges under write them. The problem will get solved quickly.
Way before AI, the problem was very similar - universities do not teach practically useful stuff. Its been a permanent complaint about the education system, really, at least for the last 50 years.
AI just gave the students a way to dodge the slog.
But university was never intended to teach the bleeding edge. That would really be impossible in practice. Pre-phd, it is supposed to teach ways to efficiently attack a problem. you can take a bunch of "play courses", and succeeding at any of those requires pretty much only that one skill.
Nowadays, the challenge they face, is to keep teaching "problem attack methods" in a way that can't be trivialized by AI.
Though fundamentally, if you go to university, and evade learning the one thing you can learn there - thats your loss.
Guess what this is probably fine. AI helps people do their work on a daily basis and probably makes them memorize less things. That’s how the world is moving. You don’t need to be able to memorize something for an exam to show mastery of a topic. And we shouldn’t organize our society around memorizing things for exams.
What is a doctor? The person recommending next course of action based on best practices. What is the best practice? Is it a globally decided on thing, a locally tested thing that has proven to be better? Is the global thing just the most popular because someone wrote it on reddit and it got millions of views (but isn't actually good).
If our society is organized on just delegating all authority to "the best answer" we can drop all jobs - and only the surgeon has some authority over the machine. When we have a surgery robot, then they can go to. And at some point we'll be recommending and forcing surgery for athletes foot because some kind of best practice or viral post going out of control and a bunch of openclaw bots randomly decide to make tons of recommendations online for amputation for athletes foot and GPT 8.6 agrees this is the way to go.
OR we can somehow decide there is a boundary, there is some room for human memory and decisions.
I actually think having a single source of truth would not be terrible.
Let me explain.
All podiatrists are not the same. All physicians in their specialties are often specializing in their own niche and are up-to-date on that niche.
I've had lisfranc ligament injury. My podiatrist wanted to perform a surgery quickly, which in 90% of cases is advised. I decided to seek a 2nd opinion because I realized she did the X-rays wrong, as she did them non-weight bearing, which is useless if there are no fractures.
The specialist was excited to see me, and up to date on all the lisfranc research. I've had a really bad sprain and he said conservative treatment should be sufficient. It took months but I healed up.
I've had quite a few injuries from playing sports and often found doctors to be "wrong" simply because their knowledge was out of date.
I am confused as to how this is an argument for a single source of truth. It seems like you've presented an argument for the opposite. If there were a single source of truth, there would be no point in seeking a second opinion - it would be the same as the first. If the first opinion was correct, excellent. But if the first opinion was incorrect (and no AI model is correct 100% of the time), then you're straight out of luck.
The problem is that the first source is incorrect because she doesn't deal with lisfranc injuries frequently enough and her knowledge is not good enough. I've had to go do my own research to find a guy who specializes in lisfranc. Most patients would not.
However, if you went to an AI agent which has access to medical research, you would get the most up to date research.
Why would I seek a 2nd opinion if the initial doctor knew what to do?
> I actually think having a single source of truth would not be terrible.
We should establish a single practice of care, MedicineGPT. It may be a rough start but over time the gene pool will adapt and the human race will live harmoniously with the machines (sorry, with the GPUs). Plus our insurance premiums should finally go down.
It would be better than most physicians now. It can quickly crunch many pages of medical history and medical research, this something a physician cannot do. They have intuition but sometimes are not equipped to deal with complex problems.
> What is a doctor? The person recommending next course of action based on best practices. What is the best practice? Is it a globally decided on thing, a locally tested thing that has proven to be better? Is the global thing just the most popular because someone wrote it on reddit and it got millions of views (but isn't actually good).
Ironically I had a friend who went to med school and in his own words roughly 90% or more of the students there are heavily relying on GPT. They even had a few instances where people were using it to cheat during exams. And supposedly during the clinical years of uni if they don't know something they literally just go to a separate room and start asking GPT for answers. Make of that what you will.
This is too simplistic. There are disciplines, like Anatomy, where you're basically just memorizing bones, glands, and other parts of the human body. For organic chemistry, students need to memorize most of the krebs cycle diagram for their exams (because doing so and drawing it lets them reference it later).
Thinking is infinitely more complex than "critical thinking" vs. rote functions. You want students to get to critical, eventually, but the lower orders on bloom's taxonomy serve important roles, too.
Course design built around multiple choice questions, essays, and a lot of the historical assessments that we're used to are largely not great for learning in the first place—–they're used because they can be auto-graded or score assigned quickly. AI, conversely, is kind of revealing the limitations of these outmoded learning design practices.
Think of the meme going around right now from Milennials to Gen Z: "I wrote a 5 page essay on a book I never read!" <- They're saying this to AI generated essays as if there's any difference in the critical thinking that comes out of an exercise like that (read: absolutely none).
AI is a new tech—so of course people are still figuring out the limits of it. In 10 years a lot of the paradigm will be set. Until then, students are unfortunately kind of in the thick of it.
>Think of the meme going around right now from Milennials to Gen Z: "I wrote a 5 page essay on a book I never read!" <- They're saying this to AI generated essays as if there's any difference in the critical thinking that comes out of an exercise like that (read: absolutely none).
As somebody that read every mandatory reading book, I wish I had had the critical thinking skills to not read them. In retrospect, they were a waste of time and made me absolutely hate reading.
How do you differentiate between mastery of a topic and mastery of using an AI in an exam setting?
Open book tests and allowed cheat sheets already solve the memorization-is-not-required gap. It seems like allowing use of AI pushes that closer to thinking-is-not-required. And if no thinking is required, then how can you prove personal mastery?
If no thinking is required, why do you need thinking in the first place. Seems useless. If an ai can do it, why you need a human to be able to do, human should be doing something the ai can't.
I had the privilege of being educated in several countries and cultures. This might come across as "your education system sucks", and that's essentially what it boils down to, but on the upside I'm truly sorry for you and will oppose a comment to your cynicism. Good exams test your ability to express coherent thoughts about the concepts you learned, and let you generalize upon them by having you use and combine them to solve cases and problems that were not those seen in the teaching material. You can even teach and test history that way, and make it much more insightful and captivating than a laundry list of dates and events (as I suppose was your case?)
Same, but all the exams I ever took mostly tested your memory more than anything else. I don't think it's possible to test people at scale any other way.
You can't play an instrument without muscle memory.
Why are commercial airline pilots required to spend hundreds of hours in simulators? So that when things go wrong they don't have time to think, they just react. And their reaction will be the correct course of action because that's what they have drilled on.
How many times have you gone "Oh, this new thing I have just encountered reminds me of this other thing?". How can you make such leaps without knowing that other thing?
How do you show mastery of a topic where you neither understand fundamentals nor have skills to do a task without help?
Mastery is about demonstrating skills. Skills take practice, hands down.
AI can help us build better tools, provide tutoring, answer broad ranging questions, help relate concepts, and do reasoning for us. All of those things can be good in that they can help us acquire skills faster, e.g. with better practices and exercises. But it cannot replace actual time spent doing a thing.
Wanna learn knitting? You're going to have to knit some stuff.
A lot of expertise is actually knowing things about the domain. You can't make a good judgement call on a topic without getting relevant knowledge in your head to make that decision. You can't even lookup the relevant knowledge without knowing what knowledge is important to make the decision, so reference material is not enough.
Memorizing stuff is good for understanding. Of course memorizing without understanding doesn't really help, which is what a lot of student last-minute "cramming" boils down to. But without concrete and memorized knowledge, any abstract reasoning skills become quickly impossible.
Wrong. Memorizing without understanding is pointless. And I'd argue that rote memorization does nothing to guide ones understanding. Because if you actually understand a subject deeply enough you can literally derive almost anything that would need to be memorized on the fly, and if you can't (say constants, or spec information, or anything else) you can simply surgically look it up online (or via an LLM).
> Wrong. Memorizing without understanding is pointless.
GP said exactly the same thing.
I agree with GP’s broader point, which is that you cannot have understanding without a specific set of baseline facts. The student studying only those facts and never achieving the understanding they are meant to support is a tragedy of our education system.
There are people who need to understand microbiology at a specific level and they cannot derive it all on the fly. They can achieve better memorization through understanding but I personally have found rote memorization can be an essential first step in some disciplines.
A good analogy to caching and memory lookups. You store in your head only the most valuable immediately necessary information, the rest can be outsourced.
This is not, in fact, fine. The point of education is to learn things, and these results show that kids are not learning things as effectively when they use LLMs.
> You don’t need to be able to memorize something for an exam to show mastery of a topic.
Yes, you do need to memorize and know things about the subject in order to be a master of it. What is the definition of "mastery" that you are using? What are the Things being memorized?
It's critical to understand the fundamentals on your own and that is different from AI being used in public. Sure school doesn't have as much value as it used to but it was always more about social credit or a certification that you took the time and effort to understand the world under pressure (obviously not true for all disciplines) or goals attached preferably in group settings.
This is something that AI cannot replace. Viewing things as memorizing is the wrong take.
The kids who used AI and studied for a similar amount of time as the non-AI high performers apparently had similar (slightly higher) performance. The kids who used AI and did poorly used the AI to do the homework for them (this is what the article says).
I believe AI is basically an amplifier of bad and good. I’m cynical about the world and assume it will be used more for bad than good, but I don’t doubt some of the best people in every field will be using AI to amplify their work in good ways.
It's not so clear that AI is an amplifier.. The paper has some fascinating analysis on this topic:
"At the other end of the distribution, AI students who spend more than 65 minutes on their homework receive homework and exam scores similar to those of non-AI students, suggesting that these students do not use generative AI for homework assignments. However, this group consists entirely of students who adopted generative AI no more than Öve months. Six months after adoption, no AI student spends more than 65 minutes completing their homework (see Figure A5). This is consistent with the gradual process of learning how to use AI tools. It also suggests that AI crowds out the highest level of e§ort."
"Interestingly, in the range of 50-65 minutes, the median and the interquartile range of exam scores of AI and non-AI students are similar. This implies that, in the range where AI students and non-AI students have overlapping homework times, students who spend the same amount of time completing homework on average receive similar exam scores."
"This pattern shows that students who spend the same amount of time on homework learn similarly, with or without generative AI. In other words, generative AI reduces time spent learning for the majority of AI students but not learning efficiency for those who spend the same time studying as the non-AI students."
Right I was not clear enough about separating my opinion from the findings.
I'm curious what happened in the source, that a copy/paste exposed such odd artifacts.
"e§ort" == effort
"Öve" == not sure. Maybe "over five"?
I wondered the same. ff and fi are common ligature letter pairs, maybe the original is in an encoding where those are represented with separate characters, which then correspond to § and Ö in whatever encoding we find here?
They look more like OCR errors than encoding errors.
It was copied from a pdf, maybe LaTeX generated. All the ligatures are wrong -- not sure where the problem lies, the pdf itself, pdf reader, or the clipboard on Linux.
Depends who's using it. Like many tools the force multiplier depends on the operator.
I'm confident it's an amplifier for people who know how learning works and already do a lot of it, successfully. However the level of "learning fluency" I'm talking about isn't reached for many until late college or grad school, and sometimes not at all. So I'm not surprised by the quoted results for 12-18 year olds.
> Depends who's using it. Like many tools the force multiplier depends on the operator.
How would you prove/disprove this assumption without falling into a True Scotsman fallacy?
Fair question. In theory you could do an experiment where the subjects were grad students or professors, or top performing college students. Give them some fixed amount of time to understand some new subject on which they'll be tested, only 1 group has access to an LLM with appropriate context for the learning task, etc. That's just one idea...
This experimental setup would only tell you whether this demographic (grad students or professors, or top performing college students) benefits from access to LLMs for acquiring a new knowledge or skill.
It would not prove that "the results depend on the operator". To prove this (and figure out the traits that make a proficient operator) you would either need a massive dataset to which you'd apply some machine learning to figure out the correlations, or at least you'd need a hypothesis on the traits you want to test.
Do left-handed people perform better with LLMs? Analytical thinkers? Dyslexic people?
It's not enough to just claim "some individuals will perform better with these tools" without any notion of who does, and whether it's possible to become one of these people, and what is the expected gain in this group.
Otherwise, sure, we could sell unsecured chainsaws as a tool for making ice sculptures and observe that "some people" are indeed more proficient at not getting their face cut off ; but if the tool is being marketed to schools, such a vague statement is not helpful in making a case for it.
Yes, I'm using "high-performing college students" or "phd students" or whatever as a proxy for "people skilled at learning". It's not perfect but I think it'd be good enough. You could do lots of refinements on the idea. You could try to do some kind of "pre-test" that more directly tested "meta-learning skills".
My only point was I think it's possible to do. I personally think it's pretty clear. And there's nothing special about AI. In rural places, you could replace it with "access to books" or "access to a good tutor" and so on.
I also think you'd see the same effect with "motivation" (if you could test it), with performance on standardized tests, and with grades. AI will boost the better (by these metrics) students more.
And people who want to learn. Most teenagers lack agency in their studies. They aren't in high school because they love it but because they have no choice.
Sometimes things are just common sense pure and simple.
> Sometimes things are just common sense pure and simple.
The discussion is about "AI", so common sense is out the window. These people's professional reputations depend on addict-level "AI" usage remaining socially acceptable.
well most university exams are designed to measure how much you study. so we didn't really need a study to tell us, "Exams continue to measure what they are designed to measure."
they're not designed to measure general aptitude, or function as admissions criteria, or screen for job applications, or any other numerous things they are used for.
there can be many questions of pedagogy. one of them is, what do our exams measure and how do we use them? professors who say, "My exam is designed to measure who studies, not be used for all these other purposes that they are actually used for" - I don't buy it. It's the same as late night comedians saying they are not responsible for solutions, even when spending 90% of their air time making political jokes.
THIS is the pedagogical issue, that pedagogy has NEVER caught up with the scope of responsibilities. This is acute in STEM - I mean, the humanities departments are generally pretty well run, all things considered, in this regard. Generative AI is accelerating that pre-existing crisis.
The fact that exam scores are correlated with how much you study is not the same as exams only reflect how much you study. Two students who study the same amount could have very different exam scores. The reason that there still is a strong correlation between exam score and time of study is because if all other things being equal, students who studies more have higher exam scores.
i'm not saying they reflect how much you study. they reflect a lot of things, including that. but you ask the people who write the tests, they're going to say, how much you know or how much you study, but nonetheless, they are limited. i agree with you. that's part of my point.
let's imagine a different study. we instead compare AI-users and non-users on a Wechsler (IQ-adjacent) test.
overall, it would be surprising if AI usage impacted your Wechsler scores. someone has done this study and the impact is quite quite small. BUT. do we care? We don't use Wechsler scores for admissions, we don't use them for jobs, we don't use them for... are you getting it now? A Wechsler family test is measuring something real, just like a university exam measures something. But what do we USE them for? Wechsler and a typical university exam are, in some senses, EQUALLY vague in terms of their fitness for purpose for answering a question like, "should we hire this guy?"
Like there is an association between IQ and earnings but it is actually surprisingly small! There is an association with math education and earnings and it is also surprisingly small. And consider how many people get by just fine without using a single piece of math education once they have finished school - like what if maximizing your earnings isn't all that it is about? Are you getting it now?
The issue isn't the AI usage. I can find tests that are immune to AI usage. The issue is using tests for things that they are not designed for. We pick and choose, for some subtle but nonetheless pervasive cultural reasons, which tests we use for which purpose, and very frequently, not because they are calibrated for the chosen purpose. This is coming from someone who scores very well on all these tests, and have kids, so I have a very strong incentive to buy into the status quo, and I'm telling you: academic testing has been fucked up for a long, long time.
Education is a complex topic. I don't claim to understand it, nor do I think we can sort this out in a hacker news thread. What I reject is the simplistic claims like "exams don't mean anything". There are certainly exams that are badly designed, but there are also exams that are well designed. A well designed exam can look very badly to different people, based on what they know and where they come from. It's a bad idea to think exam scores as the single metric of education quality, it is equally bad to reject them completely, because chances are any alternative measurement people come up with are going to be worse.
I think this deviates from the original topic. The article's intent is to argue that AI helps with learning itself, while you question whether this sample can serve as a reference. But what if the target of this sample is a group of serious college students? The article questions AI's enhancement of learning ability, not whether that exam has a real purpose.
You're right that academic exams also measure something along the lines of instruction following / obedience / willingness to jump over hoops for no good reason etc. and that's often a good signal for most kinds of jobs.
I just started teaching undergrad CS courses after ~20 years of various non-academic jobs and it took me about two semesters to realize that almost nothing matters except how I assess students.
Two weeks of ADHD-fueled research later, I concluded that academia is actively resistant to implementing assessment reform because it would expose the utter pointlessness of most of what happens in university classrooms.
The reality is that we have no idea what most university exams measure because they are ad hoc, written by amateurs (yes, most professors are untrained in pedagogical methods) with zero psychometric validity analysis.
> almost nothing matters except how I assess students
This is a reasonable opinion after two semesters.
IMO, with more experience you should have modified your stance.
If all classes were pointless, students would leave at the same point they entered. Since we observe that is not the case, something is happening in class. Do more of that.
You’re conflating the course and the classroom.
The explicitly stated goal of most courses is mastery of a well-defined set of domain-specific knowledge and skills. The only way to objectively determine if the course achieved that goal for a given student is comprehensive assessment. Bad assessment design, much like bad experiment design in science, is worse than useless—it actively misleads and confuses.
A well-designed, repeatable, reliable assessment is a prerequisite before one can even consider the effects of particular pedagogical methods inside or outside of the classroom itself.
> well most university exams are designed to measure how much you study.
Huh? They're designed to measure how much you know. They can't see how much you study, nor would they have reason to be interested.
How much you know at the moment of the exam. Not how much you truly know.
at least in my experience in university - i didn't really ask this question, since it is obvious to me, but some students have asked it during lecture, or some instructors have volunteered the answer ahead of time - if you ask how to perform better on the exam, usually the instructors say, "here's what you should study." they never say, "know more." the thing i am talking about is consistent with the paper. really, your takeaway should be, exams can't see how much you know!
Because "know more" isn't actionable. Knowing more is achieved by studying but not necessarily more time spent studying, but well spent effort. Staring at the page for hours and saying "I don't understand" doesn't help. Solve exercise problems, explain the material to fellow students, discuss it with them, make mind maps, bullet point summaries, work through derivations step by step, etc. There are many techniques.
At the end of the day though what matters is what you know. Furthermore, if it's a serious subject, it shouldn't matter whether you learned it from this teacher or from another school and teacher, as long as your knowledge is correct. Knowing the idiosyncracies of this particular teacher should not factor into the grade. A serious subject can be learned on one continent and examined on another. Bullshit courses are all about learning pet peeves and hobby horses of a particular teacher.
I would usually say something along the lines of “everything we covered is on the table” or “everything we covered since the last exam is on the table” depending on the nature of the test. That’s the same message as “know more” but I think it sounds politer.
You become a person who knows more, by studying more about the things you need to know.
I never studied in university and yet I acheived good exam scores. If you understand a topic and ave a reasonable memory and ability to apply my our understanding not much studying is required in my experience. If you don't understand the big picture then you got to laboriously keep track of and manipulate a bunch of disparate pieces.
Self reports of AI use is worthless
Social stigma against just juices people to stick to socially desirable answers; no doubt a whole bunch who self reported as non-AI users actually used AI
Sounds like the opposite of the social stigma in tech companies, where most people are under pressure to self-report as AI users when they are actually just using normal methods to get the same result (and spending the time saved however they like) (nothing against genAI for making work take more time, it's still good for business as long as the extra work can be moved around)
It would be good to see the effect of access to AI during preparation on those who previously achieved top 10% points in exams of similar topics before. I suspect they would benefit further.
My apologies for coming off as over-enthusiastic, I am currently obsessed with this study. Here is another quote:
"The negative learning effects are larger for students with higher initial achievement. The differences in the estimated full (6-10 month average) effects are substantial, with a 50% gap between the most negative effect (-24 percent) for the highest tercile and the least negative (-16 percent) for the lowest tercile. "
Not top 10% as you asked, but the closest to what you asked. My working hypothesis is that top performance is highly correlated with willingness to work hard, and AI decreases the motivation to work hard.
Interesting. Top academic performance is mostly correlated with conscientiousness (willingness to work hard and keeping track of things) and intelligence. And I'd add motivation and interest to that too.
I think if you take a physics class where the student is intelligent and intrinsically motivated through their own interest (I admit this is rare) then AI probably helps.
I think if they are intelligent, intrinsically motivated, and willing to pursue knowledge beyond what the class requires, it probably helps.
That last part is key. Intrinsic motivation doesn't mean you pursue it outside normal bounds. My kid loves soccer, its her second favorite thing in the world, she has an absurdly high tolerance for physical discomfort while playing, but she doesn't play it at home. There's other things she's rather do, such as play with her toys.
When you move the bar to something even less interesting to most kids like science, you're going to have a pretty huge falloff. You're basically selecting for kids who choose to do it in their spare time. I know a lot of smart kids (I run a boyscout troop, my wife a girlscout troop, both with lots of high achievers), and none of them do this.
I think a substantial minority of students like subjects beyond what the class requires. They go to extracurriculars to learn more interesting math and science, or are history buffs who read about it on their own time etc. These are the people who as adults move things forward and solve novel challenges so it is important to equip them well.
Personally in my own work it's pretty obvious how it's an amplifier. You waste far less time trying to find an answer to something that confuses you.
You might say "it's good to learn research skills" and that's true to an extent, but tutors have always made people better students. And AI is a tutor you can message at any time, day or night, for free.
"AI is a tutor you can message at any time, day or night, for free"
which makes it not a tutor. if it is true that tutors have always made people better students, then those tutors are definitely imposing some limits on how many answers they give you and requiring you to do some thinking. I think you could have premised your same argument by, "copying a smart kid's answers has always made people better students...."
That's why you see the split in outcomes. The fraction of kids who value self improvement are going to get smarter, and the ones who just want to slide by will fall behind.
Nothing stopping kids from asking AI to give you responses like this, it is more than capable. There is always going to be the temptation to take short cuts though.
Yes, but is it the correct answer?
I wasted several days trying to have AI teach me containers. I would have been much better off just reading the docs.
Did you initially ask the tool to read the docs?
I wouldn’t be surprised if you did and still had the same issue, but if you relied on the training data recall alone, you definitely shortchanged yourself.
I’ve had decent results with tasking a model to run pre-research and summarization as a checklist, then have a second instance of the model(s) review and then develop my learning plan learn, versus the times I just asked Claude or ChatGPT to explain something to me “from memory.”
The knowledge AI has is vast. Communicating it to you is slow and narrow baud. The chat interface itself as a medium is lacking and will need to be replaced without another medium.
You can have the AI create pedagogical web apps and quizzes for you on the fly
Then it’s teaching the wrong thing in a fancier way.
Finding an answer isn't always an objective, though. I agree it often has been too much in baseline schooling.
First, there's a certain amount of baseline knowledge we'd like students to possess. Without a certain prerequisite amount of underlying information committed to memory, it gets far more difficult to achieve fluency in a topic.
But more than that, there's other skills we're trying to build: frustration tolerance, processing contradictory information, disciplined problem solving. You only really develop these skills through productive struggle. If you find a way to shortcut the productive struggle, students truggle.
> but tutors have always made people better students.
Sure. Bloom showed us that students taught with a combination of tutorial and mastery methods, one-on-one, outperform students in a normal classroom by roughly 2 sigma.
The paradox has always been-- why hasn't technology unlocked these gains for students in normal classrooms? If we could boost everyone's performance by this amount, it would be huge for society-- but society can't afford to teach everyone with tutorial methods.
Since the 1970s, we've invested in edtech towards trying to make this happen, but most of it has actually had net-negative effects as best as we can measure. AI, so far, looks to be much worse.
I think part of the answer is that a big part of what makes a conventional classroom work are social pressures. So far, it looks like AI (and edtech in general) does more to dismantle conventional pedagogy and to break down the social fabric of the classroom, than it has improved differentiation or unlocked this tutorial effect more broadly.
> If you find a way to shortcut the productive struggle, students truggle.
I was assuming this was a typo and was thinking about making a joke about it, but it does appear to be slang that fits the context:
https://www.urbandictionary.com/define.php?term=Truggle
> 1. the standard of perpetual intellectual failure made by an individual.
> Truggle; the standard defenintion of a person who is a failureat everything.
Was it a typo or did you actually mean this?
> Personally in my own work it's pretty obvious how it's an amplifier.
The answer is obvious: you're doing homework type activity (learning is not a goal), not exam type activity (learning is the goal).
the difference in the vast majority of cases is that people go to a tutor with an intentional stance of cultivating a particular kind of practice, not merely to get an answer to a question. It's not technically impossible to do this with a chatbot but far less likely given that, unlike an even half decent tutor, a chatbot will never cultivate that attitude in you. The elimination of that friction is exactly why people talk to bots.
The good thing about a tutor is that he or she isn't always available and knows they won't be in the future, so they instill good habits and independence in you.
> a chatbot will never cultivate that attitude in you
Hence the whole "AI is basically an amplifier of bad and good" argument made earlier. The ones who do this because they want to understand, don't need to cultivate this at all, it naturally happens with chatbots. They don't just ask for the answer to a question, but then dig into why it's like that and what not. But the ones that don't care, now have to do even less to get the fast answer without understanding.
The scary thing is that 81% of AI users in this study were determined to be "outsourcing" their homework to the LLMs - and the rate increases the more exposure they had to LLMs.
The "slightly higher" performance is based on statistically insignificant samples (between 4 and 20 students, depending on the context, out of the total population of 26,000): https://bsky.app/profile/benjaminjriley.bsky.social/post/3mt...
I thought that it was generally accepted by now that homework in the volumes that it is being assigned in the modern day was not found to be beneficial in any significant way in the first place? Maybe once all students start outsourcing it to AI it might finally die like it deserves to. People these days grow up with almost no free time for themselves, it's all school, sports/extracurriculars or homework nonstop. All worker drone and no play makes for an increasingly dysfunctional society.
What I remember hearing is that exercises and practice are needed to transform words on the page to something you know and master.
What I remember being questioned is does it make sense to do those exercises as homework or would they be better in school.
Or on the flip side should school get out early like 11 or noon, like I think the german gymnasium does and have all the exercises as homework.
The US system where children get out of school at 15:30 and still have a bunch of homework seems a little lopsided someplace.
The 3 month break in-between school years is definitely questionable.
I think the answer is a complex one because it intersects with personality and neurodivergence.
Depending on who you are and your family situation any form of homework can be a real challenge. Not because of what you are studying but because of how difficult it is to sit down and do anything you aren't passionate about. Certainly anyone with an executive function disability will have that challenge.
The problem was never the volume of homework, it is the targeting and specificity. Learning happens through repetitive deliberate practice, full stop. You can’t learn through osmosis; effort is required.
Industrialized mass education has always suffered from a unit economics problem: the labor required to assign individually-tailored problem sets and manually grade them in the volume needed for most students to actually learn the material is prohibitively expensive.
This study seems to have proven the opposite. Doing the work without AI resulted in higher exam scores.
> I thought that it was generally accepted by now that homework in the volumes that it is being assigned in the modern day was not found to be beneficial in any significant way in the first place?
Citation needed? I have no clue where you got this from. I hadn't even heard of it as a conjecture, let alone as something anyone accepted, let alone as generally accepted...
https://www.tandfonline.com/doi/pdf/10.1080/00220973.2012.74...
Before I dive into the paper: the claim was that some effect was generally accepted, not that a study was performed on it and that it drew a particular conclusion. Is this actually going to establish the former or are we just assuming that the existence of a study implies general acceptance of its conclusion?
All play and no work makes for an even more dysfunctional society. Kids need to do their homework, both to train their minds but also to develop integrity and work ethic. Being able to diligently work toward a distant goal is not something you're born with.
But I keep reading here on HN about the brave future where everyone carries around a little box that you put coins in whenever you need to know how to compose a song, solve a problem, write some code, etc, and it will give you your answer you can take credit for without actually wasting time on learning or work ethic.
I think there are better ways to teach this particular skill than homework. But I agree that practicing is very important. In hindsight I think that the concept of TD (classes where you only do exercises and have a teacher guide you if needed) in France is really good. (This is not the same thing as lab classes)
How do you expect students to learn anything when:
1. They don't do any homework.
2. All the in-class time is split between the teacher babysitting and playing social worker to problem students, and lecturing, with little to no opportunity to actually practice what they've learned?
I understand that some students don't have home environments that are conductive to doing homework well. I understand that some students are enrolled in five hours a day of extracurricular university-application-padding activities. I understand that some students have incredibly poor screen discipline and impulse control.
But I don't understand that anyone has magically figured out how to teach complicated things to students, and have it stick without them spending a lot of time practicing what they are learning.
As anyone who has tried to do something hard knows, the first step to being good at something is to spend a lot of time being pretty shit at it.
A student who has written and received feedback on 500,000 written words is going to be way better at writing than that same student who wrote 50,000, just like someone who has put 5,000 hours of focused practice into playing the piano is going to be better than my dumb ass, who has only put 100 hours in.
(If you found the solution to get good at stuff without practicing it, I'd love to get good at piano without putting any homework in on it.)
> I'd love to get good at piano without putting any homework in on it
I'm pretty sure there is a way to fine-tune this [0] to auto-complete away any of your hesitations or mistakes :)
[0] https://news.ycombinator.com/item?id=49373456
I suppose I could always place my phone that's playing Rachmaninoff somewhere under a pile of sheet music, while I mash the keys on an unplugged electric piano.
You don't have to do a lot of repetitive, boring exercises to learn piano. Once you learn how to hold your hands a little, read music, and learn where the notes are, you can simply try songs you like, and switch songs when you please. On other instruments, you might not need to read music (I've met many guitarists and drummers that cannot read music, nor do they use printed music to play - and blind folks can play instruments too)
Art is similar. You can look up skills as necessary to try to make the piece you want. Adult art learners often just start with the type of art they want to make.
In both cases, the finished piece might not be the quality you want, but in neither case are you necessarily doing repetitive stuff to learn (playing scales over and over or sketching the same bits over and over). You can retry a piece or move on - the skills will still carry over.
None of these reflect homework in school. Now, I know I graduated school decades ago, but math homework was repetitive with seemingly no real-world application and absolutely no help if I needed it. I took a math course online some years later and it was much better: Instant feedback if I got the problem wrong and instant help to walk through the problem if I needed it - then a different but similar problem was given for the homework. This was actual practice in ways traditional homework wasn't. Homework in the traditional sense isn't practice - they are all miniature tests that affect your grade.
I didn't have to practice writing - I just did the papers assigned and rushed through them. You can grade papers for other subjects on prose and grammar instead of just doing it for language courses. You don't have to read classics to read better if you just read a variety of things you are interested in. You'll read plenty of boring things for other subjects and get that sort of practice.
With most of this stuff, having actual homework isn't necessary as long as students are given time for practice. Practice is what makes you good at something, not homework, and practice can take many forms.
This is kind of damning, isn't it? "Similar or slightly higher" for spending the _same time_ means it's not, in fact, helping. At the same time as being extremely damaging to a significant number of students who, of course, use the AI to cheat. Because everything about it makes cheating easy.
Yes, it strongly suggests that AI as a learning tool is at best pretty much useless, but on average its strongly detrimental to most people exposed to it as it makes it too easy to cheat
What AI definitely is as an amplifier (in my interpretation of the article) is an amplifier of "laziness": students spending less time studying, getting by with the homework with the less possible effort, and finally achieving worse results. And the longer one uses AI for, the "lazier" they get (the more they rely on it for homework, and study less).
If it is an amplifier of "good" vs "bad", I don't know, and I do not know what that means exactly. I do not think some students are just "bad" or "lazy" by character and that explains all. I think that their environment has a big role in shaping their actions. I don't think that AI cannot be used beneficially (and the articles talks about some attempts that maybe worked well), but it requires careful considerations, guidance and intentionality from educational institutions themselves. The same goes with all kinds of technology, and AI is nothing new in that sense, but often new technologies are introduced in education without careful considerations or evidence based policies. Which is why imo articles like this are very important.
> had similar (slightly higher) performance.
The data point around 80 minutes seems like noise to me. Looks like there isn't enough data/students who spend that much time and also used AI.
It would be nice if AI was a force for good as well as bad, but the data here doesn't support it
If you use AI to outsource your thinking, then AI is not “just an amplifier” at all.
A huge proportion of people who use AI use it to outsource their thinking.
That’s why I said it will be used more for bad than good. Maybe I should have said, “it will be used for far more bad than good for the purpose of human intellectual flourishing”.
It can be used for good for this purpose. You can use it to find primary source information. You can use it to generate challenges for you. You can use it to critique your work when there’s nobody else available. Etc etc.
Intelligence and perseverance are two different traits. Smart + hard working kids will be fine. The problem is smart and lazy kids will find the easiest path, which in a world of AI leads to little actual learning. Maybe it was ever thus, but I fear the currently dwindling endangered book + teacher approach was better at getting slack asses to learn and eventually gain skills and knowledge for themselves.
The dangerous thing about amplifying is that bad people are often more willing to amplify their activities, because they don't care about the negative effects. We are seeing this now as AI companies (and large companies of all kinds) rush to secure whatever advantages they can, regardless of the negative externalities. Meanwhile people who actually care about doing the right thing get trampled.
We need to shift the incentives by adding ruinous penalties for things that are currently quite commonplace if they are done by large players. Some dude training his own AI on his own computer can scrape and train. The fine for OpenAI or Meta using a single copyrighted book without permission should be in the tens or hundreds of millions.
What we're seeing currently in our society is a "loophole inversion" where the rules have an effect mainly via their loopholes. The most profitable activity is to find loopholes and exploit them as frenetically as possible to gain as much advantage as you can before the loophole is closed, or get people hooked on the loophole so it's retroactively legalized. Entities that are big enough to do this are big enough because they have lots of money behind them. Entities doing the same kinds of things without lots of money are not really doing much harm. So the best approach is to adopt a "sliding scale" in which even tiny violations by wealthy actors result in penalties enormously greater than fairly large violations by small players.
> they don't care about the negative effects [...] rush to secure whatever advantages they can, regardless of the negative externalities
I think we don't yet know the long term view - what is harmful or damaging in the long run. Too early to tell and pass judgements. Every new technology from writing to internet had its detractors and they all pointed out negative externalities, as they seemed to appear at the moment.
You aren’t being cynical, you’re arguing with disingenuous entities with money on the line. We already know it’s used primarily for the negative case. Everyone who ever intended High School or College knows this.
Definitely had a sheltered public school experience, took AP stats my senior year and realized all the top students were sharing answers from an earlier period through text messages. After figuring this out I too joined in on the action, then shortly after I quickly deskilled.
Suppose it's good to learn how elastic the brain is, in both directions, at a young age where it doesn't matter.
> at a young age where it doesn't matter.
Doesn't it matter the most at a young age?
Not really. I would rather not find out later in life, with way more responsibilities, that when you purposely neglect mental skills they atrophy quite fast. People already have a hard time noticing that they age, being able to truly self asses is a skill that few, including myself here, possess.
> I believe AI is basically an amplifier of bad and good.
Same can be said of technology in general tbh.
It's a familiar story. If you copy/paste Wikipedia and turn it in, you learn nothing. If you skim Wikipedia, hit up the references, the consult other sources, then synthesize your own thought, you probably get farther faster than you would without it.
We could probably cook up thousands of examples of the same problem (YouTube DIY tutorials, GPS navigation, etc.)
Yeah the single thing it makes easiest is to think "i neither need nor want to do this because it is hard"
Evidently the AI users think hard things are beneath them. They want to win without doing any work. Sympathy, I have not.
This is my thoughts as well! It makes smarter people smarter and dumb people dumber!
There is zero evidence for this.
LLMs make significant mistakes frequently and smart people have no way of judging those mistakes outside their domain expertise. They are also sycophantic and great at being an echo chamber which makes people feel smart even if they are not.
So I think the burden of proof is on you to prove that they somehow amplify intelligence, it seems highly unlikely.
Most domains have some kind of internal consistency/theory building you can do. A smart person can certainly notice inconsistencies when trying to learn something. In fact they're likely to be points of confusion that the smart person will dive into just to try to make sense of things, even if they don't suspect the LLM is at fault.
People who think they are smart often think they can get by in any domain without knowledge and context, but at a certain point this fails and knowledge is required, not just intelligence.
We're talking about learning here, i.e., building that knowledge and context. People also refer to more than one source, and I think you overstate the degree to which LLMs make mistakes in current year. Human teachers also make mistakes, and it's typical for a bright student who's paying attention to correct even small details.
LLMs are not a shortcut to mastery, in fact the opposite most of the time and their errors are qualitatively different from human errors.
I don't think anyone is claiming they're a shortcut to mastery. Personal tutors aren't a shortcut to mastery either. Both simply have the advantage of being able to deeply know where the student's knowledge frontier is, so they can tailor their answers to that, make connections and draw analogies to things the student has mastered (while pointing out distinctions), etc. The student still needs to put in the work.
LLMs are not tutors and do not deeply know anything.
In the sense that they are not programmers and do not deeply know e.g. performance engineering, sure. But I can still give one a program and ask it to optimize it, and it will turn it into a zero allocation streaming SIMD implementation that's orders of magnitude faster than where it started. It may not "know" how to do any of that, but it does it anyway.
Pretty much this. With the internet and my websearch-fu, I can find some facts on a lots of things. And while I do for some questions I have, I try to never interject answers when I don’t have a modicum of mastery in the domain. Mastery is the formation of an internal model that guide your actions and thinking. It’s not merely knowledge and intelligence.
Yes sorry for the shorthand, an internal model is a good way to describe mastery. It’s something LLMs don’t have and can’t transmit.
No evidence but somewhat of a counterpoints:
Smart people know LLMs confabulate and tell them they’re Absolutely Right! Smart people don’t want to be embarrassed by trusting the hallucination machine and revealing their gullibility to others.
Smart people are handsome, charming, and always make the right decisions even on things they don’t know about. And if they ever are deceived, well I guess they were no true smart person.
Really widely reported in normie news and on reddit, hallucinations and sycophancy. Stand by that whether you’re a petroleum engineer or a bartender, if you’re clever you were probably either well informed or curious enough not to blindly trust the plagiarism machines.
Why would a smart person go to an LLM for an answer they cannot judge or test, be succeptible to flattery and sycophancy rather than picking up on the emotional manipulation and being suspicious/sceptical of the interaction, or looking for support from an echo chamber rather than a Socratic opponent?
All of those sound like flaws and defects of dumb people?
> Why would a smart person go to an LLM for an answer they cannot judge or test, be succeptible to flattery and sycophancy rather than picking up on the emotional manipulation and being suspicious/sceptical of the interaction, or looking for support from an echo chamber rather than a Socratic opponent?
Great summary of the flaws with LLM "research"/"reasoning". It's always trying to con you, and I question the literacy and intelligence of the people who can't see this.
Many people think they are smart and are taken in by LLMs. Also many genuinely smart people are unable to judge their competence in other areas (eg Nobel Laureates making pronouncements in fields they know nothing about).
All at the cost of ... {List of negatives regarding the construction and powering of AI data centers }
If you added up the negative effect of every single "ai data center" you'd end up with less than a single coal mine.
It's good to think about second order impacts but this is not useful.
Citation needed,
Just one 20 GW AI data center which is being built with gas powered turbine generator power plant, will be the largest fossil fuel power plant in the world... How exactly is that the same as one coal mine
The social costs are worse.
I dunno, concentrating the environmental impact of 20GW of gas based power generation would be the equivalent of 6-9 Million cars running... All in the space of a corner of a small city
That sounds much worse than I had expected. Is this enough consumption to significantly affect the local price of gasoline?
Edit: Why don't they have lower-emission power sources?
We do have lower emission alternatives, but not at the cost point... Remember these are corporations trying to make the largest profit possible
Most importantly, it makes investors think dumb people are smart.
Unfortunately this results in overwhelming net dumbing because the dumb are the ones lacking the smarts to avoid the "AI".
It helps smart people be barely more effective and helps dumb (more importantly, people who do not value effort, people who are lazy, people who are self absorbed) people shit out endless streams of worthless tokens that can swamp out everything.
Raising the noise floor like this only makes it that much harder to find "Smart" people, which we were already doing terrible at.
I use Claude every single day, but this is such a bad tradeoff. Maybe it will help me standup a quick fix when that is needed. Maybe it can help me dig through documentation to find relevant bits and figure out the unstated assumptions underlying it. Maybe it helps me generate test cases.
Meanwhile, my day to day life is now noise. All social media is noise. All content is noise. Slop pours onto me from all directions. Writing more test cases isn't helping me.
Am I smart? Am I dumb? I don't care, right now I'm deafened
All I've noticed is AI creating a perverse incentive to make everything as complicated and bureaucratic as possible, so only the people that know how to leverage AI to cut through it ever succeed.
I'm using Claude at work myself and am impressed with the product, but notice that this is the only reason I need to use it at all. Our product pages were shit to begin with, now they're AI-generated and somehow even worse. Our procedures are incomprehensible spaghetti with enough arbitrary context switching to give a sadistic Soviet municipal administrator an erection at the thought of watching anyone try to actually follow them.
Use AI to create inefficiencies, then use AI to bypass them. Those who can't do the latter will struggle to survive.
> Raising the noise floor like this only makes it that much harder to find "Smart" people, which we were already doing terrible at.
Clarification: to value “smart” people, which we were already doing terrible at.
> It helps smart people be barely more effective and helps dumb (more importantly, people who do not value effort, people who are lazy, people who are self absorbed) people shit out endless streams of worthless tokens that can swamp out everything.
... including ultimately stuff needed to make the smart people.
> Raising the noise floor like this only makes it that much harder to find "Smart" people,
It does give us a new heuristic, though: people who are willing to completely cut generative AI out of their lives (cold-turkey, if you ever started using it) are a much smaller group of, predominantly thoughtful, people. You do have to give up Claude to be part of this group, but from what you say, that's no great loss, and no longer being deafened is worth it.
This has considerable advantages over conventional elitism, because the barrier-to-entry is negative in almost all cases.
The one exception I've found is assistive tech, where the state-of-the-art is so poor that vibecoded slop is genuinely an improvement over the state-of-the-art, and in many cases the tooling simply isn't available to make your own assistive tech (unless you want to bootstrap an entire networked computing environment, which isn't very helpful when you want to do your online banking and do not, in fact, work at your bank).
But there are not many principled exceptions where you could seriously argue that the trade-off is worth it. Take mathematics, for example, which we often see touted as a "good use-case" of generative AI. The primary advantage of generative AI in mathematics is being able to search though a vast corpus of ivory towers and inconsistent terminology (without proper attribution) to locate and connect ideas that can help solve problems. The deficiency this is addressing is elitism, inadequate communication, and inadequate indexing within academic mathematics. This problem is entirely created by the academic mathematicians, and has been known for nearly a century (per https://en.wikipedia.org/w/index.php?title=Nicolas_Bourbaki&...):
> Bourbaki was founded in response to the effects of the First World War which caused the death of a generation of French mathematicians; as a result, young university instructors were forced to use dated texts. While teaching at the University of Strasbourg, Henri Cartan complained to his colleague André Weil of the inadequacy of available course material, which prompted Weil to propose a meeting with others in Paris to collectively write a modern analysis textbook.
To my knowledge, this is the only organised project to clean up and improve mathematical communication. Everything else (Metamath, Mizar, AFP, Lean) is yet another ivory tower. The Wikipedia article on this topic (https://en.wikipedia.org/wiki/Mathematical_knowledge_managem...) risks deletion as non-notable, that's how little anyone's actually trying. They made their own bed, and generative AI will only provide a brief respite from having to lie in it. (I was surprised how many other "compelling" use-cases evaporated when I applied this razor to them: the sibling comment https://news.ycombinator.com/item?id=49392265 points out one such.)
Vibe-coding assistive tech which doesn't yet exist, as a temporary scaffold to improve the quality-of-life of yourself and others in a social world dominated by non-essential access barriers is, to my knowledge, the only exception to this principle that can be justified. If you treat people who make other excuses, or who don't even bother with excuses, as not worth listening to, you lose little – and doubly-so, if you make your stance clear, so that others know the "cost" of gaining your attention.
AI does not make me "smarter" in the important thing I am working on.
In software particularly, what AI does do is give me back my time from the drudge work that I don't care about. Keeping my build system configured and my tests up to date and my documentation synchronized is a good use of AI because I mostly don't care about how crummy the result is as long as it "works".
> ... a good use of AI because I mostly don't care about how crummy the result is as long as it "works".
This is "AI"s forte. Producing more bad work at lower cost.
There’s been multiple articles at this point about AI’s impact on education.
It’s alarming how near universally AI has been a cataclysm in class rooms.
On homework the short answer is is that we give kids home work exercises.
AI helps learners reduce the effort expended to exercise and get results. This is making homework moot.
All AI did here is exposed an old problem in education. Kids are expected to get everything perfectly for the first time but a lot of them don't so the class gets easier next year to keep up pass rates.
We need to restructure the system to treat failure as a signal instead of a disaster. Grades should come from hard randomized exams with unlimited retakes so one bad day won't hurt you. Homework should be optional material for self study, evaluated by teachers if you choose to do it but never forced.
Hold back students for individual classes instead of a whole grade so failing one can't ruin your social life and teachers are more willing to do it. F students will realize they have to study, start actually learning and then pass on the second time. No big deal. It happened to my friends in college, no reason they can't do it in high schools.
Discipline is a skill and it's one you have to get from experience. If you try and force kids to study when they don't want to "for their own good" you're not actually helping them. Everyone needs to find their own path. Let people fail.
> Homework should be optional material for self study
At Caltech, homework was assigned but had no bearing on your grade. The grades were based on the midterm and final exams.
But not mastering the homework usually resulted in flunking the exams. There were "retch" sessions after each homework assignment that was staffed by a grad student, and the purpose was to help the students understand the homework problems. I knew only one person (Hal Finney) who was so smart he didn't need to do the homework.
I learned the hard way that the path to success was:
1. never miss a lecture, no matter what
2. take notes by hand during lecture
3. do the homework on time, and make sure you understand every problem. Take advantage of the retch sessions.
And that worked for me.
Likewise at Oxford - only the final exams counted (though you had to pass first year exams to continue to the second year).
But you had tutorials each week, and if your tutors thought you weren't doing enough work they could set you exams mid-course called 'penal collections' and if you failed them you could be thrown out. They were rare but definitely not unknown.
Naval Nuclear Power School had (maybe still does) this model 20 years ago as well. Pass rate for my class cohort was 28% over the entire 2 year pipeline.
We didn't call them retch sessions, and they were taught by the instructors though. We also were encouraged to peer tutor and since we were all restricted to one building the homework was always group work allowed.
Also had badges to track time spent in the building for required study hours, though some people gave up and just slept at their desks when they started sliding down the grade scale and the hours racked up.
Generally I think I did 30 hours of studying/homework (went up and down depending on what was being taught, but was around that) a week (for 12-15 hours of actual lecturing), with some of my friends putting in 50% more. Generally the only day we weren't there was Saturdays. Most of the day Sunday was usually spent in class preparing for the next week.
I think only a third or so of my class wound up graduating. Caltech doesn't flunk people out. What happens is students will transfer to easier schools, or drop out, or decide driving a truck is more to their liking.
I've given this advice many times, and the ones who followed it did well. I only added "leave your laptop in your dorm room" for modern times.
The general pattern at Caltech was 2 hours of study for every hour of lecture. Which was quite a shock to me.
I'll add one more thing to this, given that my experience as a math major at SIU was that each hour of lecture was expected to carry four hours of studying: read and summarize the course's texts (additionally, for math courses, attempt the problems) before lecture.
I found that (1) I didn't need to take anywhere near as many notes during lecture and (2) I could ask way, way, way more relevant questions.
This also helped tremendously when it came to studying for the actuarial exams.
> 1. never miss a lecture, no matter what
As a student of a top-tier French Master's degree, I consulted with a teacher to deal with exhaustion and to ask for a class rescheduling for my case. Explaining my situation, the teacher looks at me and interjected:
—Waitwaitwait. You... you went to all lectures!?
And he was right. Rather than following the curriculum, I should have developed my taste for various engineering topics and only used the classes as entertainment.
I think the first semester, or at least half of that, you should go to every lecture until you develop taste for what's useful. By the time you do your master's you should have a good sense for what classes are serious and have useful lectures and where you can just cram last minute because it's a bullshit class and where you're better off studying from a textbook.
That only works with competent lecturers. I can speak for my experience at a more mid university, where that would apply to about two thirds of the classes. For the remaining third, going to the lectures felt genuinely counterproductive and you could actually feel yourself losing braincells listening to the confused nonsense or classes held in English for Erasmus students by someone who could barely speak it coherently. It was just a complete waste of already little available time.
So the endgame was figuring out what the tests in previous years looked like (cause it was likely gonna be a copy paste affair), do a targeted study run for those exercises and 9/10 you would pass.
do you have any evidence that CalTech's pedagogy is immune to today's threats and trends in education everywhere else? (no)
Nope.
When I was there, a long time ago, exams were timed and were usually open book open note. Blue books were filled in. You were trusted to adhere by those rules, and most students did their exams in their dorm rooms.
The evidence that the students honored the rules was some exams resulted in a 50% failure rate.
As for me, I went there because I wanted to learn the material. I did not care about getting a diploma. (Mine is in the basement somewhere.) I did not take any "easy A" classes, because I wanted a return on my time and tuition investment. (Though, easy A classes were hard to find at Caltech.) I wasn't even going to attend graduation, but my parents showed up and I attended to please them.
The classes, year by year, were dependent on mastering the previous year's classes. So if you cheat with AI, you're digging yourself into a bigger and bigger hole. Caltech rewires your brain. If you don't learn the stuff, you're going to be one of those EEs who carries around a card with V=A*R, V/A=R, V/R=A printed on it.
Unlimited retakes would be an enormous amount of work for professors/TAs/teachers.
Optional homework is often a disaster. At best, students would do it right before an exam and the goal of education is not to just pass exams. They’d probably still get a lower score than if they did the homework when they were supposed to.
What I think is better is to have a due date, but just make the maximum 10% each day it is late. So after 2 days, the highest score you could receive would be 80%.
I liked that system because it gave some flexibility with deadlines while still encouraging you to turn things in on time.
Late work policies like this can be a logistical nightmare at scale. Rather than play the penalty-over-time game, have hard deadlines that are inflexible. Students don't have to engage in mental gymnastics with their scores and instructors don't need to track this stuff.
My view is this: a student can turn in an incomplete assignment. Work they completed get evaluated and graded. Anything not completed gets zero credit. Homework should be an investment in effort over time anyway. Unfortunately, procrastination is a deeply rooted problem that usually kills this.
> If you try and force kids to study when they don't want to "for their own good" you're not actually helping them. Everyone needs to find their own path. Let people fail.
This simply does not work for most, as pupils are too immature. You put sunscreen on children, keep them away from the street and brush their teeth. Having everyone re-learn the same stupid lesson is wasteful, it's better to institutionalize this knowledge in the form of mandatory participation.
What you described about exams is like how I approach most certification exams: go through a test bank of questions until I get 80% then take the real test.
> Grades should come from hard randomized exams with unlimited retakes
Exams will have to be a lot longer if you allow unlimited retakes. Generally exams work on a sample principle, but this breaks with retakes.
The study was done in China, where making the class easier to improve pass rates isn't really a thing, as the Gaokao operates more like a stack ranking where it doesn't matter much how well you did in an absolute sense, only that not too many others who did better than you are competing for the same spot. Unlimited retakes are possible in theory, except each costs you a year of your life and except for some extreme cases of repeat test-takers, most people are going to give up after just one or two bad results. Homework is definitely not optional, but going home from school is. (Due to the hukou system, children often have to go to school far from where their parents work, so boarding schools with teacher-supervised self-study are common.) I don't see how holding back for individual classes is supposed to work considering scheduling conflicts.
> F students will realize they have to study, start actually learning and then pass on the second time
Do you have evidence for this beyond your friends (who were accepted into college)?
You put way too much faith in kids' desire to learn and their ability to make responsible choices.
> Homework should be optional material for self study, evaluated by teachers if you choose to do it but never forced.
My wife is a teacher and her school does this. The result is that nobody does homework. The kids who need the extra practice don't get it and they fall into a spiral of failure and apathy. This is especially bad for subjects like math that build on themselves.
IMO the better solution is to require and grade shorter homework and provide students with optional, supplementary assignments that can be used to make up for missed credit on homework for the same material. That way students who grasp the material quickly can demonstrate and move on while others who need practice are naturally encouraged to get it without being permanently punished for struggling initially.
Homework is no longer useful in the era of AI. Options are: treat kids like they are able to make responsible choices, and accept that much more of them will fail; or, keep the charade of grading AI-"assisted" homework and push kids farther along the track, even though they definitely don't know enough.
I much prefer the first.
School shouldn't be impossible, but you shouldn't have a charade of schooling.
Obviously, extra practice is a good idea.
It's not necessarily useless. If you make assignments short, provide some time in school to work on it, and do not allow students to use devices with access to general-purpose "AI", then there is a good chance most students will do the work like they're supposed to. Some will still cheat, but it's foolish to reorganize everything around the few who stubbornly refuse to learn.
> If you try and force kids to study when they don't want to "for their own good" you're not actually helping them. Everyone needs to find their own path. Let people fail.
this rhetoric is pretending to be an alternative to coercion. IMO the ideas you are talking about are well trodden and are still coercion nonetheless.
I could be mistaken but my interpretation is that "studying for your own good" is being contrasted with "studying so you don't fail". If you actually fail students, they'll have a second motivation beyond trusting some authority figure's advice. Namely consequences.
Coercion explains it perfectly. A lot of kids see no future because of the environment they live in, so they have exactly zero motivation to study
You are missing the point. Getting a certificate for all of the programs you passed and still graduating could completely change the course of a person’s life. Community college has certificates and associates degrees to let people start working when they reach a practical limit in their general education.
It's all cute but naively glances over what this all boils to - competition.
Education system, contrary to popular belief/name, isn't tailored to educate but to select winners and losers which then will be picked on the job market.
That's why it seems absurd when you think about it as an institution that aims to educate. That's because that isn't the real purpose of it. The purpose is to stratify and classify early.
Schools do not operate in a vacuum; they serve as credentialing gatekeepers for a hyper-competitive capitalist job market. If everyone could easily retake exams until they got an A, grades would lose their primary utility for employers and universities: differentiation. Society relies on schools to provide a neat hierarchy of candidates that for one reason or another thrived in difficult environment of adolescent schooling.
The system often prioritizes compliance, endurance of boredom, and social maneuvering over actual critical thinking precisely because those traits align with corporate hierarchies.
> compliance, endurance of boredom, and social maneuvering
Compliance and endurance of boredom I can somewhat agree with (thought not to a great extent), but social manoeuvring? As long as a student completes their classwork and homework, and does well in their exams (none of which mandate any social interaction apart from team projects), they easily pass through the school system. In fact, I used to hear about how the school system disproportionately benefited studious, shut-in students over street-wise, socially-savvy ones.
Moneyshot: "The results are eye-opening. After six months, pupils using ai saw their average homework score rise by 18% across all subjects. The time they took to complete each assignment fell from an average of 64 minutes to 45. But come exam time, the same students scored 20% below their classmates who had not called on ai’s help. Homework scores once predicted exam performance; now those who score highest are, perversely, more likely to do worse in exams."
Study design: "David Stromberg of Stockholm University and Victor Lei and Wu Yanhui of the University of Hong Kong set out to fill the gap. They tracked 27,000 pupils aged 12-18 in China, where ai adoption has been fast. Around 80% reported using models such as Doubao and DeepSeek; the other 20% formed the control group."
> Homework scores once predicted exam performance
As with most training the journey is the point, not the destination.
That said, I think smart use of AI could help. It could explain concepts in a way that might help you understand better, it could probe your knowledge in a more dynamic way by tailoring questions, and so on. This requires the AI be restrained by some harness, not free to write down the answers for you.
> help. It could explain concepts in a way that might help you understand better
People say that all the time, but does anyone really think a lack of good explanations for things is a limiting factor in 2026? Or even 2010?
When I was in college, we had office hours with the prof and the TAs, we had group study sessions, and private tutors. Or you could ask your friends. All of those required going somewhere at a specific place and time and asking someone else to give up their time for you.
AI gives you a way to get the same help without asking another human. Say of that what you will, but not everyone was comfortable asking other humans for help even back then.
You also have your book and a wealth of online resources, which also don't require interacting with someone, and which have the same information the AI is trained on. Fundamentally nothing is different. We don't see positive outcomes from students using AI as an aid even if hypothetically it could be possible.
We didn’t have YouTube and the unending number of Indian teachers giving lectures on the sciences.
The act of building the mental muscle is what creates education.
Back when I was studying there were topics which I had a really hard time grasping. Having a personal tutor that I could ask specific questions and have a back and forth with would have helped a lot I think. At least the few times I did have such an opportunity that was definitely the case.
Kids are, by and large, lazy however. Hell, adults are lazy, this isn't even an indictment on kids or even people in general, I think our bodies & brains are simply wired to seek the path of least resistance from an evolutionary POV.
Kids won't be using LLM tooling as a personal tutor following some sort of Socratic method, they'll ask it to solve their home/coursework for them and blindly copy/paste the answer. Hell, they'll just manually copy down what's on their screen if copy/pasting isn't possible for whatever reason.
Obviously exceptions exist, but I'd wager from being an ex-kid myself the type of kid who would genuinely use these tools for actual proper self-tutoring would be an extreme rarity.
> This requires the AI be restrained by some harness
How would this magic harness look like and why would anyone use it?
"I need help with my homework, but I don't want answers, just guidance on how to get to the answer".
Put that in your agents.md.
I'm way beyond homework, but I do appreciate the compliment on seeming young!
I meant: what's going to stop the average lazy student from ignoring whatever harness the university recommends and instead use an unrestricted LLM, thus learning nothing?
> what's going to stop the average lazy student from ignoring whatever harness the university recommends and instead use an unrestricted LLM, thus learning nothing?
Nothing until they start failing exams and maybe seeing the error of their ways. My suggestions wasn't for you, it was for the smart student who wants to use AI to enhance their learning but not have it do all the work.
But cheating isn't new. People have been cheating in school for centuries. It's easier now, but the consequences were always the same. At some point the chickens come home to roost and you pay the piper.
> But cheating isn't new. People have been cheating in school for centuries. It's easier now, but the consequences were always the same. At some point the chickens come home to roost and you pay the piper.
My concern is that with LLMs and above average bullshitting abilities, the chickens might never come home to roost.
I think that is fine, certain things are becoming less important to learn and certain things more important. With the pace of change it is likely to continue shifting.
I really don't see how anything but social skills are rising in prominence. Perhaps twenty years from now everything will depend on your connections as any "hard" problems will be readily solved by AI (seeing as few problems are ever really novel).
College stopped being about getting an education a long time ago.
It's just a low pass filter for the job market. Students aren't actually interested in learning anything, they have been condition for a long time to jump through the hoops.
It would be much better if we actually returned to exam and performance based evaluation of students, and stopped giving everyone As just for trying hard. But parents look at college fees as buying their kid a job opportunity, and so grade inflation has basically ruined the SNR of academic performance for all but the lowest performers.
I meet a lot of noj-college people and let's just say, "just a low pass filter" is still A LOT of value.
Although surely highpass filter would make more sense in the analogy here?
It depends on college attendance. In developed countries, it is a low-excluding - or the inverse of a low-pass - filter as most graduate, while in developing countries it is a high-pass filter as few have the resources to reach an undergraduate level of education.
I'm confused by your point here. You seem to be saying it's a highpass in developing countries and also a highpass in developed countries?
For some of the classes my kids were in, the lessons and homework time were reversed.
Learning the material (using the text book and videos) was what they had to do at night and class time was spent working through problems or applying the material in some way.
Flipped classroom was a popular method when I was teaching math. Some students found it very helpful. The trouble was getting the students to do the reading before class. If students came in unprepared they struggled to keep up.
I did this for the last college programming course I taught (2025) as an anti-LLM measure. Every class started with a very short closed-book auto-graded multiple-choice test on the reading. I was careful to specify which was the core reading needed to learn the material (on the quiz) and which was extra (not on the quiz - I think many professors assign this extra reading to everyone, thinking of their strongest students). I think this developed an expectation in the students that doing the reading was easier than finding a way not to.
It's especially hard when multiple subjects try to do it, because it can add enormous amounts of home-time expectations. And many families can't dedicate that much time.
I do like it though, and I like that college generally leaned that way more (and was better balanced, as it had substantially less time in classrooms, so you could study during the day).
I taught a flipped classroom online. Students had to successful pass a quiz on the before class materials to get the join link to the session.
Popular yes; based on good quality research… no, not really. It’s really pretty unfortunate timing with the AI boom, because the field of education research is not very high on the science quality totem pole. And the field as a whole was and is also slow to incorporate neuroscience findings, too.
I think for my ADHD child-self, this would have been a huge improvement. I didn’t need help learning, I needed help doing homework.
They’re using flipped learning even in universities now.
I experienced this method in accelerated class context, decades ago
This is typical at the college/university level - when I attended 20-25 years ago, we were expected to learn the material through readings outside of class, and then class time was spent on discussion, application, questions, and working through problems.
IMHO there's a lot to recommend this style. A lot of students learn better when they can wrestle with the material at their own pace, on their own time, in their own setting. It quells a lot of anxiety about "I'm not following what the teacher is saying, am I stuped, will I look bad in front of all my peers?" And then class time can be spent identifying holes in your knowledge and getting instant feedback from an expert, which is where they are most useful.
Although obvious, I think it's worth stating that this method mostly (only?) makes sense for higher education, where students are conscious, involved and engaged in their curriculum.
I think it makes sense as soon as students have the fundamentals down. You have to be able to read, listen, make connections, think critically, devote sustained attention to a task, and manage your time effectively. Some students develop these skills in elementary school, and then starting in middle school, the flipped classroom approach can work. Some adults never develop these skills. Although arguably, people that never develop these skills will likely be sitting there zoning out in a traditional classroom lecture and not actually learn anything.
Flipped classroom method?
Yes! I didn't know it had a name but a quick googling confirms what you said.
Interesting. Ages?
This was probably middle school.
What do you think about it? I think that the ability to study by yourself is probably the most valuable thing one could get from school and most kids usually don't. Your setup would suggest that demanding that a kids know how to study by themselves is going mainstream...?
Your answer - middle school - I find it extremely young for this, which is extra interesting.
IMHO, my kid was assigned way too much homework. One year I eventually told her to do some amount per class (I think it was 45 minutes) and then be done with it so that she would have time for sports and family stuff. She was spending 4 or 5 hours every night on homework and this was in 4th grade.
That year I sent an email to her teacher to let them know that I'm the one responsible for her not doing all the work and if that's a problem, we need to talk. It wasn't a problem.
The one class that was flipped was a relief because she could breeze through the lesson much faster than would normally be spent on it in class and the amount of time on exercises was limited to the class time slot.
Here's research showing the same exact effect 15 years ago, except it's about homework copying: https://news.mit.edu/2010/homework-copying-0318 Students who copied their homework more saw their grades go down more throughout the semester. A/B students became C/D students.
A name people use for this phenomenon now with AI is cognitive offloading.
I coincidentally took advantage of the cheap DeepSeek Flash pricing the past couple of weeks to compile & synthesize a knowledge base/wiki of hundreds of educational AI research studies like this one here: https://edtechdev.github.io/aied/ Here's the summary of this article: https://edtechdev.github.io/aied/articles/stromberg-generati...
Education sustem is not about the actual knowledge, but about training various aspects of the brain during kids development. So even memorizing literature have benefits, even though it doesn't bring any career use.
If you skip this, well your brain won't develop as much at period of life it's able to do so.
I see this as a memorize-vs-understand fallacy many repeat.
It is really a combination: you need to memorize a large corpus of information to operationalize knowledge — understanding a bunch of theorems in mathematics does not help much (even if you are able to prove them when you see them) if you can't remember the boundary conditions they hold under.
Or having good understanding of foreign language grammar won't help you if you do not memorize words that you need to express your thoughts.
Good literature will show you some of life's challenges and potentially let you think through them in a non-stressful situation (other than "I've got to finish the last 200 pages by Monday" ;)).
> if you can't remember the boundary conditions they hold under.
At the outskirts of one's knowledge this is inevitably the case. My impression it is still useful to know that a certain implication is possible - and one can look up the exact conditions.
This is why I said operationalizing: you need to memorize enough to either be able to know what to look up, or know it outright if you need to be fast(er).
Maybe. As the nature of assistive tools changes, so does pedagogy. Calculators are the easy example since most of us are (still) familiar. But there are others that were just as pronounced -- e.g. log tables and slide rules were transformational and instrumental at one time, and you've (probably) never used them. If you had used slide rules to solve your homework problems then were excluded from using them on exams, you'd see a similar effect.
AI is a big change; pedagogy is going to change too.
> As the nature of assistive tools changes
I mean, does it? The way I see it, the best LLMs have to offer is infinite patience (until they inevitably and unpredictably start to confabulate, which should be a gigantic red flag, anyhow), I don't see how LLMs can be used to explore teaching paradigms that haven't been explored before. We know pretty well from centuries of empirical experimentation how children's brains develop under different stimuli. It's not exactly something the software industry needed to "hack".
I wish that was the main way countries understood education
Taking a forklift to the gym increases the number on the bar, but mysteriously your gains go down.
To extend this analogy:
Using the forklift as a spotter and to assist in loading weights increased gains. Then again, a human can do all those things, and provide real human connection.
If you structure learning in such a way that makes learning just a means to some end, and overindex on that end being the ultimate goal, that's what you get.
This is a pedagogical problem that AI merely exposed. Educators need to figure out How to make students choose the scenic route instead of having them optimize for the most efficient completion of a task.
School has always been about creating compliant workers, not educating people. They will double down on "performing the right things" and "morals" while the economy will continue on its K shaped path as AI gets more capable.
Could it be that you vastly overgeneralize? While there's no perfect education system, many are much better than the nightmare you describe.
Yea, also the programs are too different. My psychology bachelor felt like a tea party. My computer science master was 60 hours per week during the hardest courses (which were supposed to go for 20 hours per week - so I could only do one course if it was at this level). My game design master felt more like we were dumped into an art school masquerading as a psychology/CS master but it really was art school. My information science bachelor felt like the only "normal" study program.
All of this was at university of which 3 of them were at the same university.
Especially the artsy game design program definitely did not feel like it was preparing me to be a cog in some giant corporate wheel.
I was thinking how tricky it would be to learn to program now, honestly a sense of laziness is one of the reasons I learned programming - so that I could automate away repetitive tasks. Our generation might be the last people who can actually program properly, everyone will just learn some basic architecture and point the AI in vaguely the right direction.
Seems if any assignments are to be handed out they should be structured in such a way that takes into account the potential of LLM use. Additionally to make the output (hardcopy? handwritten?oral presentation?) in such a way that even if LLms are used at the very least some part will be forcibly undertaken by the students in preparation of finals/real life.
Open book exams existed for a long time and even take home assignments where the net was usable to complete the tasks. Difference was one actually had to do the reseach and learn something along the way regarding research skills or the topic itself.
LLms just make the automation and laziness with a lower barrier to polished output.
I full heartedly recommend anyone who wants to learn, to use the Socrates skill, and then build a daily 1 hour curriculum that you will tackle. It's painful and not pleasant, but it works quite well when to don't have a teacher and want to learn something really hard and complicated
“ Those who used AI but spent as long on assignments as non-users paid little penalty.”
What is the point of using of AI if that is not going to speed up learning.
I am waiting for AI Viva Voce (“AVV”). I am really surprised we aren’t hearing more about this idea?
Try it for yourself by making a prompt like this (adapt as needed):
Pretend I am an undergraduate student of Computer Science. I am learning about early microprocessors from the 1970s. I want you to ask me an examination question as if you were doing a viva voce exam with me, to test my understanding of concepts. I want you to receive my answer and then based on what I said I want you to ask me a more specific question to probe my understanding. Repeat this interaction up to 5 times. Then grade my understanding so far, by giving me a pass, merit, credit, or distinction. Can you explain how you arrive at the grade based on my answers and your expectation of undergraduate knowledge of microprocessor theory?
I've tried stuff like this, but the problem is that you don't know if it's answers are correct, and more importantly, you don't know if its questions are on base or not. Plus its too easy to go off on tangents, especially when you ask it questions to clarify. You can get some more success by writing down a framework beforehand, but openended dialog is still not a good way to learn with AI, from my experience.
Good points. I’ve tried this a few times and I’ve been impressed by the dialogue. Given this was a very simple prompt, I am convinced a more precise version could be developed, including a rubric, a library of course content, starter questions and other guidance to the AI. I haven’t seen anything like this and schools / universities seem to be “struggling” with AI, which has me wondering why not embrace LLMs for learning and assessment, like this?
I mean, everyone I know has been using socratic dialogue for months to aid understanding, this seems like a variation of that. The issue is that if the answer is too easy to obtain it's hard to have the discipline not to cheat.
https://archive.is/w1eng
Students seem to optimize for local maxima/short-term rewards (homework scores) at the expense of the global maximum/long-term rewards (learning).
Which was pretty much the whole damn premise of needing a teacher to begin with. What a surprise!
In my professional career, my employers give me a lot more assignments than exams.
Meetings are still like exams, if you can't craft a line of thinking on the spot it doesn't come off well.
You don't need a final answer but you can't just answer "I don't know" otherwise you risk being slimed (shout out to anyone who still remembers "You Can't Do That on Television")
“I don’t know but after this meeting I will find out and get back to you”
Is an acceptable answer.
It's only an acceptable answer when used sparingly. The more often it's used, the more questions arise about your role in that meeting.
I find people’s on the spot reasoning abilities considerable worse than even cheap LLMs. This “fast” mentality is what causes stupid decisions because we need to “keep moving” (To where? Wrong meeting!) Answers, answers, decisions, decisions, quick, quick! next quarter In hindsight we need to do things differently. Answers, answer, quick quick! Time is of the essence! If you don’t start producing sound within 5 seconds we will all spontaneously combust!
Exactly, maybe there was no need to waste his time with the meeting, he could have been doing productive work instead
People can also ask questions that it is unreasonable to expect people to know on the spot too and after a while the role of the questioner in the meeting can be questioned. :)
I prefer to share a video, ask people to watch it, and have them send me questions. That way I can spend time preparing answers to those questions, and then then meeting can be about discussion, not information delivery or recall.
Asking people to be conclusive on the spot is a recipe for acting on bad data. There are too many biases related to stature, they interfere with accuracy.
I can see that. Personally, I use that for people who like to go down tedious rabbit holes and want to free up the rest of the attendees to go back to work.
True, but only for things that that your colleagues don't expect you to know already. Remember the "I'll circle back" meme about Jen Psaki? This response can only be used sparingly before it's seen as a sign of underpreparedness/inability to do back-of-the-envelope thinking.
Why would they ask you when they can ask Claude then?
Not really, in any company I've ever worked.
The answer is "accepted", in the sense that it won't get you fired. But the meeting plows on, and decisions are made in absence of the answer to the question! — and never revisited once the answer is known.
And like 90% of the time it comes up, it's because the data contradicts the decision.
Every conversation you have with a person is an "exam." Not being able to extemporaneously express yourself or reason analytically without running to Claude seems undesirable.
In my professional career, my clients value the depth of ready knowledge that helps me use my tools with particular insight and finesse -- a kind of knowledge that exams demonstrated far more effectively than homework when I was back in school.
The exams aren't the thing that matter, they're an experiment to measure the thing that matters. Which, presumably, still matters at work!
Stand-alone bottom-up observations (even if valid) obscure the fundamental problem that AI (or "AI" if you like) is inducing systemic change across human civilization.
If cohorts are graduating into the workforce with decreasing stand-alone skills, that implies the economic value-add of labour is shifting to AI, which implies broadly falling standards of living, falling political power, and an ongoing transfer of power and wealth to capital.
If students are taking shortcuts with AI when their brains have the highest capacity for learning they may be permanently weakening their prospects. It doesn't seem likely that whatever "AI management skills" they incidentally learn will make up for, over their lifetime, the economic value loss they suffer due to reduced cognitive skills.
On an individual basis, in the next few years, this might not matter. Across decades, economies and polities, it will.
> Stand-alone bottom-up observations (even if valid) obscure the fundamental problem that AI (or "AI" if you like) is inducing systemic change across human civilization.
Students have been cheating and taking shortcuts on their work long before the current generation of AI tools existed (and probably for as long as graded exams have existed).
Somehow, civilization has managed to survive.
When you lower the barrier to taking the shortcut, you get more people to take it who wouldn’t otherwise have done so. If cheating becomes incredibly easy, lots of students who would have otherwise been honest take the easy way out. Moral conviction gets harder to keep as doing the right thing becomes less rational.
Doing the right thing never becomes less rational. It may suck compared to the temptation towards the alternative, but part of growing up and being responsible is realizing there is a reason the shortcut isn't supposed to be normalized.
And civilizations do fall. The bronze age civilizations did collapse. There was something dark about the dark ages. There is no law of nature which says that things will be okay, and plenty of examples where things have fallen apart.
Just to add to this, if AI is as epochal a change as its boosters say (and I am open to that argument), then surely it must have the potential for awful consequences, in proportion to its potential for good.
Your career is a continual exam and the assignments are just parts of it.
Disagree. School exams test for academia-readiness too. And sometimes the schools and lecturers are filled with ideology irrelevant to real work.
Career exams tend to be a mix but a lot less ideology.
Fortune cookie ass statement.
Your career is more akin to the totality of school than it is to any specific facet of it imo.
The social aspects are more important than the exam sitting most of the time.
> Fortune cookie ass statement.
> The social aspects are more important than the exam sitting most of the time.
To spell it out more clearly: every single aspect of your career is an exam. The interview is an exam, quite literally. The day-to-day responsibilities (meetings, planning, problem-solving, collaborating) are parts of the exam. If you're failing at those or automating them away, then what is your role? The assignments (solve X bug or add Y feature) are parts of the exam too. Failing to do these things will mean that, yes, you are failing the exam.
And yet if nobody likes you your career is going to stall. If you’re annoying, if you smell bad, if you dress poorly, if you’re fat and ugly, just like in school you’re going to have worse career outcomes.
And I could also call most of those things assignments instead of exams and nobody would bat an eye.
Your analogy was both thoughtless and stands up to no scrutiny
Your employer gave you your job based on your exam results.
Exactly. And what is a job interview but an exam?
Even if we grant that education is to prepare for employment, which a lot of people disagree with, it does not follow education should do the same thing as what's done in employment.
Your employers want those assignments done. That’s why they pay you. School assignments belong in the trash can and you pay for someone to look at them before throwing them away.
As an example: my employer doesn't require that I learn how to communicate well. It requires that I already learned to do so.
evaluating/reviewing your LLM's output/code/reasoning is an exam
unless you're just blindly passing it on for someone else to deal with
One of the main reasons safety standards are written in blood.
What if learners used teacher AIs such that the AIs would keep track (and submit to class teacher) of the scoring for different "mini exams" it put the learner through, so in effect better preparing for exams.
It is stopping some of the adults I work with from learning.
In the case of some management, is causing them to unlearn, forgetting about proper review and maintenance practices.
"Won't that be grand! Computers and programs will start thinking, and the people will stop!" —Walter Gibbs, Tron
I know this study is focused in China but one thing I'd like to understand is the nuance in what kind of student is likely to always reach for ai for homework help as I think (especially in the US, can't speak for other countries) the varying powers that be that can determine the quality and type of education you can receive (does your country have means to buy textbooks per student or shared, do you have funds to give students ipads to take home or no, what sets the bar on how far curriculums can go, are teacher and faculty pay and incentives simply tied to student pass rates, etc.) and those do more in shaping what "use AI responsibly" will look like and why it's so different.
AI itself is not the problem imo. The core issue is basically outsourcing your entire brain to the LLM and never trying to solve a problem yourself
When I was in school there were multiple instances where I would be stuck on a problem for nearly an hr but I always learnt something from it. They key I think is to accurately identify when to and when not to use AI
Glad to have scientific results on this, though in my view you could get this from first principles. Homework is onerous but it forces you to learn the material and get it into a configuration that works in your head, which you then validate with the exam.
as a fun aside here from the abstract it does come down to how you use it:
> AI users who maintain similar homework completion time as non-AI users experience small learning losses.
If the pure completion time of homework goes drops significantly with an LLM tool, I suspect these students are spending the extra time turning the material over in different ways to internalize, which is interesting.
The homework helps the teacher track their performance/progress.
When the signal is removed, this is what you get.
To the surprise of no one who’s doing any teaching these days.
Use AI for homework, you will get it done faster and get a higher grade, and then get crushed on the exam.
If others studies tend toward this result, a question comes to my mind: it is a problem with AI or our educational system? or both? I would like to have some comments...
Now, what this tells me is that LLM AIs can supply the knowledge of the moment for the prepared human person, and the person does not need to learn that knowledge, for a extremely large class of problems. Which is fantastic for capitalism and business in general, and very very bad for humanity in general.
Are all of our LeetCode scores down, too? We should probably code the old fashioned way periodically to ensure we still "got it".
The article does not relate AI usage for homework to exam score drops.
It tries to link them together with no obvious cause and effect.
The score could have dropped due to worse environment at schools or kids who were stuck at home studying online and now in physical locations could not adapt fast enough (meaning, possibly just temporary drops). We do not yet have enough metrics gathered yet.
This type of article bashing AI for the root of any problems, I find it appalling without any evidence.
It's nothing but a conjecture.
(This was originally posted in response to https://news.ycombinator.com/item?id=49389565, which has a different article. We've since merged the threads.)
They left 20% of the 87,000 students as a control group. Is that not enough to allow for differences in environment, etc?
Do the exam scores matter when in the real world ai is the one handling the tasks. Why are we measuring human exam scores.
i’m pretty sure there’s already a correlation between technology in the classroom and worse learning outcomes
I find we are using AI wrong.
AI shouldn't be "helping" with homework. AI should be the homework.
Start with any topic or problem, students should be encouraged interact with AI, and learn stuff using Socratic method.
Teachers should rate the chat session instead.
And iterate the process to high profeciency.
Honestly I wouldn't demonize AI in education. When we were students we were forbidden from using calculators, then the internet and now it's become the norm. It's the same with AI, I think. AI has already become a part of our lives. The important thing is just teaching children how to use it correctly.
Unlike AI, calculator use has been studied and shown to be useful for math education. Our understanding of these things evolved over time, however the results of this study are concerning.
Teachers have become proctors and not educators.
It's an ad hominem attack, I know, but why am I not surprised that the Economist loves this?
True or not, it supports a narrative of "kids these days...", of a morally bankrupt educational system, of indulgent adults spoiling children, of just desserts to the wicked and lazy.
Again, I'm not saying the article is right or wrong. I'm saying I'm skeptical because it so tidily aligns with the Economist's editorial world view.
I used to skip homework in high school because it was boring and cut into my personal life outside of school (including my job) but I would always do very well on tests.
Then I stopped caring about tests too.
I did the same thing through college. The only reason they passed me and I got a degree is that I built the school’s website and I built personal ecommerce sites for the head of art and his wife to sell their paintings.
I haven’t done shit since like 7th grade.
Worked at Facebook, Apple, Microsoft, same behavior there - did basically nothing for them while making thousands off my games on App Store.
Fuck authority
I think we need to accept that AI causes cognitive decline, like Alzheimer's. For an Adult this may be manageable, but in a young brain, this is detrimental.
I remember reading about people talking how writing and reading books leads to cognitive decline because people won't need to remember things anymore.
Pretty sure the way social media is designed contributes more to short attention spans and atrophy of the cerebral cortex.
The fact people were wrong in the past doesn’t mean it’s wrong with regards to AI
And they were not wrong. It's just that being able to consult with the dead, the distant, and your own past was an offsetting benefit.
There was a decline in brain size that greatly predated widespread literacy:
https://www.bbc.co.uk/future/article/20240517-the-human-brai...
FYI: The rhetorical fallacy here is called a false syllogism.
People using books did in fact lead to cognitive decline since remembering could be offloaded. But that didn't stop the spread of books, was the point I tried to make.
Same for social media. The negative effects have been apparent for years.
I don't think we can go back to a world without books, without social media and without AI.
Not doubting this one bit but is there actually any research linking AI use to Alzheimer's? I think that takes long-term studies.
Comparing it to Alzheimer's is an allegory. The cognitive loss is already well documented. I might also compare it to Chronic traumatic encephalopathy. But I think more people understand what Alzheimer's is.
> I think we need to accept that AI causes cognitive decline
That is catastrophic because I heard several CEOs that said that we cannot do anything about it and that the technology is here to stay. So, I suppose that we will just need to adapt to a world with declining mental cognition for millions of children and a future with diminishing production capacity because ... checks notes ... this quarterly profits.
abolish homework
I got graded homework even in a few college courses and thought it was dumb, but then I realized it's always in the low-tier classes where they can't trust the students to study on their own. Those who skip homework hurt themselves by doing poorly on exams, but that's not enough to stop them. This ruins the curve in ways the instructor can't really fix unless they're prepared to fail half the class, which they aren't.
Around here kids get homework in kindergarten. It's absurd.
Errr, what does it contain then? :-D
This sounds really strange, never heard of.
BTW / Offtopic: I have read some weeks ago, today if you are hosting a party for your kids birthday, parents are preparing "give away bags" for handing over when leaving the party
Yeah goodie bags are totally a thing.
Really well thought out goodie bags are a treat, they follow the theme of the party and they have memorable (even if low priced) gifts.
Most goodie bags are filled with disposable toys that break after a few minutes of play.
The often underworked (most are autograding these days and reading old powerpoints rather than do real curriculum development) and (for the quality) overpaid teachers/professors will mass riot against this because it means they might have to actually teach. We might even get yet more anti-boy treatment than already exists among them as a social class in retaliation.
Please also abolish PowerPoint and force oral exams while you are at it.
Teachers are also luddites as a social class. Math teachers cried about calculators yet everyone of them who refused to adapt did disservice to their students (especially anyone doing statistics). My state's education system is utter trash despite being well off economically because of strong teachers unions keeping them from undergoing even a tiny bit of scrutiny.
They do not deserve the social credit/grace that they get. They are ultimately gatekeepers, whose extremely biased decisions in grading/treatment of their flocks decide which kid grows up on the streets and which kid grows up to eat lobster thermador every day.
Teachers also should not be disciplinarians. Children with discipline problems bad enough to disrupt a whole class should be thrown out of the class room as basically the only thing a teacher does for "discipline". The "right to education for the shit kids means we can't do that" framing is such trash for those who want an education and will inevitably be disrupted by a small minority of terrible students.
Also it's interesting that one of the only other things Max Stirner wrote about besides philosophy was education and his myriad problems with it (given his history as a school teacher).
https://theanarchistlibrary.org/library/max-stirner-the-fals...
I like school and think teachers are good and that your comment is the nonsensical ranting of a lunatic and as a result don't feel compelled in the slightest to read whatever it is that you linked.
What a great outcome of the school system!
This is the answer. Many (all?) European schools don’t give homework (this is partly because of the way the grad programs work). It’s on the students to study in order to pass the (difficult) exams.
There is no such thing as a European school. Each country is different. And if the country is a federation, each state might be different. And we have homework in most European countries I’m aware of.
What are you basing that claim on?
A European school is a school in Europe. All of my European peers in grad school reporting similar experiences.
Ah you’re talking about university. There isn’t a European-wide rule or custom for homework, it will very likely be up to each university and/or teacher to decide if they want to give homework or not
Seems this did effectively abolish homework if AI is now doing it all for them. The result seems to be that fewer students pass the exams, not more.
As with any skill, practice is necessary to achieve competence. If homework is no longer a viable way to force students to practice, you'll need to find some replacement, you can't just abolish it and expect there to be no negative consequences.
The replacement is students maturing into functional adults who learn they must study to pass exams, not to treat them like babies.
> The replacement is students maturing into functional adults
By what means? Have you got a magic wand that will turn 3rd graders into mature functional adults who will study on their own without homework?
I’m talking about university students. I don’t think I third graders are turning into functional adults any time soon.
Okay, but this article isn't about that:
> The study followed 27,000 pupils aged 12 to 18
The same applies to high school students. Just make it exam based and give suggested reasons.
(Third grade is 9-10 years old I believe.)
Too many of them don't really care. Otherwise they'd be doing the homework for real before they cheat it.
I don't think this is true, and frankly, I don't care what they do in Europe.
It is what all my European peers reported to me during grad school, and you should care because it is an example of a system which solves this problem successfully (at the university level).
Oh, university level? The article is about 12-18yo. American grad school already doesn't have graded homework, and undergrad sometimes does or sometimes not.
I’m talking about the university level. My peers in grad school were reporting their university experiences.
And American grad school does have homework. I’m not sure why you think it does not.
I know someone who pasted all their six sigma certification questions directly into an AI tool and they got a 98% on their exam. Seems like it would help their exam score too if they were allowed to use it on both the homework and exams /s
Should have let the kids use AI during exam too, later in life they will be solving tasks with AI not manually. Just make the test questions difficult enough even with AI.
I see our current approach like a driving school training its students at fast running.
The goal of education is not and should not be creating human robots for a specific type of job.
There is problem in both cases. - Without AI, they are just rot learning. - With AI, whoever will have advance AI will top the class.
This feels like the calculator debate all over again.
Yes, using a calculator leads to worse outcomes in your learning. But also, yes, everyone always carries a calculator on their person 24/7 these days and you'll never be as fast and as accurate as a calculator.
I think it would have been kinda cool if you could have unlocked the privilege of using a calculator in school through good grades in math class. Prove you can do what the tool can, then you get to use it. Would have also provided at least one incentive for students to get good grades in at least one class.
> Yes, using a calculator leads to worse outcomes in your learning.
Does it? https://www.jstor.org/stable/749255
Offloading problems makes it so you do not have to solve them.
If you don't have to solve a problem, your brain does not have that work to do.
Your brain working less means it has less opportunities to train.
The consequence is worse cognitive abilities which is tied to worse learning outcomes.
-
From your link:
> Sustained calculator use in Grade 4 appears to hinder the development of basic skills in average students.
It noted the following positive outcome:
> apparently improves the average student's basic skills with paper and pencil, both in working exercises and in problem solving
But this only measured the outcome for solving math. Yes, offloading problems will make it easier to solve problems, at the cost of not training your brain, which leads to worse learning outcomes across all subjects.
You do sometimes get to unlock it by being allowed to use calculators in more advanced classes
When you get out into the real world, what will matter is how effectively you use AI. When you are done with school, you can take the "exam" using AI. The problem is that teaching is trying to teach skills that are no longer relevant and is always behind what the latest going on in the industry. Every time without fail you got interns, who after two years at stanford would learn more in the 3 months on a google internship than in those two years at stanford. They learn useless shit.
My own experience is that college is fun. You can spend the time wisely, but school stuff, is not what makes you money.
Make student loans dischargeable in bankruptcy and make colleges under write them. The problem will get solved quickly.
Way before AI, the problem was very similar - universities do not teach practically useful stuff. Its been a permanent complaint about the education system, really, at least for the last 50 years.
AI just gave the students a way to dodge the slog.
But university was never intended to teach the bleeding edge. That would really be impossible in practice. Pre-phd, it is supposed to teach ways to efficiently attack a problem. you can take a bunch of "play courses", and succeeding at any of those requires pretty much only that one skill.
Nowadays, the challenge they face, is to keep teaching "problem attack methods" in a way that can't be trivialized by AI.
Though fundamentally, if you go to university, and evade learning the one thing you can learn there - thats your loss.
Guess what this is probably fine. AI helps people do their work on a daily basis and probably makes them memorize less things. That’s how the world is moving. You don’t need to be able to memorize something for an exam to show mastery of a topic. And we shouldn’t organize our society around memorizing things for exams.
What is a doctor? The person recommending next course of action based on best practices. What is the best practice? Is it a globally decided on thing, a locally tested thing that has proven to be better? Is the global thing just the most popular because someone wrote it on reddit and it got millions of views (but isn't actually good).
If our society is organized on just delegating all authority to "the best answer" we can drop all jobs - and only the surgeon has some authority over the machine. When we have a surgery robot, then they can go to. And at some point we'll be recommending and forcing surgery for athletes foot because some kind of best practice or viral post going out of control and a bunch of openclaw bots randomly decide to make tons of recommendations online for amputation for athletes foot and GPT 8.6 agrees this is the way to go.
OR we can somehow decide there is a boundary, there is some room for human memory and decisions.
I actually think having a single source of truth would not be terrible. Let me explain.
All podiatrists are not the same. All physicians in their specialties are often specializing in their own niche and are up-to-date on that niche.
I've had lisfranc ligament injury. My podiatrist wanted to perform a surgery quickly, which in 90% of cases is advised. I decided to seek a 2nd opinion because I realized she did the X-rays wrong, as she did them non-weight bearing, which is useless if there are no fractures.
The specialist was excited to see me, and up to date on all the lisfranc research. I've had a really bad sprain and he said conservative treatment should be sufficient. It took months but I healed up.
I've had quite a few injuries from playing sports and often found doctors to be "wrong" simply because their knowledge was out of date.
I am confused as to how this is an argument for a single source of truth. It seems like you've presented an argument for the opposite. If there were a single source of truth, there would be no point in seeking a second opinion - it would be the same as the first. If the first opinion was correct, excellent. But if the first opinion was incorrect (and no AI model is correct 100% of the time), then you're straight out of luck.
The problem is that the first source is incorrect because she doesn't deal with lisfranc injuries frequently enough and her knowledge is not good enough. I've had to go do my own research to find a guy who specializes in lisfranc. Most patients would not.
However, if you went to an AI agent which has access to medical research, you would get the most up to date research.
Why would I seek a 2nd opinion if the initial doctor knew what to do?
> I actually think having a single source of truth would not be terrible.
We should establish a single practice of care, MedicineGPT. It may be a rough start but over time the gene pool will adapt and the human race will live harmoniously with the machines (sorry, with the GPUs). Plus our insurance premiums should finally go down.
It would be better than most physicians now. It can quickly crunch many pages of medical history and medical research, this something a physician cannot do. They have intuition but sometimes are not equipped to deal with complex problems.
While I am not 100% opposed to this idea, it should come with liability to the provider AND its error rate be better than best human doctors.
> What is a doctor? The person recommending next course of action based on best practices. What is the best practice? Is it a globally decided on thing, a locally tested thing that has proven to be better? Is the global thing just the most popular because someone wrote it on reddit and it got millions of views (but isn't actually good).
Ironically I had a friend who went to med school and in his own words roughly 90% or more of the students there are heavily relying on GPT. They even had a few instances where people were using it to cheat during exams. And supposedly during the clinical years of uni if they don't know something they literally just go to a separate room and start asking GPT for answers. Make of that what you will.
This is too simplistic. There are disciplines, like Anatomy, where you're basically just memorizing bones, glands, and other parts of the human body. For organic chemistry, students need to memorize most of the krebs cycle diagram for their exams (because doing so and drawing it lets them reference it later).
Thinking is infinitely more complex than "critical thinking" vs. rote functions. You want students to get to critical, eventually, but the lower orders on bloom's taxonomy serve important roles, too.
Course design built around multiple choice questions, essays, and a lot of the historical assessments that we're used to are largely not great for learning in the first place—–they're used because they can be auto-graded or score assigned quickly. AI, conversely, is kind of revealing the limitations of these outmoded learning design practices.
Think of the meme going around right now from Milennials to Gen Z: "I wrote a 5 page essay on a book I never read!" <- They're saying this to AI generated essays as if there's any difference in the critical thinking that comes out of an exercise like that (read: absolutely none).
AI is a new tech—so of course people are still figuring out the limits of it. In 10 years a lot of the paradigm will be set. Until then, students are unfortunately kind of in the thick of it.
>Think of the meme going around right now from Milennials to Gen Z: "I wrote a 5 page essay on a book I never read!" <- They're saying this to AI generated essays as if there's any difference in the critical thinking that comes out of an exercise like that (read: absolutely none).
As somebody that read every mandatory reading book, I wish I had had the critical thinking skills to not read them. In retrospect, they were a waste of time and made me absolutely hate reading.
How do you differentiate between mastery of a topic and mastery of using an AI in an exam setting?
Open book tests and allowed cheat sheets already solve the memorization-is-not-required gap. It seems like allowing use of AI pushes that closer to thinking-is-not-required. And if no thinking is required, then how can you prove personal mastery?
If no thinking is required, why do you need thinking in the first place. Seems useless. If an ai can do it, why you need a human to be able to do, human should be doing something the ai can't.
Good exams don't just test memorization
Yup, when you find such exam please let us know.
I had the privilege of being educated in several countries and cultures. This might come across as "your education system sucks", and that's essentially what it boils down to, but on the upside I'm truly sorry for you and will oppose a comment to your cynicism. Good exams test your ability to express coherent thoughts about the concepts you learned, and let you generalize upon them by having you use and combine them to solve cases and problems that were not those seen in the teaching material. You can even teach and test history that way, and make it much more insightful and captivating than a laundry list of dates and events (as I suppose was your case?)
Same, but all the exams I ever took mostly tested your memory more than anything else. I don't think it's possible to test people at scale any other way.
Mastery requires memorization.
You can't play an instrument without muscle memory.
Why are commercial airline pilots required to spend hundreds of hours in simulators? So that when things go wrong they don't have time to think, they just react. And their reaction will be the correct course of action because that's what they have drilled on.
How many times have you gone "Oh, this new thing I have just encountered reminds me of this other thing?". How can you make such leaps without knowing that other thing?
Completely agree.
Absolute waste of time getting babies to walk now that we have cars and electric scooters!
> to show mastery of a topic
How do you show mastery of a topic where you neither understand fundamentals nor have skills to do a task without help?
Mastery is about demonstrating skills. Skills take practice, hands down.
AI can help us build better tools, provide tutoring, answer broad ranging questions, help relate concepts, and do reasoning for us. All of those things can be good in that they can help us acquire skills faster, e.g. with better practices and exercises. But it cannot replace actual time spent doing a thing.
Wanna learn knitting? You're going to have to knit some stuff.
A lot of expertise is actually knowing things about the domain. You can't make a good judgement call on a topic without getting relevant knowledge in your head to make that decision. You can't even lookup the relevant knowledge without knowing what knowledge is important to make the decision, so reference material is not enough.
I can get quite a lot of knowledge on a domain pretty quickly these days. But yeah domain is important
Memorizing stuff is good for understanding. Of course memorizing without understanding doesn't really help, which is what a lot of student last-minute "cramming" boils down to. But without concrete and memorized knowledge, any abstract reasoning skills become quickly impossible.
Wrong. Memorizing without understanding is pointless. And I'd argue that rote memorization does nothing to guide ones understanding. Because if you actually understand a subject deeply enough you can literally derive almost anything that would need to be memorized on the fly, and if you can't (say constants, or spec information, or anything else) you can simply surgically look it up online (or via an LLM).
> Wrong. Memorizing without understanding is pointless.
GP said exactly the same thing.
I agree with GP’s broader point, which is that you cannot have understanding without a specific set of baseline facts. The student studying only those facts and never achieving the understanding they are meant to support is a tragedy of our education system.
There are people who need to understand microbiology at a specific level and they cannot derive it all on the fly. They can achieve better memorization through understanding but I personally have found rote memorization can be an essential first step in some disciplines.
Have you never watched The Karate Kid?
"Wax on. Wax off" -- memorization and repetition are the basis for understanding.
How useful it is to remember useless information that one day turns out to not be so useless after all.
A good analogy to caching and memory lookups. You store in your head only the most valuable immediately necessary information, the rest can be outsourced.
"You don’t need to be able to memorize something for an exam to show mastery of a topic."
Yes you do???
This is not, in fact, fine. The point of education is to learn things, and these results show that kids are not learning things as effectively when they use LLMs.
Definitely not fine. People are graduating from Computer Science programs literally unable to write a line of code without AI assistance.
These people have fallen into OpenAI and Anthropic's trap and are illiterate without the assistance of coin-operated slop generators.
Yup this is the devious plan.
Thank god for china!
so how _do_ you show mastery of a topic?
One more step towards eliminating the value of labor entirely. What "work" will people be doing on a daily basis, and what makes it valuable?
> You don’t need to be able to memorize something for an exam to show mastery of a topic.
Yes, you do need to memorize and know things about the subject in order to be a master of it. What is the definition of "mastery" that you are using? What are the Things being memorized?
It's critical to understand the fundamentals on your own and that is different from AI being used in public. Sure school doesn't have as much value as it used to but it was always more about social credit or a certification that you took the time and effort to understand the world under pressure (obviously not true for all disciplines) or goals attached preferably in group settings.
This is something that AI cannot replace. Viewing things as memorizing is the wrong take.