There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities.
The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM company, and still then it's a struggle. Those few that do train, they spend most of their budget on compute and have relatively small teams.
Getting experience in this field requires having access to very expensive hardware to begin with. And the skills will be quite hard to convert into any real value for someone, leading to a decent income, unless you have a ton of funding from patient investors, or you have decent contacts in Bay Area networks to get hired at the right place.
With all due respect, paulg is in somewhat of a bubble, this is not congruent with the global situation.
I agree with mostly all of this, but personally I wrote a toy LLM almost 5 years ago and while it never saw much use outside of boring my wife with a shitty command line demo with glee it did help me understand how they worked and how to apply them, played a lot with JAX and pytorch, ended up building a ghetto version of MCP and an LLM-Pool to proxy requests to my baby local models and so I didn't struggle to see the evolution of openrouter and MCP agentic workflows. The same way i'm really glad when I was younger I built a bad webserver by myself, a really painful SQLx type database, etc etc etc - none of these things led me to developing for Nginx or Oracle nor will knowing JAX get me a job at an AI research lab, but I do have a lot of depth in understanding how the technology works so that the flavors on top of them are easy to digest and make more use of immediately, and I think the same can be said for engineers coming into the field - if it's a spooky LLM box you aren't going to be squeezing the same amount of juice as the guy that knows how they work inside and out so having at least the understanding of a _babys first LLM_ is going to get you miles ahead of people who don't.
For anyone who wants to dork around there is https://github.com/rasbt/LLMs-from-scratch which is something amazing that I think anyone who wants to engineer things around LLMs should at least blast through and read.
Game cheating and reverse engineering MMO backends taught me a lot: databases, networking, securing a backend (and frontend), limitations of simpler languages when comparing them to more native options for building backends.
Yes. It's like looking at the (Apollo) moon rocket launch and then suggesting teenagers should learn to build rockets in their garages for the coming space age.
It is viable as a toy project, but there are vanishingly few career opportunities.
I mean... that's actually amazing advice. Not because they would grow up to create browser startups. But because they would grow up to create web startups that succeed because of very fundamental of how the web is rendered.
Really? The latter was immediately useful to lots of people which is motivating, and it had a nice smooth learning curve (html -> js -> php -> databases -> apps -> backend). Learning HTML is the first step to learning how to make full blown apps. Making a browser at 17 is like trying to climb Everest as your first hike. The expected outcome is burnout and demotivating failure. At best you'll learn some C++ or Rust.
17 is an interesting age. There are way too many comments here saying things like, 17 year olds should just do whatever seems interesting or bum around the world or focus on getting into university. But historically most kids were expected to be productive adults at 16 or 18. 17 is about the right time to be thinking seriously about what kind of work you'll do, how you'll make a living. University won't help and will just delay this decision.
17 is a weird age but ymmv. I left home at 16 alone to study abroad. I had tons of free time due to dorm curfews and such. Unfortunately, I was too poor to have a computer and the computers we had access to were completely locked up. (Naturally we waltzed past the locks to play some games but it was also under surveillance)
Paradoxically I coded way more between ages 12-14, I regret my wasted late teens.
I contributed to a browser engine around that age (KHTML, which later became WebKit and Blink), and while I don't work in browsers right now, much of that knowledge, mindset and of course the professional network have done much to shape my life. And a fairly successful career, for that matter.
Early web didn’t have JS. Nor PHP. In fact a lot of early web pages were written using static HTML with C++ invoked via CGI/bin for processing form data. So writing a browser would teach you the HTML plus C++ too.
The early Internet (which the GP mentioned) didn’t even have the web. But that’s nitpicking.
"Necessity is the mother of invention" - limited hardware has always forced people to find cleverer ways of doing more with less. Current models are clearly nowhere near the efficiency limit (the brain does vastly more with far less power).
And so I think the idea is more to understand tomorrow ... from first principles.
In the late 80's, as a teenager, I learned x86 assembly and C because that was the only way to squeeze out enough juice from my shitty CGA (and later VGA) card to programm the games/graphics that interested me.
I haven't written assembly in years.
But whatever I did in my career: it helped me and gave me an edge over my peers to have a foundation that is very close to the metal.
FWIW, at least 20 Y Combinator startups have published ML research recently at ICLR, NeurIPS, ICML, and so on.
I think a lot of people assume that only the big AI labs can do cutting edge research, but there's a strong argument you can do it as part of little tech as well.
That's like saying the only way to do real engineering is with Google-scale Borg deployments. You can do quite a lot on very little hardware, r/StableDiffusion is a prime example.
You can do plenty of "real engineering" under normal conditions. But specifically when it comes to LLM engineering, no there's really not much you can do, they are called "large" for a reason. You can play around at small scale, but those lessons you learn will not be very relevant to the real problems in the market.
Sure you can gradually climb the ladder by demonstrating your skills bit by bit and getting access to more resources. It has very good prospects if you do manage to push through. But it's a hard and risky path, and you will not be able to get any interesting results for the longest time.
For a young middle-class student, it just doesn't make much sense. You can do much more impressive and impactful things with your time without getting into that black hole.
I know how to build an LLM, I know plenty of fellow young engineers that do too. It's really not that complex. But they can't do much with it without capital or access.
Good engineering has never been a bottleneck in this field, it's been all about having access to capital and taking smart but dangerous risks burning it on compute, without much idea of how long you need to keep burning for. There's still no end in sight, some are still managing to convince investors and keep burning, and we are seeing progress, but the business case is still unclear. If you want to get in that game, go ahead, but it's not something I would advice the average young engineer.
> I know how to build an LLM, I know plenty of fellow young engineers that do too. It's really not that complex.
I’m 40, and I don’t.I took that abstraction for granted and “left it to the big labs”. However I want to build my own LLM for learning purposes.
On needing big expensive hardware.. necessity is the mother of great innovation. Perhaps 18year olds trying to build their own LLMs in constrained resources environments will result in ground breaking ideas of achieving better intelligence than the one we currently have….
The world needs pragmatic folks who work at a higher abstraction and make LLMs useful, AND also folks who think why not “this other way”? And build newer ways to do fundamental things.
Given the usefulness of current LLMs, I would certainly encourage anybody to try and build their own LLMs, and see what they come up with…
Heck if they build a rack full of old laptops and run something with it that could be done “better” with modern servers, I’d still appreciate the learning running things on those little machines bring.
Well, that's not how it works. You don't just put some old laptops into a rack.
Maybe with a decent consumer GPU like a 4090, you could look at doing experiments like distilling and fine tuning a small image model for edge deployment for specific tasks.
Probably even there many use cases may require renting compute for $10/hour and investing a few hundred.
A LLM from scratch? Forget it. You can do theoretical experiments, but not build anything remotely useful with that kind of budget. If you're such a genius as to come up with revolutionary methods, maybe an university or actual AI lab would be the place to be.
To me the bottleneck is not even the compute, which is an issue for sure, but the data. All these large companies got their hands into petabytes of data, a lot of which of illegally acquired, but now they are large enough to pay the fines.
> On needing big expensive hardware.. necessity is the mother of great innovation. Perhaps 18year olds trying to build their own LLMs in constrained resources environments will result in ground breaking ideas of achieving better intelligence than the one we currently have….
Agreed. It's hard to learn unless you have access to quite high end hardware, and even paying by the hour is expensive. There's a low ceiling on what you can learn without doing training runs.
You can however learn everything you need to know to get on the career ladder as a software engineer on a regular home PC.
While the topic here is narrow, the concept is broader.
Do you take the first step or rule it out because you don’t yet see the complete picture.
As a teenager I never hesitated to try things out. As a young adult I wanted the whole picture. Now I’m back to playing / trying things out. I kinda wish I’d not given it up. PG being a bit older and reminiscing - I bet he’s in that bucket too, whereas someone trying to establish themselves professionally probably (aka me early 20s) wants to see the path.
> . But specifically when it comes to LLM engineering, no there's really not much you can do, they are called "large"
The “large” qualifier dates back to pre-transformer language models, where even training a multi-million model was hard due to how poorly it scaled. GPT-2 was a large language model, despite being only 124 millions parameters.
Due to how much high quality data is readily available, anyone can now train a sub-billion (L?)LM on commodity hardware.
And I'm personally convinced that pretty much any enterprise use-case of an LLM (except coding) is better served by a fine-tuned small (<2B) model that is trained specifically on the task, rather than a generalist frontier model, so learning the engineering around fine-tuning is a key skill that companies will realize they need sooner than later.
Why? Oersted is correct, for any size class you can find an LLM that is free and well trained at this point. They are highly adaptable even without fine tuning, in-context learning is still superior to fine tuning in most cases also. And real world fine tuning is mostly about data gathering and cleaning. The actual adapter training is automated and put behind simple APIs.
I saw my first language model in action in 2014, I was writing blog posts about them back in 2016. In recent years, like many of us, I spent some time learning ML frameworks to see if it'd be a fun career pivot.
But:
1. It doesn't seem especially creative. All the frontier labs have nearly identical model personalities, capabilities and even app designs. We're not seeing them differentiate from each other, implying that the design space might not be that large. In which case the opinions and unique approaches of specific engineers aren't that important, they are interchangeable at the right level of skill, and what to do next is usually obvious to everyone.
2. It doesn't seem like a big job market. A lot of ML jobs were wiped out in recent years by the rise of LLMs. Lots of NLP specialists etc were suddenly replaceable with a cheap API. The jobs that remain have compacted into a small number of companies. It's a small community which greatly increases career risk, especially as so many are unprofitable and/or have strong ideological requirements.
3. It's unclear how much demand for better models there actually is. Do we actually need smarter models? In robotics clearly yes and robotics is interesting and high potential, but for pure LLMs/image models, most users are already incapable of setting tasks that stress the best models and are happy with the cheaper smaller ones.
Using the models on the other hand is a very large design space, and has a lot of scope for creativity. I see use cases for AI everywhere, but most companies seem to stop at putting a chatbot on their website or asking Copilot to rewrite an email before they send it. A lot of companies have hollowed out their IT departments over the past twenty years. It feels like a new golden age of consulting work could be upon us.
I really don't think I'd tell a 17 year old to learn how to train LLMs. Learn how they work and how to use them, sure, absolutely.
No, but the skills Google needed (back then) were just normal programming and sysadmin skills scaled up. I had eight interviews that covered Linux sysadmin, programming, debugging, networking, maths and more. If Google hadn't wanted me there'd have been plenty of other companies who needed those skills.
If you look at model training jobs a lot of the work at this point is creating RL gyms (normal programming work), but most people still think the work is all neural architecture research. Doing the former is fine but won't teach you much about how to build LLMs, whatever that means now. Doing the latter is a very hard market to get into: not many jobs and requirements are often like, "you must have published at one of the following conferences". Prior experience is assumed. Most of them seem to treat Google as ML university and source of new recruits. It's understandable given the cost of training runs.
> requirements are often like, "you must have published at one of the following conferences".
I'm not sure why it's like this. If you look at the real world, you have stuff like ggml, which is about as hardcore as it gets in the LLM space, and it was made buy just a guy. Same for this like ComfyUI
If you get enough academics in a place, they tend to close rank, and not let anyone in without the same credentials. Data science used to be like this, they were constantly on about how you need a Math Phd to even apply, yet when I met these guys IRL, most of them were just running Python math libraries.
These previous examples show that if you understand at least a part of the problem space, you can 100% contribute without academic credentials.
You can do it at home, for sure. I've done some NanoGPT training runs and modified the architecture, it wasn't that hard. Came up with some potential research ideas too. It does take money for GPU rental so for a 17 year old, it's not so easy unless their parents give them a budget. For an adult with income you can do it.
The question is more one of opportunity cost. At 17 you need to start finding your way in the world. It's best to learn skills lots of people need.
Interesting/capable diffusion models are much smaller than similarly interesting language models. But yes you could always scale things down to learn the fundamentals.
I think a lot of this is based on preconceptions. A lot of apps were made with Electron, because it was common wisdom that native is 'too hard'.
Now with LLMs, people write native apps in Rust, and I'd like to think some of them found that there isn't such a huge jump in difficulty they assumed there would be.
I just finished fine tuning Gemma e2b for local code completion on my local machine.
This comment just reinforces what the post actually means. We need people that are LLM natives, computing solves itself with time and with scale adjustments
> +1, I have a friend, math PhD that's been working on ML research 5+ years in London yet he has not been able to find any position.
Maybe people simply don't want math PhDs but something else? Since 1-2 years ago I started doing consulting/freelancing in the ML space, but more on the infrastructure, deployments and similar stuff, as a general purpose developer, and I have a waiting list of clients interested in more work, some of them even trying to recruit me to work for them full-time as well. I'm based in continental Europe, fwiw.
How've you gone about getting into this btw? I have extensive experience in infra and pipeline rollout but have struggled to find freelance clients for this kind of thing. Would be great to tie it into ML as a learning opportunity there
Spent a year of freetime catching up on everything and learning as much as possible, started sharing what I've found works or not, write a bunch of comments on HN and elsewhere, and have a email in your profile, eventually people will find you if you put out good stuff :)
Also bunch of past workplaces who've adopted AI in various ways who reach out once they find out what my current focus lies, but that's harder for others to replicate unless you've already had a career as a developer.
This only proves the original point which is that there is not much demand for actual machine learning expertise because that is only carried out in a small number of places and what demands there is is for the more basic software carpentry like infrastructure and operations rather than the actual technology and Engineering side of things
What parent says about "there are very few available positions" for "engineers that can do real LLM machine-learning" is fair, yeah, I'd agree with this.
I don't think the "incredibly small minority of companies in the world do any real training or optimisation" part is necessarily as true, as some parts of the work I do get is about helping them optimize training and infrastructure around training. Mind you, none of this is for building LLMs from scratch, it's 99% fine-tuning existing checkpoints.
I'd also agree with "paulg is in somewhat of a bubble" regardless of this, which is worth remembering whenever you read his content. Same goes for any person living in SF, and dare I say the US. But also, YMMV, I live and work in Europe, probably why I have this perspective.
I think he does, but it seems he still considers himself a remote SV native rather than someone working in the UK's economy.
Worst of both worlds - no casual gossip feed in the Bay Area, no big dog meetings in the UK. (Which mostly has no idea he exists.)
As for the question - what are the odds LLMs will be anywhere near the top of the tech tree five years from now?
The trend seems pretty clear to me - local/offshore models are snapping at the heels of the big names in the US, and the current investment arc is insane.
I wouldn't bet on Anthropic or OpenAI being leaders five years from now. Longer term, I especially wouldn't bet on the US build-yourself-a-monopoly corporate model surviving AI at all.
I mean it makes sense right? Anthropic for instance has like, a couple hundred staff in London with plans to expand to somewhere just shy of a thousand. There are far far more ML/Maths/CS/Stats PhD's than there are openings. Especially in London there is no shortage of suitable candidates given Cambridge/Oxford/Imperial/UCL are surrounding it. 2% of the UK population has a PhD alone...
I think his point is, if LLMs are the future (like computing is the future in the 90s), you should be an LLM-expert (equivalent of becoming a software developer).
I can see the point. It's unlikely that a 2.4T LLM will be integrated into, say, a pesticide drone. You'll still need some kind of LLMs to achieve maneuvers that "normal" programmings can't achieve.
But what if everything basically turn into that? Essentially, instead of build me a web app to solve X and do Y, build me an LLM to serve X and do Y. (unless the current LLMs are able to do it end-to-end but then they can hardly write coherent software/personal opinion).
I think companies of all sizes will want their own models, or at least customised ones, for their own specific use cases or competition and security issues.
1. Both training and optimisation will get significantly cheaper and easier quickly.
2. Politics will probably get even more insane before a potential reprieve on the 20th of Jan 2029.
3. The big AI firms will become part of the surveillance capitalism network, if they're not already.
So I think for self-protection a lot of companies will be looking near to medium term AI independence.
The argument is sound, but the maths don't math for now, and it's unclear when/if they will.
For the time being, unless you truly have millions, the outcome from training will be very net negative, while focusing on building on top of existing AI will yield amazing things if you apply the same talent and effort.
When it does get cheaper, then it will be easier to acquire the skills and experience too, and the struggle you went through by trying to do it now will be somewhat wasted.
Besides, I am well versed in this field, and it is not rocket science. There are plenty of software engineering domains that are a lot more challenging, like high-end graphics, large-scale data engineering or kernel programming. People will learn to train LLMs when people want them to.
In reality, enterprises are happy to offload even risky tasks to others as long as they get some contractual guarantees about their data. Would they like more choice in who to buy from? Yes, but not enough to in-house such a specific discipline.
The cost of training a model from scratch is going to be cost prohibitive for the vast majority of companies (even if renting the hardware needed for the 1-2 month training time). It's an interesting learning exercise, and some of the things learned can be applied to other parts of the process. There's also the issue of needing a huge amount of data needed to get decent weights.
Fine-tuning a model or LoRA based on the companies data set is more feasible but you're likely going to need several runs as you test/try out different base models, parameters, etc. This is why there are a lot of fine-tuned models on huggingface based on base or instruction-trained models from the larger AI companies that have released open weight models (Microsoft, Google, IBM, Mistral, DeepSeek, Qwen, etc.).
Training is limited on memory first (storing training data and weights) and computation second. Realistically you need to own or rent 2-8 H100/B100 devices or Google's TPUs.
The majority of workflows for a company providing AI capabilities are likely best solved by tailoring a system prompt for the chosen model, evaluating the prompt and model with tools like promptfoo, and then running it on a compute cloud provider (including AWS Bedrock). If the company is big/financially well off enough they could look at buying the hardware needed to run it on their own servers.
For other uses like agentic software development you'd need to spin up a suitable model on a compute cloud provider (or local hardware if the model is small enough) and then tell your IDE/editor to use that model. You would need some way of benchmarking and evaluating the models to see if they are capable of doing the tasks you need. -- There have been some tests done by people on YouTube that suggests that Qwen 3.8 27B is a decent model, but your needs may vary.
Most companies that build physical goods don't care for one second about their IT department other than how much money they can save per month, starting by outsourcing whole of it, thus they have little use for internal LLMs.
And it's across the industry, thinking banks, private banks, insurance, pharamcy etc don't outsource their IT, including development...
I believe US outsource even more than Europe on this matter.
Except knowing how LLMs work don’t actually provide much understanding for using them. People don’t use LLMs the way we’ve built on most other tons or platforms. It’s more learning Unity hoping to be a better gamer.
That's not true, because everyone, everyone, everyone seems to want to do training. Which results in a 50 person company training, say, a voice model that then fails, because it's just not good enough.
In reality the problem is that it gets blasted out of the water by a much worse architecture trained on 10000x the infrastructure. And while I'm sure the freshly brought in ML student came up with a 10%, even 30% better architecture, it just doesn't matter. (and never mind that even OpenAI hasn't really solved a voice model yet. Try it. It can probably match 2026-quality call centers, but it's no substitute for an actually empowered human)
... and yet, if you look at what hyperscalers are getting paid for ... comfortably more than half the income is training. Which makes no sense on so many levels.
The big question is whether companies hold enough proprietary data to do useful things that for e.g. Anthropic, etc. can't easily replicate.
For some very niche cases I think this is probably the case but for the vast majority, the company's data isn't as useful as they think it is or anywhere near the size needed.
Everyone says they want to do training, because it's sexy and an easy way to justify raising mad funding rounds. Some manage, most don't.
I don't know where you are located, but in EU, in China, and yes even in Silicon Valley, the vast majority of companies do not do any real AI engineering. There's nothing wrong with it, it's just not a smart path for most purposes. You can do amazing things without training, and if you try to train, you cannot get anything amazing unless you burn millions.
Very few people can afford to play the long game and cross that dessert. And, sure, you will not get far without good engineering, but good engineering is definitely not sufficient and is not the primary bottleneck.
It took a long time to cross that desert, and no sane company would want to get stuck in a desert, unless it's specifically an R+D "desert crossing" company.
A lot of people here are responding to the message but not to the meaning.
It would be a good idea for young people to deeply know how these programs work. Not so that they can spend their career building them, but so that they can approach the next class of problems we'll all start trying to solve, with intuition all the way down to the weights and underlying mathematics. And also, to develop a healthy intuition of when "Just LLM it" will not be the right choice.
"Build an OS" wasn't a common university project because we were all expected to go out and work on Windows, but because understanding the bare-metal firmware for a computer helps you deeply understand how to intuit building for a whole class of problems.
Terrible advice. If I were 17, genetics and bio tech at the next frontier, with opportunities to be more than another corporate drone. AI is a lot of bureaucracy and nepotistic who ya know already.
I'm (more than) twice that age, but I've spent time learning this exactly this from videos by Andrej Karparthy and from books by Sebastian Raschka.
I didn't do it because it was useful to me in a practical sense. It's because LLMs are fascinating and I want to know how they work. From that perspective it's been a great experience. I have afirm grasp of the basics. This makes it much easier to understand frontier concepts like compressed latent attention. I can follow the field and understand it.
Not sure I would have got as much out of it at seventeen. I have a lot of background and experience which made it much easier to learn. I wasn't struggling with the linear algebra or with python. I already knew pytorch and neural networks. That helped a lot and I covered these tutorials fast and could skip over large sections. A few evenings and the odd weekend day over a couple of months was enough for me.
For seventeen year olds the tutorials are good enough to make it possible to learn this but it would have taken a lot longer to understand. On the other hand I would have learned a lot more. I think I would have learned a lot of valuable stuff.
However I also think 17 year old me was studying for his A levels and probably this was right choice in terms of maximising future opportunities. I'm not sure I think learning about LLMs instead is sensible. Indeed it might be bad advice. But I can absolutely agree with the sentiment.I think 17 year old me would have wanted to do this too.
While knowledge is always great, I would encourage people not to seek advice from successful people like this (survivorship bias).
Moreover I am not sure it is even good advice? Would you advise a 17 y.o. to learn how transistors work or how to code (i.e. is LLM training the right level in the stack)? LLM training, a discipline where relevant work is already out of reach for 99.999% of budgets really as essential as this post implies?
> I would encourage people not to seek advice from successful people like this (survivorship bias).
Personally I don't see the problem, as long as you're aware there is survivorship bias involved here.
What's the alternative really, seek advice from unsuccessful people? That seems worse :)
Personally I do both, read about what worked for people, also read about what didn't work for people, then ignore both and do whatever the fuck I want.
It's best not to take advice on direction of careers from anyone. It's better to find and work on things that interest you, and then take advice from people that are amazing in that specific field. In mid 2000s in Australia all the "top people" were telling me not to get into a software engineering career because it was dead. It's certainly challenged right now, but it took off during those 15+ years.
“It's best not to take advice on direction of careers from anyone. It's better to find and work on things that interest you, and then take advice from people that are amazing in that specific field.”
> It's best not to take advice on direction of careers from anyone. It's better to find and work on things that interest you,
Agreed, my previous stated "ignore both and do whatever the fuck I want" approach has worked out very well for me in life, people should probably focus on identifying better what their gut tells them, rather than what randoms on the internet thinks and writes.
I don't know; I think people in general are just not great at this. Successful people tend to underrate luck and overrate the brilliance of their own decisions, but the rest of us are prone to either reversing that and blaming everyone but ourselves, or being so determined to take accountability (or just depressed) that we become overly self-critical, or simply not understanding why things happened the way they did and reaching for any explanation that resolves the chaos into something narratively satisfying.
Intuitively, that'd make sense if that unsuccessful person eventually found success, otherwise who knows if they actually picked up what made them unsuccessful in the first place?
I think it's reasonably common for people to understand what makes them unsuccessful but be unable or unwilling to resolve the problem.
For me, I know that one major reason for putting an upper limit on my success is my inability to form effective professional relationships. I know I should go to events and talk to people and use these relationships to my advantage, but that's just not something I've ever been good at, and I find it so incredibly unpleasant that I also just don't want to do it.
Yep, many successful people greatly underappreciate the effect of simple dumb luck in their lives. And often they just make up complex reasoning chains, even wholly believing them, that more of it was in their control/talent, etc.
Nonetheless, there are many successful people I would gladly listen to for advice, though they are often successful in a different meaning than what venture capitalists would use (e.g. parents with great kids, managing to keep a healthy work-life balance, happiness, and maybe even having time to spend on some cool hobby project -- you are heros!)
same do unsuccesful ones. They usually overappreciate luck of succesful people from what I see, as a way of coping, many say "that guy was just lucky, i was unlucky". If you dig deeper, that person wasn't just "unlucky", they lacked the persistence, work ethics and other qualities compared to other succesful ones.
Of course, there are degrees and exceptions on every side, but the coping mechanism is very strong among people.
Luck is a huge factor in success, so that logic leads nowhere. If one of the quacks selling how-to-succeed recipes had got it right, everyone would do it by now.
Kids don’t even know what it is, even less actually feel what it is. At 17 you think that it won’t hit you, that you will be the one to survive until you don’t.
I would seek advice from people who have a theory of why or why not they were successful.
A lot of those results were happening in very specific contexts and usually should not be regarded as a blueprint, but as inspiration to whatever I do.
That’s what you get from listening to “successful people”. You get to learn about all the things they tried that failed, then the things that did work on that 24th try, which was successful.
The “survivorship bias” people always seem to assume that the “survivor” lucked into his fortune on his first try ever, so he can’t have learned anything, so we don’t have to listen to him. But that’s seldom the case.
> seek advice from unsuccessful people? That seems worse :)
Not sure why would you think so.
Inverse reasoning is very powerful, and unsuccessful people can give you plenty of "don't do this mistake", which the survivors would not even think about.
> unsuccessful people can tell you teach you plenty of "don't do this mistake",
But how can I know for sure that that particular mistake is actually why they were unsuccessful? Has exactly the same issue as listening only to successful people as they hardly know what actually made them successful most of the time, but they still compose large blog posts with their reasoning for why.
Again, I still think my approach of reading both but then regardless go my own way is the preferable approach, at least for me, ymmv.
An advantage that is not "advisable", like being born in january, in a rich country, in an above average family, or just having luck, might have more influence on the outcome than any conscious action. It is almost sure that one-in-a-million level people only edge over the other 999,999 they competed with is just "have more luck".
Right - like advising people to learn nuclear power back in 1992. Perhaps a good idea, but super specific already. My gut is ML/AI is even more complex in 2026… I can’t even remember all the abbreviations and the new ones emerging. And every sub component, such as attention or embeddings, are actually a discipline of its own already.
I think it is. He isn't saying to learn how to train a LLM so that you can go on to train LLMs. He's saying to learn it so that you gain a deep understanding of how LLMs work. Ordinary startups can still benefit from things like training or fine tuning highly specialised smaller models, knowing how to select and configure an appropriate model for the task at hand, knowing what software to use and why, understanding what's going on behind the scenes instead of treating everything like a black box, having a higher level of intuition about LLMs generally, etc.
Most computer science courses do in fact teach things which are lower level than coding, such as how transistors work.
> Would you advise a 17 y.o. to learn how transistors work or how to code
how many of us out here are doing work directly in what we got a degree in? I majored in economics and now I'm a CTO.
I would absolutely advise a 17 yo to learn how to code, understand how transitors work and how to code an llm. even if he never works on llms, you basically end up with a kid with applied knowlege of statistics, math, physics hardware, logic and a whole lot of practice in critical thinking.
IMO these kind of advices never matters. Any individual still needs to make tens or hundreds of little decisions (every day) themselves, and that's what really makes all the difference.
As a 17 y.o (way back in the last century) I didn’t need to be advised to learn about transistors. I just had a thirst for the knowledge.
I would encourage everyone to learn something about transistors. They are one of mankind’s most useful discoveries.
I just don't think he realizes how saturated it got over the years. Or maybe he knows at a conscious level, but not subconsciously.
In 2000 (his era), it would have been really smart to study the source of Linux or Apache. Would have paid dividends over decades. Cuz that knowledge was so rare. The number of people hacking on LLMs now dwarfs the number of people hacking on web servers 30 years ago, by several orders of magnitude.
And if you turn back the clock even more, I mean just even having access to a computer, let alone owning one, would have put you at a massive advantage.
I don't know what to call it. The pioneers should be respected obviously, but at the same time you need to understand that for them, the game wasn't nearly as played out as it is now.
I just don't think you can afford to be dicking around with LLMs like you could afford to dick around with random Linux distros 20 years ago. Too many people willing to do it for free these days.
You don't wanna end up being the 2030 equivalent of a certain SNES emulator developer, or maintainer of a package manager for jailbroken iPhones, I mean the list goes on and on. Being a hacker doesn't automatically give you a path to being rich, or even making a decent living. It hasn't been that way for a while.
I am kind of amazed how negative the comments are here, especially on HN.
Learning to hack something together in high school using the latest technology (vacuum tubes, radios, microprocessors, web/javascript) has been a common theme in the tech world for generations. With LLMs and online tutorials, this isn't even a difficult suggestion. Do people think learning new tech is somehow wasted effort?
> I am kind of amazed how negative the comments are here, especially on HN.
I can't recall or point out exactly when, but there is a stark before/after moment where the opinions of anything pg went from "Interesting and maybe true in some ways" to what we see today, lots of knee-jerk reactions and hardly any comments about the actual content.
Hazarding a guess, I think the moment Altman became the CEO and later during COVID, the sentiment seemed to have been shifting towards what we see today. But this is all based on hazy memory, rather than looking at the data. I'm sure there is a blog post waiting to be written about analyzing the sentiment of comments to PGs articles on HN, and you'll see a shift somewhere.
Because at some point in life everyone gets tired of fairytales. He started mending the anecdotes to his content which always rubs people the wrong way.
So, this comment of yours obviously isn't in the "knee-jerk reaction" category of comments, I suppose? What exactly from the linked tweet(s) are fairytales here? There is hardly any text at all, so strikes me as a comment about previous pg content, but then this would be one of those comments I talk about? Very confusing.
Your hand waving doesn't make it knee-jerk. It's just what happened to his writing since COVID. He goes for more of a shock and awe style and not everybody likes it. He's been writing for over 20 years now, hasn't he? His style has clearly changed, and an changing style attracts a different audience so it's no surprise his original readers might not connect with his newer work...
> I can't recall or point out exactly when, but there is a stark before/after moment where the opinions of anything pg went from "Interesting and maybe true in some ways" to what we see today, lots of knee-jerk reactions and hardly any comments about the actual content.
Hard disagree. This submission is still being highly upvoted, while another recent post[1] on the harms caused by Graham’s fellows[2], with a fairly tame comment section, has been flagged. That is a constant on HN. It’s not a fluke, it’s as predictable as the sunrise and getting more pronounced.
I’m sure we’re both biased in our perceptions. Mine is that HN in general (certainly more than any other website) used to worship[3] everything he wrote, together with others like Musk, until things started to really go to shit and many eyes have been opened to the effects of the unfettered greed of rich tech guys out of touch with reality.[4]
[3]: That word I choose hyperbolically but deliberately. It definitely was not “interesting and maybe true in some ways”, it was much more hardcore than that.
[4]: That is not “knee-jerk” but a slow realisation still ongoing.
> Mine is that HN in general (certainly more than any other website) used to worship[3] everything he wrote [...] It definitely was not “interesting and maybe true in some ways”, it was much more hardcore than that.
Maybe you need to enable "Show Dead" or something? Pgs articles on HN definitely never was free of any critique in the HN comments, just like any article. Although I do agree with you that it used to be different than it is today, and same with Musk too, and Altman, and probably more individuals, where they were lauded before but now pretty much just mentioning them poisons the conversation.
That post has barely any points and comments. It’s not a good indicator of general sentiment, it’s just an indicator of people who were on HN at that time.
> Maybe you need to enable "Show Dead" or something?
I have it enabled.
> Pgs articles on HN definitely never was free of any critique in the HN comments, just like any article.
Of course. I very explicitly wrote “in general (certainly more than any other website)”. That does not mean “always”, or “never”. HN is not a hive mind, there’s never going to be 100% agreement. The general trend is what’s being discussed, and we both agree that in general the sentiment on Graham used to be higher. We’re just disagreeing (we may be able to find ourselves agreeing through tough thorough thought, though[1]) on where exactly it is now and how to interpret it.
[1]: Sorry, can’t believe there was a real organic opportunity to use that sentence, had to take it.
Indeed, no hard disagree, merely details :) Overall you're right though, general/overall tone definitely shifted hard for a bunch of individuals over the years.
Completely agreed. The point is the knowledge, the learning and the journey. If a kid has a passion for building or toying with LLMs, then of course, by all means, please start tearing them apart or even build and train your own model. You'll learn a ton, even if you won't necessarily end up using it here and now. The learning experience will compound and of course that will be useful.
The above is, after all, the whole genesis of the word 'hacker'. We should celebrate that.
How exactly does one go about "tinkering" with an LLM? Any architectural change you introduce needs fine tuning. That needs data and compute
I tried to modify the embedding output of bert to make it generate box embeddings instead of point ones. At the time I had access to university provided A100 gpus but even with all that a training run took half a day. Models these days I don't think I can train it in any reasonable time with that much compute.
Now I'm curious, do people actually tried to hack vacuum tubes or other big servers that's barely 1MB RAM? It seems like another thing that needs big investment to work properly, unlike those other techs where results can be shown even with little materials.
Yeah, there are retrocomputing hobbyists who mess around with sometimes-physically-large computers that were important many decades ago. I don't know if anyone is hacking on vacuum tubes of the kind that you could in principle build a computer with - there's a reason they became obsolete for digital computation almost as soon as the transistor was invented. On the other hand, I personally think it would be neat to try to build a CRT in a garage, which is of course a type of vacuum tube. I don't think this would be an easy garage project, but it does seem like might be tractable for someone who understand physical manufacturing and electronics well, has access to glassblowing equipment, etc.
> Do people think learning new tech is somehow wasted effort?
No. But funnily enough that is a promise by some of the AI cretins and their boosters. Oh yeah best case scenario you learn how to build LLMs for us. We’ll employ you. And then ultimately that just becomes training data for the LLMs to do it themselves.
> With LLMs and online tutorials, this isn't even a difficult suggestion.
Don't many of the commercial ones prevent you from using them to build LLMs?
I would say the reason for the negativity is not because it's a bad idea for a project, or that doing projects in general is a bad idea (it's not!), it's because it's a very specific thing that is not for everyone. The best thing about computing is the low barriers to entry. You can basically work on anything that takes your fancy. So those who are interested in ML will be drawn to learn about LLMs. They don't need anyone to tell them to do it. Telling everyone to do it reminds me of the "just learn to code" stuff of a decade ago. No, please don't, please find something you enjoy.
I get this is basically advice for young founders and entrepreneurs, but i would ignore that request and encourage 17 year olds to spend time trying to find a happy medium between work and life.
Being a super rich and an unhappy workaholic, or a super-impressive engineer who wakes up one day at 45 and realizes they regret wasting half their life (I ran into way too many of these) is a much worse fate than "not being rich from your startup" and working a relatively regular job while feeling fulfilled and happy by more than just work.
Especially in the US, which is uniquely bad at this and encourages people to work themselves to death, mental health and work life balance are much more valuable things for 17 year olds to focus on than finding good startup ideas.
In case you think i'm being a bit dramatic, let's look at the state of 17 year old mental health in the heart of Silicon Valley:
"The City of Palo Alto and the Palo Alto Unified School District approved a funded contract to place 24/7 human security guards and monitors at all four local Caltrain grade crossings, including the Churchill Avenue crossing directly adjacent to Palo Alto High School."
(in case it's not obvious, it's because of suicides by high school students)
The 17 year olds do not need advice on better startups, and this situation will never get better if we focus our advice on how to be better at work instead of how to be better at life.
This will require redirecting the conversations.
I think a problem a lot of people are grappling with here is that due to LLMs and AI generally, it’s basically impossible to predict what the future will look like or what jobs will still be around.
I’d probably say something like: do something you enjoy and seems like it might be useful, but accept that the pace of change may mean that whatever you study ends up being irrelevant.
Whatever solution there ends up being to this, it’s not going to be one that an individual 17 year old can implement. We’re past the point where individual good and bad choices matter that much to economic outcomes.
I would advise any somewhat ambitious 17 year old to avoid tech and get into healthcare if they can stomach human interactions and bodily fluids. Sure, it is not all sunshine and rainbows, but there will still be plenty of work helping people who are ill or elderly. Even in the worst-case economic scenario, medicine will be a more socially rewarding and stable life path.
This seems like a much more interesting question to me.
Telling other people's children what to do is easy and basically doesn't have any downside to being wrong. With your own children things are a bit different.
So: what are people here with school age children telling their own kids about the future? Assuming their kids ask, what kind of careers would they encourage them to pursue, assuming they have the skills and interest?
When I was last in the Bay Area, maybe about a decade ago the bookshops were full of titles like "Python for Preschoolers" (I exaggerate, but only slightly). Clearly at the time a lot of people working in tech thought that cultivating an interest in programming was going to be the path to being a successful (by some metric) adult. Is that still the case?
certainly better than wasting time with harness and agent workflows that will become irrelevant at the next evolution, same thing happened with 'prompt engineering'
I'll use this post as a shameless opportunity to tell more people about a little side project, I made:
http://languagemodelbuilder.com teaches you (in a few hours to days) how to build an LLM from scratch. It's entirely free, without accounts, and without data collection.
Oddly, I just remembered I did the nearest possible thing to this when I was 17... back in 1991.
On an Amiga, I took various public domain text documents from cover disks and counted the probability of the next word given the previous word. Then spat out random sequences of words from it and printed them out. It was called "Splurge". Basically a very very simple single layer statistical language model.
Some of the sentences were randomly not bad sentences, which seemed amazing at the time!
That kind of thing (and Core Wars and Tierra etc) did lead me to getting a job at an artificial life startup at the end of the decade. But that was in turn about 10/15 years too early (no GPUs).
There's some lesson from this about timing, but honestly I've gained the most as a person when I did something that was fun, ethical and gained an audience. A tricky combination.
Horrible advice. This may have been good advice 10 years ago, but not today. There are no positions for people who "kind of understand how toy LLMs work" because so many engineers do these days. Most of the real LLM optimization work is at the edge of research and highly proprietary and not something you could ever do without infra that costs millions.
Throw the computer and the smartphone out of the window.
Or more reasonably, the same old thing : use Linux, hack a little, why not learn programming basics. But learn to own your technology, fight against centralization of technology. The same old RMS story.
IDK where the tech industry is going, if there will be jobs anymore or not, but what I'm sure (and what have been the case for the last 10-15 years anyway) is that for most tech jobs, having good technical knowledge beyond the basics is pretty useless and will probably not be recognized.
If you can, stay a computer geek if that's your thing, but don't make it your career choice, the Eldorado is behind us.
"it doesn't matter what you choose to study because no job is safe from being outsourced overseas, handed over to an indentured servant, or made redundant by a machine. you will spend your whole life surviving while the cannibalistic pedophiles who own everything invent new ways to make you own nothing. be frugal, don't get married, don't have children, do everything you can to stay healthy and independent, and you just might live a reasonably comfortable life."
Yeah he's a moron with a lot of money, that's about it. I'm sure he's said the same thing about various other bags he had bets on throughout the years.
If I were 61 and wealthy, I wouldn’t write a stupid shit like this.
I’m no paulg, but if you’re reading this - and you’re 17 - just focus on getting into a university and having a good time that you won’t regret later. Play games/sports, make relationships, fall in love, explore.
As opposed to what, neglecting the human experience to grind yourself to the bone for those who own capital, and then voting to uphold that capital? Makes no sense.
I mean, have you seen the options for people graduating right now? How people are behaving?
Or forget the data, look at how the story of the new future technology is being told. The people making it recognize that it has the potential to put swathes of white collar workers out of jobs, and they are openly talking/warning/PR-ing about it.
People in tech and SV, the places which have a underlying culture of near delusional optimism, are talking about trying to avoid being part of "the permanent underclass".
Gambling is up, and prediction markets are being treated as financial investments. Wall street bets is a thing, and outright speculative investments are the hope people have to get ahead.
This is happening in the USA, forget the weaker or smaller economies.
When people see the future as one massive zero sum game, with no way to win by building, then they are going to change how they plan their future.
Is going to university really that good of advice nowadays?
Everybody goes to college nowadays and the average white collar has lots of debt and relatively minor financial benefits over a skilled trade worker.
edit: woah, so many people insulted by that. In my bubble and friends, me and another friend are the only people that make very good money compared to non-graduates. Plenty of others opened their shops, went into trades, one learned to tattoo fake eyelashes, one became a (successful) farmer and most make significantly more than the average law/chemist/mathematician/physics/architecture/languages graduates. Sure, the lowest salaries are to be found among the non-graduates too, but I don't see any evidence that graduates make that much more, and that graduating is worth it.
Some answers talking about how "formative college is", but my 25 years old friend with her own shop knows more about real life, business and economy than ivy league MBAs.
Depends on where you live (lots of countries have no tuition fees or far lower than the US), what funding you have, how good a university you go it, what you want to do (some careers require a degree), whether you will enjoy it, and whether you are there just for financial benefits or more than that.
Its not good advice for everybody, but it is good advice for a lot of people. What if you want to be a doctor? What if you want to work in R & D? Not everyone enjoys working in a shop or a farm. Also, how old is your friend group? If they are mid twenties you are ignoring the greater scope for advancement in a lot of white collar careers.
> my 25 years old friend with her own shop knows more about real life, business and economy than ivy league MBAs.
Within the narrow limits relevant to her business. How much does she know about macro-economics or financial economics, or scaling up a business? I also suspect you are comparing her to people who went straight on from bachelors to MBA (which is a bad path - study business after having some experience IMO) and lack experience. How will she compare in 10 years time when those people also have real world experience?
It depends on the university and your goals, I suppose.
Many people are surprised to learn how affordable elite colleges are if you genuinely need financial aid. I had no idea -- was pleasantly surprised when my alma mater took over 80% off of my tuition.
Indeed, would the advice be good if university was free?
Where I live it's basically free and I still sometimes regret not going into the trades. But I suspect this feeling might mostly be a "grass is greener" thing.
Going to college is worth it for basically everyone other than those that are choosing between a mediocre college / mediocre degree and a blue collar route. AND if college is going to put you into debt.
Financially, intellectually, socially, it’s a good idea - college is a formative period of life in American culture. That is more and more true the better the college gets. At the upper tier (Ivy League, etc.) you don’t really have to pay anything if your family isn’t already wealthy, and the connections and degree you’ll make more than pay for themselves.
As someone who's at a similar age and was interested in learning how to do this, there just aren't enough resources to do so. Most LLM research is in the form of academic papers, and there isn't any 'popular' way to learn these things, and besides that all said research assumes you have a B200 cluster ready to go. If you have weaker hardware (say an 8GB nVidia GPU, which is what I have) you're going to be limited to fine tuning small models or torturing yourself working the GPU for days per iteration trying to run things like https://github.com/karpathy/nanochat, which is hardly an educational experience. Renting cloud GPUs is expensive, and at this age the most I could muster up for experimentation is probably $100 or so, which only gets me 25 or so hours on a B200 which just isn't enough. So why would I bother myself with this if I'm already at a disadvantage because of not having access to the right hardware and when surely there are better ways to spend my time? I concluded the only way to learn and be competitive is by finding work at an AI lab somehow (not happening at 17), or studying ML at the right university.
Putting this in contrast with programming, I learned coding when I was 8, and it was incredibly stimulating to learn because you can quickly iterate and there were thousands of books and YouTube tutorials that dumb everything down and teach you fundamentals. All you needed was a $300 computer, and you can learn nearly anything you want, without being gatekept from this or that because you don't have enough vRAM / an sm_100 GPU.
If I were 17 again I'd prepare to go for volunteering overseas after high school for 1-2 years (plenty of free options in the EU where you might only need to cover the plane ticket). See the world, you learn a new language, help others and then think about what you want to do.
If you're 17, and seriously curious about how modern day AI works, you might as well just sit down and look at a couple of courses on linear algebra + calculus, machine learning, deep learning, and more LLM specific deep learning. Those courses will teach you how to go from writing your first perceptron to a MVP language model. But also so much more.
As a platform engineer being based mainly out of Australia/Hong Kong, opportunities seem to be getting less unless targeting high frequency trading or banking.
It seems like building a startup with the help of some AI tools might be the best bet.
Writing, supervising and training LLMs are now the purview of... even larger LLMs. Optimising CUDA kernels; hand-writing SIMD assembly to speed up data loading; tinkering with your particular brand of DRAM to see if there's anything to gain from optimising for its memory topology and NUMA --- these are now the job of AI.
There is very little reason for humans to get all too engrossed in this type of work now, today, with the hope of being good enough at it to command a high salary in 3-5 years. AI can already do it incredibly well, and they can do it persistently and doggedly 24 hours a day.
With the hindsight of experience, the remnants of my 18-year old energy go “woah, that’s cool!” at plenty of engineering feats… and my decades-older second brain goes “well d’oh, I could’ve just learned a trade to work on that!”
I think the last one was seeing a skilled electronics repairman do surgery on a CT machine controller.
This only changes whether you are naive enough to believe “honest” business means anything in today’s age. If anything, I worry being honest is holding back smart people who try to compete in a rigged game.
There's two ways to read my comment. I believe you read it along the lines of "you need to be rich to start a business".
What I meant was that if you're rich you don't need any trade school or to run a business, you can just sit on your ass living off interest from capital. Only "poor" people need to work for a living.
> you are naive enough to believe “honest” business means anything in today’s age
Might be that these people are from outside the US as well, where things like "honest business" is very much possible today, probably most businesses I interact with AFK on a daily business are "honest businesses".
Why are people so negative about this? It feels like a fun project and at 17 the stakes are not really high. Something one could easily do on summer break in a couple of weeks.
When I was not 17 at the times of GPT2, I decided to not bother with learning how to build LLMs because it’s too expensive for an individual. This escalated quickly.
I read a bit about how LLM works, but as a hobbyist it is pretty frustrating that I won’t be building anything useful without throwing a lot of money at it
i am a small fan of pg, nevertheless i find this to be an exceptionally good take and it is strange to me to see so much piling on to this one in particular here.
learning about llms is not useful so that you can make llms later, you want to learn about it so that you can work on next generation architectures. llms before long i imagine will be left in the dust by ebm / physics oriented models especially that can have an embodied understanding of the world. but a lot of things you learn about them are transferable by doing something like this
I would (and am) going into MLOps. Not just the general infrastructure/systems administration but how to do inference optimization, caching, quantization, memory pinning, vfio passthrough of gpus etc.
I learned HTML when I was 17 in about 1995 and it's certainly taken me on a pretty fun career path. Less technical than LLMs for sure, but 'figure out where the industry is going and move what you're learning to there' is solid advice.
Learn how to make language models from scratch, yes. But learn how to use them, in the context of other machine learning tools, on very small hardware.
When the bubble bursts (and I still tend towards thinking it could burst rather than be deflated in a manageable way), the focus will be on uses of AI that are not like the hyperscalars' products.
People will still be interested in useful AI being added to small things — assistive technologies, home security, garden monitoring, their phones and smartwatches, robotics.
Instead of reductive, reactive make-an-anthropic-competitor advice like this, what about advising 17 year olds to focus on broad, integrated, helpful AI — or on going back through eighty years of history to look at AI projects that failed and reassess them?
If you’re 17, you might learn hands on knowledge, like tacit knowledge in areas such as lathes, precision engineering, metrology, and other very niche fields. You can also try climbing and explore arts like music, painting, and drawing. And, of course, spend some time in nature.
As an aside, for someone interested and who's an absolute beginner, can someone please recommend good resources on how to build LLMs from scratch? Thank you in advance.
2 years before everyone was doing custom training. What happened to all those today when frontier models itself become more powerful than custom trained ones?
I mean first that is already what plenty of 17yo are actually doing, because that is what they do at school or in parascholar activities. There are already countless of such tutorials where you can do that in an afternoon.
The pointless part though is precisely why Amazon and others are hunting for rare books, all the low hanging fruits have been picked already so just training a bigger model will simply mean burning more energy and money. Sure training a small one for the basic principle is a great pedagogical thing, training another one, medium, then maybe a large one, is also good in term of learning the process and architecture, but one should not expect it to be useful out of that context.
Pure players are precisely doing everything they can to corner the market by making their own scale unreachable by others. Smaller players with access to lesser infrastructure are thus betting on different market, e.g. embedded systems.
17yos should definitely build their (L)LMs from scratch and whatever bigger model they can train for free, or for cheap, but they should not expect that to bring them any riches.
Why would 17 year old do something that only brings them money? I do not think Mr Graham here is advocating for the path that makes most money as a result of learning how to train a model. I assume that tinkering and learning about LLMs is what enterprising 17 year olds will do to discover ways they can get a competitive edge or further the SotA with their insights further down the line.
No wonder the state of everything when 17 year olds are getting pressured to be “enterprising” and “get a competitive edge”. How about learning to be empathetic, respecting your fellow humans, caring for the place you live in, enhancing the lives of others? We shouldn’t be teaching 17 year olds to be greedy, selfish, self-aggrandising blowhards like Zuckerberg, Musk, and Graham. They are not good role models for the future of humanity.
I think pg answered the question as “what I’d do as a project” and not “what I’d do as a career.” So the critical comments are kind of missing the point, IMO.
I don’t see why learning how LLMs work is a bad project for a 17 year old.
Optimizing your entire career and the next decade+ of your life on LLMs? Yeah, probably not ideal. It’s almost always a bad idea to make long term decisions based on current trendy things.
And since everyone is using this topic to give their ideal advice to 17 year olds, my advice as a mid-30s guy: seriously consider becoming highly skilled at a specific thing, and don’t be scared off by the idea that it’ll take 5-10-15 years to get there.
When you’re 17-25, the timescale of a decade seems infinite. But it’s really not, and a decade spent “exploring and keeping your options open” sometimes just ends up with you being pretty decent but not amazing at a lot of random things.
Sometimes I wish I had just become a carpenter, chef, electrician, etc. – a specific skill set that leads to mastery over time, rather than the endless exciting-new-thing hamster wheel of working in tech.
Andrej Karpathy has a great Youtube series on how to build LLMs from scratch. Perfect for somebody who just learned a lot of high school math. Would start there and get busy with some handson python coding.
Somewhere in rural America is a 17 year old that doesn't even have working plumbing in their house still.
I'm sure they'll get right on powering up their computer from the hamster wheel, Paul.
My heart goes to all the kids out there that didn't get the fair shake let alone fair access to tech that gets these condescending "learn to code/learn to LLM" bootstrappy talks from rich pricks that don't know what life really can be like for a lot of American kids out there.
If I were 17, I’d learn how to invest and build financial literacy, and plot potential growth of my networth throughout my life, before even thinking about a career. Then smoke a bowl.
I get the sense things have changed a bit since I graduated and there are lot more jobs in AI outside of academia these days, but it's still a very different field from other SWE pursuits, and it's not really accessible to hacker-minded people.
Learning AI isn't like learning HTML in the 90s then expecting to get a job at a tech company building websites. You can't just "learn how to build LLMs" and expect a frontier lab to hire you so I'd argue this is rather bad advise.
Additionally, unlike web development in the 90s you cant really do anything interesting yourself... All of the interesting/useful stuff will require huge amounts of compute and data so there isn't even much point in learning to start your own thing either.
As someone whose built many of NNs from scratch (hand written code, long before the days of LLMs), it's more or less useless knowledge if I wanted to work in a frontier lab or do anything interesting in the field.
I also think anyone thinking about going into a field which is basically a crossover of CompSci and Maths is absolutely insane right now. Even if you think there is a place for CompSci and Maths post LLMs, there's almost no chance anything you learn today will be relevant to the skills required in say 5-10 years.
It seems this is poor advice in that it’s suggesting young people should focus on the current problem as opposed to future problems. Focus on the current problem can result in making some money but it will result in making the incumbents more money, which is not disruptive. Isn’t the goal of radical software startups to maximize disruption?
If the two current bottlenecks, for this LLM madness that could very well be a bubble, are processing capacity and accuracy (a second processing problem) then what comes next? Isn’t that where young people should be looking or are we just giving up on innovation?
The times are different. When I was 17, we had to buy records; a 17yo today can listen to the whole of the available produced music, plus interviews and all other uncommon and related material, for free from the comfort of "here and now".
Possibilities exist now that did not exist before. Those who do not exploit this are fools.
I also did, and had written code for/on other mobile devices before, and did write a large number of even very ambitious software for other mobile devices later.
I do not see any past constructive experience as a waste of time.
Incredible counterexample, but oddly relatable.
I'd probably have achieved techbro 'post-economic' status earlier if I focused on Android dev instead of the shiny (and new at that time) Xamarin for Windows phones.
God I almost invested in Xamarin after Windows Phone got aborted, I did spend a lil time on UWP, but thank god Flutter came out not so long after that. After all these years I learned to stay away from Microsoft tech stack.
I don't think individuals have the resources to build an interesting llm. The l stands for large. You need a dataset too. Llms are only interesting because theyre large
And it's basically a weekend project to put transformers together in a ML library and train it.
The follow up comment,train it to play a game also doesn't make sense? Llms Sony really play games and there are better ml approaches to do that?
What the fuck does this guy know about? I'm sure if we went back through similar statements he's said over the years he's said the same thing about various technologies that are no longer relevant. The guy is a talentless hack who larps as a blogger and his only "redeeming" quality is having lots of money.
Owner of Golf Club Company says I should dedicate my life to golf lmfao.
hackers and painters and kids and ROI and startups and capitalism and the destruction of nature and old guys with money talking like they know better in fascist social networks
The amount of people who missed the point here is absurd. He's advocating for learning about how LLMs work. For the sake of learning. Because no one's going to invent the next thing without at least some understanding of the current thing.
I do not think it is a proper thing to do for 17 y.o., unless they are exceptionally mathematically gifted, as proper understanding of how LLMs are trained requires a good grasp of calculus, understanding modern OS and SDE tools for proper implementation of pipeline etc.
I'd rather simply write another mnist implementation and check if I really like all that AI stuff at first place. Even then, before going into mature-on-the-way-to-dying tech (LLMs) I'd rather focus on fundamentals - good ols linear models, regressions, stat etc.
> I do not think it is a proper thing to do for 17 y.o
If I'd get a buck every time someone said something like this to me when I was in the 13-18 range, I wouldn't have a ton of money, but it's so very annoying when people tell you this.
Regardless if they're "gifted" or not, regardless if you believe in myths like that or not, let children explore what they want to explore, even if you don't understand what it is or why they want to explore that, just let people explore, regardless of age.
It was such a terrible experience being a young kid growing up, with so many adults spending hours trying to convince me to stop sitting in front of the computer so much doing whatever; "why are you even trying to learn that stuff, you have to go to school to understand anything of this" and so much other similar trash.
Sorry, not your fault and I'm borderline trauma-dumping now, but really sad to see this sort of gatekeeping on HN of all places, age is irrelevant to learning ANYTHING, in my humble opinion at least.
Kids, find anything interesting? Jump into it, ignore what adults tell you, and do whatever you feel like, you'll find your place eventually.
Agree with this - started programming through learning scripting in ROBLOX when I was like 12 (this was back like 17 or 18 years ago) and it developed into a life-long passion for software engineering. I am thankful I had people around me (my parents), who were aware enough to realize I wasn't just playing video games and gave me the time I needed on the computer to learn and experiment with programming...
This also meant that by the time I was actually offered to take a programming class in school (junior year of HS), I had already been able to self-teach myself well beyond what that class was covering, thanks to just working on random projects that scratched an itch I had at the time, looking up anything I didn't know or understand, and internalizing those concepts over time.
In short though, I definitely agree, young kids and teens (and also, frankly, adults too!) should be encouraged to explore things that they have a passion for, without being told 'you need to go to school for this' or 'you cant understand this at your age'
I just voiced my opinion. I just think buiding an LLM from the scratch for 17 y.o. is pointless exercise, advising a teenager to do so is borderline irresponsible, and frankly PG is simply virtue signalling here, as LLMs are still trendy, esp. in his circles.
There still will be varyy small number of outliers among youngsters who'd be able to extract tremensous value from such an excercise, but for most that'd be _IMO_ waste of of time, with illusion of understanding w/o actually having any.
Same! I just happened to disagree with your opinion, and frankly, I'd say trying to gatekeep what people learn is closer to "borderline irresponsible" compared to asking people to build/learn/do X.
> youngsters who'd be able to extract tremensous value from such an excercise
But they're youngsters, who are about "extracting value"? Life is about fun, not extraction, not value, not avoiding waste of time but literally enjoy what you do, nothing is more important (IMO).
Then who knows, doing fun stuff sometimes lead to useful stuff, like in my life. But if you only think about "extracting most value for time spent" or similar "optimization strategies", then you'd never discover this part of life.
> Life is about fun, not extraction, not value, not avoiding waste of time but literally enjoy what you do, nothing is more important (IMO).
This is, pardon, demagoguery. There is always "future fun" and "present fun" which a normal person would assign different nonzero weights (https://en.wikipedia.org/wiki/Discounted_utility). Besides, building a LLM _truly_ from the scratch, just using the famous 2017 paper and numpy manuals is not fun at all, esp. for a high schooler.
> Besides, building a LLM _truly_ from the scratch, just using the famous 2017 paper and numpy manuals is not fun at all, esp. for a high schooler.
To you it isn't, is my entire point here. But why extrapolate what you think is fun, to others? Sure, I don't find that fun either (although useful), but who am I to say it isn't fun for others?
We can continue this pointless conversation, in the tone "who you are to tell what is fun to others and whst is not". You'd be impervious to any argument stating that dealing with far beyound someone understanding and requiring countless hours of digging into difficult math is not fun even for those who thinks it should be fun, as they presumably, loves everything STEM.
Why would you tell people that the correct order is to build foundational knowledge before exploring a subject? For some (many?) people, a 'proper' understanding develops _after_ the exploration.
> Why would you tell people that the correct order
Because I can?. JK. Because that was my experience, of someone who is 2.5 older than 17?
> For some (many?) people, a 'proper' understanding develops _after_ the exploration.
I am afraid you have a too confrontational attitude here, but I'll answer anyway: because I do not believe you can simply "explore" such complex topics like building an LLMs. You'd simply be unable to build LLM drom scratch, unless you'd call cargo-cult chaining magic numpy incantations you've taken from Karpathy's tutorials "exploring".
If I were in "exploratory" state of mins, I'd rather go from entirely different side - I'd try playing with LoRA-ing existing small LLMs, such as venerable 2 y.o. Mistral Nemo, to get "feeling" for what training is and how hyperameters influence the process.
Then it is not "building llm from scratch" in my book. Just mindlees following instructions. Could be educational yes, but only trivially useful, if you have no bloody idea what you are doing.
> Fooling around and getting results is what made it fun. Understanding came later.
> No it's not useful, no you don't know what you're doing, and no it is not 'proper'. But you're learning.
It is sold by PG as something special though.
> Maybe just accept that not everyone starts from fundamental theory, and there are lots of people who start learning by fooling around.
Even then LLMs are strange thing to advice to play with, when there are so much more interesting and theoretically accesible for a 17 y.o. so I wonder why would you'd particularly single it out.
> I do not think it is a proper thing to do for 17 y.o., unless they are exceptionally mathematically gifted
I attempted many projects at a young age that I was absolutely not equipped for. The result of the attempts more often than not left me equipped, every time it left me better off. This is terrible advice.
That'd would be a terrible advice if there weren't a plenty of other things "you are not equipped for", but far less daunting both theoretically and practically. Such as, say, convolutional neural networks, or some older ML tech. Or even something totally unrelated to ML.
Transformers are difficult to understand even to people with strong ML background, let alone a teenager.
You are assuming the 17yo in question as an untrained underdeveloped savage. If I were 17 in 2026, I would certainly have exploited all the availabilities from 2010 on - including YouTube, OpenCourseware, the Web simply (Sebastian Raschka etc.) and LLMs.
That 17yo would have already built many uncommon bases, and would build further.
How would you filter out garbage from your training data, for example? If you are trying to use someone elses corpus, would it be "from the scratch" then?
He bases this decision on all of the experience he has amassed, as a 61 year old man in the tech industry. An actual 17 year old, with 17 years of experience, would not think like this, nor should they.
Yes. And they almost certainly have a better understanding of their own situation that him. This is not a dig at Paul Graham, the closer anyone is in age, the better they understand what they have to deal with. I'm roughly in the middle between Paul G and the 17 year old, and even though I'm really quite fascinated with zoomer culture and probably come more in touch with it than most (due to relatives in the age range etc.) I realize I have very little idea what it's like to grow up in the world they grow up in.
>Whoa. I’m 19 and I trained a 100M language model from scratch. Did a v2 now with a new SFT experiment to see if I can get better results on same size.
The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM company, and still then it's a struggle. Those few that do train, they spend most of their budget on compute and have relatively small teams.
Getting experience in this field requires having access to very expensive hardware to begin with. And the skills will be quite hard to convert into any real value for someone, leading to a decent income, unless you have a ton of funding from patient investors, or you have decent contacts in Bay Area networks to get hired at the right place.
With all due respect, paulg is in somewhat of a bubble, this is not congruent with the global situation.
For anyone who wants to dork around there is https://github.com/rasbt/LLMs-from-scratch which is something amazing that I think anyone who wants to engineer things around LLMs should at least blast through and read.
It is viable as a toy project, but there are vanishingly few career opportunities.
Not a lot of demand, but also probably not a lot of supply.
Which is paulg's point really
17 is an interesting age. There are way too many comments here saying things like, 17 year olds should just do whatever seems interesting or bum around the world or focus on getting into university. But historically most kids were expected to be productive adults at 16 or 18. 17 is about the right time to be thinking seriously about what kind of work you'll do, how you'll make a living. University won't help and will just delay this decision.
Paradoxically I coded way more between ages 12-14, I regret my wasted late teens.
I contributed to a browser engine around that age (KHTML, which later became WebKit and Blink), and while I don't work in browsers right now, much of that knowledge, mindset and of course the professional network have done much to shape my life. And a fairly successful career, for that matter.
A lot of people will make better decisions with a few yeas more maturity, and spending a few years developing themselves.
University will help a lot of people, and for some it will help.
There is a lot more to life than making a living.
The early Internet (which the GP mentioned) didn’t even have the web. But that’s nitpicking.
And so I think the idea is more to understand tomorrow ... from first principles.
In the late 80's, as a teenager, I learned x86 assembly and C because that was the only way to squeeze out enough juice from my shitty CGA (and later VGA) card to programm the games/graphics that interested me.
I haven't written assembly in years.
But whatever I did in my career: it helped me and gave me an edge over my peers to have a foundation that is very close to the metal.
I think a lot of people assume that only the big AI labs can do cutting edge research, but there's a strong argument you can do it as part of little tech as well.
Sure you can gradually climb the ladder by demonstrating your skills bit by bit and getting access to more resources. It has very good prospects if you do manage to push through. But it's a hard and risky path, and you will not be able to get any interesting results for the longest time.
For a young middle-class student, it just doesn't make much sense. You can do much more impressive and impactful things with your time without getting into that black hole.
I know how to build an LLM, I know plenty of fellow young engineers that do too. It's really not that complex. But they can't do much with it without capital or access.
Good engineering has never been a bottleneck in this field, it's been all about having access to capital and taking smart but dangerous risks burning it on compute, without much idea of how long you need to keep burning for. There's still no end in sight, some are still managing to convince investors and keep burning, and we are seeing progress, but the business case is still unclear. If you want to get in that game, go ahead, but it's not something I would advice the average young engineer.
I’m 40, and I don’t.I took that abstraction for granted and “left it to the big labs”. However I want to build my own LLM for learning purposes.
On needing big expensive hardware.. necessity is the mother of great innovation. Perhaps 18year olds trying to build their own LLMs in constrained resources environments will result in ground breaking ideas of achieving better intelligence than the one we currently have….
The world needs pragmatic folks who work at a higher abstraction and make LLMs useful, AND also folks who think why not “this other way”? And build newer ways to do fundamental things.
Given the usefulness of current LLMs, I would certainly encourage anybody to try and build their own LLMs, and see what they come up with…
Heck if they build a rack full of old laptops and run something with it that could be done “better” with modern servers, I’d still appreciate the learning running things on those little machines bring.
Maybe with a decent consumer GPU like a 4090, you could look at doing experiments like distilling and fine tuning a small image model for edge deployment for specific tasks.
Probably even there many use cases may require renting compute for $10/hour and investing a few hundred.
A LLM from scratch? Forget it. You can do theoretical experiments, but not build anything remotely useful with that kind of budget. If you're such a genius as to come up with revolutionary methods, maybe an university or actual AI lab would be the place to be.
10000%.
You can however learn everything you need to know to get on the career ladder as a software engineer on a regular home PC.
Do you take the first step or rule it out because you don’t yet see the complete picture.
As a teenager I never hesitated to try things out. As a young adult I wanted the whole picture. Now I’m back to playing / trying things out. I kinda wish I’d not given it up. PG being a bit older and reminiscing - I bet he’s in that bucket too, whereas someone trying to establish themselves professionally probably (aka me early 20s) wants to see the path.
The “large” qualifier dates back to pre-transformer language models, where even training a multi-million model was hard due to how poorly it scaled. GPT-2 was a large language model, despite being only 124 millions parameters.
Due to how much high quality data is readily available, anyone can now train a sub-billion (L?)LM on commodity hardware.
And I'm personally convinced that pretty much any enterprise use-case of an LLM (except coding) is better served by a fine-tuned small (<2B) model that is trained specifically on the task, rather than a generalist frontier model, so learning the engineering around fine-tuning is a key skill that companies will realize they need sooner than later.
I saw my first language model in action in 2014, I was writing blog posts about them back in 2016. In recent years, like many of us, I spent some time learning ML frameworks to see if it'd be a fun career pivot.
But:
1. It doesn't seem especially creative. All the frontier labs have nearly identical model personalities, capabilities and even app designs. We're not seeing them differentiate from each other, implying that the design space might not be that large. In which case the opinions and unique approaches of specific engineers aren't that important, they are interchangeable at the right level of skill, and what to do next is usually obvious to everyone.
2. It doesn't seem like a big job market. A lot of ML jobs were wiped out in recent years by the rise of LLMs. Lots of NLP specialists etc were suddenly replaceable with a cheap API. The jobs that remain have compacted into a small number of companies. It's a small community which greatly increases career risk, especially as so many are unprofitable and/or have strong ideological requirements.
3. It's unclear how much demand for better models there actually is. Do we actually need smarter models? In robotics clearly yes and robotics is interesting and high potential, but for pure LLMs/image models, most users are already incapable of setting tasks that stress the best models and are happy with the cheaper smaller ones.
Using the models on the other hand is a very large design space, and has a lot of scope for creativity. I see use cases for AI everywhere, but most companies seem to stop at putting a chatbot on their website or asking Copilot to rewrite an email before they send it. A lot of companies have hollowed out their IT departments over the past twenty years. It feels like a new golden age of consulting work could be upon us.
I really don't think I'd tell a 17 year old to learn how to train LLMs. Learn how they work and how to use them, sure, absolutely.
If you look at model training jobs a lot of the work at this point is creating RL gyms (normal programming work), but most people still think the work is all neural architecture research. Doing the former is fine but won't teach you much about how to build LLMs, whatever that means now. Doing the latter is a very hard market to get into: not many jobs and requirements are often like, "you must have published at one of the following conferences". Prior experience is assumed. Most of them seem to treat Google as ML university and source of new recruits. It's understandable given the cost of training runs.
I'm not sure why it's like this. If you look at the real world, you have stuff like ggml, which is about as hardcore as it gets in the LLM space, and it was made buy just a guy. Same for this like ComfyUI
If you get enough academics in a place, they tend to close rank, and not let anyone in without the same credentials. Data science used to be like this, they were constantly on about how you need a Math Phd to even apply, yet when I met these guys IRL, most of them were just running Python math libraries.
These previous examples show that if you understand at least a part of the problem space, you can 100% contribute without academic credentials.
The question is more one of opportunity cost. At 17 you need to start finding your way in the world. It's best to learn skills lots of people need.
Now with LLMs, people write native apps in Rust, and I'd like to think some of them found that there isn't such a huge jump in difficulty they assumed there would be.
I just finished fine tuning Gemma e2b for local code completion on my local machine.
This comment just reinforces what the post actually means. We need people that are LLM natives, computing solves itself with time and with scale adjustments
The only jobs that he found he was highly over qualified or paid very little.
In any case, it doesn't look like there's this crazy rush to hire all ML talent, even the one that understand the math and technology deeply.
Maybe people simply don't want math PhDs but something else? Since 1-2 years ago I started doing consulting/freelancing in the ML space, but more on the infrastructure, deployments and similar stuff, as a general purpose developer, and I have a waiting list of clients interested in more work, some of them even trying to recruit me to work for them full-time as well. I'm based in continental Europe, fwiw.
Is it possible to see some of your old works? Personal research?
Also bunch of past workplaces who've adopted AI in various ways who reach out once they find out what my current focus lies, but that's harder for others to replicate unless you've already had a career as a developer.
I don't think the "incredibly small minority of companies in the world do any real training or optimisation" part is necessarily as true, as some parts of the work I do get is about helping them optimize training and infrastructure around training. Mind you, none of this is for building LLMs from scratch, it's 99% fine-tuning existing checkpoints.
I'd also agree with "paulg is in somewhat of a bubble" regardless of this, which is worth remembering whenever you read his content. Same goes for any person living in SF, and dare I say the US. But also, YMMV, I live and work in Europe, probably why I have this perspective.
Worst of both worlds - no casual gossip feed in the Bay Area, no big dog meetings in the UK. (Which mostly has no idea he exists.)
As for the question - what are the odds LLMs will be anywhere near the top of the tech tree five years from now?
The trend seems pretty clear to me - local/offshore models are snapping at the heels of the big names in the US, and the current investment arc is insane.
I wouldn't bet on Anthropic or OpenAI being leaders five years from now. Longer term, I especially wouldn't bet on the US build-yourself-a-monopoly corporate model surviving AI at all.
Citation needed. I'm guessing you're not counting children.
There are plenty of areas were we need people to do this for insurances, banks etc.
AI/ML exists on many levels.
I can see the point. It's unlikely that a 2.4T LLM will be integrated into, say, a pesticide drone. You'll still need some kind of LLMs to achieve maneuvers that "normal" programmings can't achieve.
But what if everything basically turn into that? Essentially, instead of build me a web app to solve X and do Y, build me an LLM to serve X and do Y. (unless the current LLMs are able to do it end-to-end but then they can hardly write coherent software/personal opinion).
1. Both training and optimisation will get significantly cheaper and easier quickly.
2. Politics will probably get even more insane before a potential reprieve on the 20th of Jan 2029.
3. The big AI firms will become part of the surveillance capitalism network, if they're not already.
So I think for self-protection a lot of companies will be looking near to medium term AI independence.
For the time being, unless you truly have millions, the outcome from training will be very net negative, while focusing on building on top of existing AI will yield amazing things if you apply the same talent and effort.
When it does get cheaper, then it will be easier to acquire the skills and experience too, and the struggle you went through by trying to do it now will be somewhat wasted.
Besides, I am well versed in this field, and it is not rocket science. There are plenty of software engineering domains that are a lot more challenging, like high-end graphics, large-scale data engineering or kernel programming. People will learn to train LLMs when people want them to.
In reality, enterprises are happy to offload even risky tasks to others as long as they get some contractual guarantees about their data. Would they like more choice in who to buy from? Yes, but not enough to in-house such a specific discipline.
Fine-tuning a model or LoRA based on the companies data set is more feasible but you're likely going to need several runs as you test/try out different base models, parameters, etc. This is why there are a lot of fine-tuned models on huggingface based on base or instruction-trained models from the larger AI companies that have released open weight models (Microsoft, Google, IBM, Mistral, DeepSeek, Qwen, etc.).
Training is limited on memory first (storing training data and weights) and computation second. Realistically you need to own or rent 2-8 H100/B100 devices or Google's TPUs.
The majority of workflows for a company providing AI capabilities are likely best solved by tailoring a system prompt for the chosen model, evaluating the prompt and model with tools like promptfoo, and then running it on a compute cloud provider (including AWS Bedrock). If the company is big/financially well off enough they could look at buying the hardware needed to run it on their own servers.
For other uses like agentic software development you'd need to spin up a suitable model on a compute cloud provider (or local hardware if the model is small enough) and then tell your IDE/editor to use that model. You would need some way of benchmarking and evaluating the models to see if they are capable of doing the tasks you need. -- There have been some tests done by people on YouTube that suggests that Qwen 3.8 27B is a decent model, but your needs may vary.
You’re missing the point. Understanding how Unity works fundamentally makes you a better Unity dev.
Writing your own game engine makes you realize that the Unity engine is not really that well written....
In reality the problem is that it gets blasted out of the water by a much worse architecture trained on 10000x the infrastructure. And while I'm sure the freshly brought in ML student came up with a 10%, even 30% better architecture, it just doesn't matter. (and never mind that even OpenAI hasn't really solved a voice model yet. Try it. It can probably match 2026-quality call centers, but it's no substitute for an actually empowered human)
... and yet, if you look at what hyperscalers are getting paid for ... comfortably more than half the income is training. Which makes no sense on so many levels.
e.g. https://valueaddvc.com/blog/inference-chips-vs-training-chip... (I get it, not great first source, but st
For some very niche cases I think this is probably the case but for the vast majority, the company's data isn't as useful as they think it is or anywhere near the size needed.
I don't know where you are located, but in EU, in China, and yes even in Silicon Valley, the vast majority of companies do not do any real AI engineering. There's nothing wrong with it, it's just not a smart path for most purposes. You can do amazing things without training, and if you try to train, you cannot get anything amazing unless you burn millions.
Very few people can afford to play the long game and cross that dessert. And, sure, you will not get far without good engineering, but good engineering is definitely not sufficient and is not the primary bottleneck.
It would be a good idea for young people to deeply know how these programs work. Not so that they can spend their career building them, but so that they can approach the next class of problems we'll all start trying to solve, with intuition all the way down to the weights and underlying mathematics. And also, to develop a healthy intuition of when "Just LLM it" will not be the right choice.
"Build an OS" wasn't a common university project because we were all expected to go out and work on Windows, but because understanding the bare-metal firmware for a computer helps you deeply understand how to intuit building for a whole class of problems.
I didn't do it because it was useful to me in a practical sense. It's because LLMs are fascinating and I want to know how they work. From that perspective it's been a great experience. I have afirm grasp of the basics. This makes it much easier to understand frontier concepts like compressed latent attention. I can follow the field and understand it.
Not sure I would have got as much out of it at seventeen. I have a lot of background and experience which made it much easier to learn. I wasn't struggling with the linear algebra or with python. I already knew pytorch and neural networks. That helped a lot and I covered these tutorials fast and could skip over large sections. A few evenings and the odd weekend day over a couple of months was enough for me.
For seventeen year olds the tutorials are good enough to make it possible to learn this but it would have taken a lot longer to understand. On the other hand I would have learned a lot more. I think I would have learned a lot of valuable stuff.
However I also think 17 year old me was studying for his A levels and probably this was right choice in terms of maximising future opportunities. I'm not sure I think learning about LLMs instead is sensible. Indeed it might be bad advice. But I can absolutely agree with the sentiment.I think 17 year old me would have wanted to do this too.
Moreover I am not sure it is even good advice? Would you advise a 17 y.o. to learn how transistors work or how to code (i.e. is LLM training the right level in the stack)? LLM training, a discipline where relevant work is already out of reach for 99.999% of budgets really as essential as this post implies?
Personally I don't see the problem, as long as you're aware there is survivorship bias involved here.
What's the alternative really, seek advice from unsuccessful people? That seems worse :)
Personally I do both, read about what worked for people, also read about what didn't work for people, then ignore both and do whatever the fuck I want.
What bucket should I put this advice in?
Agreed, my previous stated "ignore both and do whatever the fuck I want" approach has worked out very well for me in life, people should probably focus on identifying better what their gut tells them, rather than what randoms on the internet thinks and writes.
Intuitively, I would guess that they have a better grasp of what made them fail than successful people have of what made them succeed.
For me, I know that one major reason for putting an upper limit on my success is my inability to form effective professional relationships. I know I should go to events and talk to people and use these relationships to my advantage, but that's just not something I've ever been good at, and I find it so incredibly unpleasant that I also just don't want to do it.
Nonetheless, there are many successful people I would gladly listen to for advice, though they are often successful in a different meaning than what venture capitalists would use (e.g. parents with great kids, managing to keep a healthy work-life balance, happiness, and maybe even having time to spend on some cool hobby project -- you are heros!)
Of course, there are degrees and exceptions on every side, but the coping mechanism is very strong among people.
Seek advice from the averagely successful people, since that is statistically what you're most likely to be.
Do both. Get advice from successful and unsuccessful people and take the diff
That’s what you get from listening to “successful people”. You get to learn about all the things they tried that failed, then the things that did work on that 24th try, which was successful.
The “survivorship bias” people always seem to assume that the “survivor” lucked into his fortune on his first try ever, so he can’t have learned anything, so we don’t have to listen to him. But that’s seldom the case.
I’ve written about this before:
https://expatsoftware.com/Articles/survivorship-bias.html
Not sure why would you think so.
Inverse reasoning is very powerful, and unsuccessful people can give you plenty of "don't do this mistake", which the survivors would not even think about.
But how can I know for sure that that particular mistake is actually why they were unsuccessful? Has exactly the same issue as listening only to successful people as they hardly know what actually made them successful most of the time, but they still compose large blog posts with their reasoning for why.
Again, I still think my approach of reading both but then regardless go my own way is the preferable approach, at least for me, ymmv.
https://www.youtube.com/watch?v=3LopI4YeC4I
An advantage that is not "advisable", like being born in january, in a rich country, in an above average family, or just having luck, might have more influence on the outcome than any conscious action. It is almost sure that one-in-a-million level people only edge over the other 999,999 they competed with is just "have more luck".
I think it is. He isn't saying to learn how to train a LLM so that you can go on to train LLMs. He's saying to learn it so that you gain a deep understanding of how LLMs work. Ordinary startups can still benefit from things like training or fine tuning highly specialised smaller models, knowing how to select and configure an appropriate model for the task at hand, knowing what software to use and why, understanding what's going on behind the scenes instead of treating everything like a black box, having a higher level of intuition about LLMs generally, etc.
Most computer science courses do in fact teach things which are lower level than coding, such as how transistors work.
how many of us out here are doing work directly in what we got a degree in? I majored in economics and now I'm a CTO.
I would absolutely advise a 17 yo to learn how to code, understand how transitors work and how to code an llm. even if he never works on llms, you basically end up with a kid with applied knowlege of statistics, math, physics hardware, logic and a whole lot of practice in critical thinking.
Hell yeah. Transistors are pretty awesome.
In 2000 (his era), it would have been really smart to study the source of Linux or Apache. Would have paid dividends over decades. Cuz that knowledge was so rare. The number of people hacking on LLMs now dwarfs the number of people hacking on web servers 30 years ago, by several orders of magnitude.
And if you turn back the clock even more, I mean just even having access to a computer, let alone owning one, would have put you at a massive advantage.
I don't know what to call it. The pioneers should be respected obviously, but at the same time you need to understand that for them, the game wasn't nearly as played out as it is now.
I just don't think you can afford to be dicking around with LLMs like you could afford to dick around with random Linux distros 20 years ago. Too many people willing to do it for free these days.
You don't wanna end up being the 2030 equivalent of a certain SNES emulator developer, or maintainer of a package manager for jailbroken iPhones, I mean the list goes on and on. Being a hacker doesn't automatically give you a path to being rich, or even making a decent living. It hasn't been that way for a while.
Learning to hack something together in high school using the latest technology (vacuum tubes, radios, microprocessors, web/javascript) has been a common theme in the tech world for generations. With LLMs and online tutorials, this isn't even a difficult suggestion. Do people think learning new tech is somehow wasted effort?
I can't recall or point out exactly when, but there is a stark before/after moment where the opinions of anything pg went from "Interesting and maybe true in some ways" to what we see today, lots of knee-jerk reactions and hardly any comments about the actual content.
Hazarding a guess, I think the moment Altman became the CEO and later during COVID, the sentiment seemed to have been shifting towards what we see today. But this is all based on hazy memory, rather than looking at the data. I'm sure there is a blog post waiting to be written about analyzing the sentiment of comments to PGs articles on HN, and you'll see a shift somewhere.
Hard disagree. This submission is still being highly upvoted, while another recent post[1] on the harms caused by Graham’s fellows[2], with a fairly tame comment section, has been flagged. That is a constant on HN. It’s not a fluke, it’s as predictable as the sunrise and getting more pronounced.
I’m sure we’re both biased in our perceptions. Mine is that HN in general (certainly more than any other website) used to worship[3] everything he wrote, together with others like Musk, until things started to really go to shit and many eyes have been opened to the effects of the unfettered greed of rich tech guys out of touch with reality.[4]
[1]: https://news.ycombinator.com/item?id=49411762
[2]: A better English word is escaping me.
[3]: That word I choose hyperbolically but deliberately. It definitely was not “interesting and maybe true in some ways”, it was much more hardcore than that.
[4]: That is not “knee-jerk” but a slow realisation still ongoing.
I guess it depends on what submission you look at, previous comment of mine solely based on memory. Now I went to https://news.ycombinator.com/from?site=twitter.com/paulg, clicked "More" a bunch of times, and seems my memory was more or less correct, none of the submissions I clicked on are "pg worship" (hyperbolic or not). Just one example: https://news.ycombinator.com/item?id=19418701
Maybe you need to enable "Show Dead" or something? Pgs articles on HN definitely never was free of any critique in the HN comments, just like any article. Although I do agree with you that it used to be different than it is today, and same with Musk too, and Altman, and probably more individuals, where they were lauded before but now pretty much just mentioning them poisons the conversation.
That post has barely any points and comments. It’s not a good indicator of general sentiment, it’s just an indicator of people who were on HN at that time.
> Maybe you need to enable "Show Dead" or something?
I have it enabled.
> Pgs articles on HN definitely never was free of any critique in the HN comments, just like any article.
Of course. I very explicitly wrote “in general (certainly more than any other website)”. That does not mean “always”, or “never”. HN is not a hive mind, there’s never going to be 100% agreement. The general trend is what’s being discussed, and we both agree that in general the sentiment on Graham used to be higher. We’re just disagreeing (we may be able to find ourselves agreeing through tough thorough thought, though[1]) on where exactly it is now and how to interpret it.
[1]: Sorry, can’t believe there was a real organic opportunity to use that sentence, had to take it.
Hard agree! Though the discussion was short, I thank you for it. Good start of the week, I wish you a good one.
The above is, after all, the whole genesis of the word 'hacker'. We should celebrate that.
I tried to modify the embedding output of bert to make it generate box embeddings instead of point ones. At the time I had access to university provided A100 gpus but even with all that a training run took half a day. Models these days I don't think I can train it in any reasonable time with that much compute.
No. But funnily enough that is a promise by some of the AI cretins and their boosters. Oh yeah best case scenario you learn how to build LLMs for us. We’ll employ you. And then ultimately that just becomes training data for the LLMs to do it themselves.
But why are people cynical? they ask.
Don't many of the commercial ones prevent you from using them to build LLMs?
I would say the reason for the negativity is not because it's a bad idea for a project, or that doing projects in general is a bad idea (it's not!), it's because it's a very specific thing that is not for everyone. The best thing about computing is the low barriers to entry. You can basically work on anything that takes your fancy. So those who are interested in ML will be drawn to learn about LLMs. They don't need anyone to tell them to do it. Telling everyone to do it reminds me of the "just learn to code" stuff of a decade ago. No, please don't, please find something you enjoy.
Being a super rich and an unhappy workaholic, or a super-impressive engineer who wakes up one day at 45 and realizes they regret wasting half their life (I ran into way too many of these) is a much worse fate than "not being rich from your startup" and working a relatively regular job while feeling fulfilled and happy by more than just work.
Especially in the US, which is uniquely bad at this and encourages people to work themselves to death, mental health and work life balance are much more valuable things for 17 year olds to focus on than finding good startup ideas.
In case you think i'm being a bit dramatic, let's look at the state of 17 year old mental health in the heart of Silicon Valley:
"The City of Palo Alto and the Palo Alto Unified School District approved a funded contract to place 24/7 human security guards and monitors at all four local Caltrain grade crossings, including the Churchill Avenue crossing directly adjacent to Palo Alto High School."
(in case it's not obvious, it's because of suicides by high school students)
The 17 year olds do not need advice on better startups, and this situation will never get better if we focus our advice on how to be better at work instead of how to be better at life. This will require redirecting the conversations.
I’d probably say something like: do something you enjoy and seems like it might be useful, but accept that the pace of change may mean that whatever you study ends up being irrelevant.
Whatever solution there ends up being to this, it’s not going to be one that an individual 17 year old can implement. We’re past the point where individual good and bad choices matter that much to economic outcomes.
Telling other people's children what to do is easy and basically doesn't have any downside to being wrong. With your own children things are a bit different.
So: what are people here with school age children telling their own kids about the future? Assuming their kids ask, what kind of careers would they encourage them to pursue, assuming they have the skills and interest?
When I was last in the Bay Area, maybe about a decade ago the bookshops were full of titles like "Python for Preschoolers" (I exaggerate, but only slightly). Clearly at the time a lot of people working in tech thought that cultivating an interest in programming was going to be the path to being a successful (by some metric) adult. Is that still the case?
http://languagemodelbuilder.com teaches you (in a few hours to days) how to build an LLM from scratch. It's entirely free, without accounts, and without data collection.
Thanks a ton for building this.
On an Amiga, I took various public domain text documents from cover disks and counted the probability of the next word given the previous word. Then spat out random sequences of words from it and printed them out. It was called "Splurge". Basically a very very simple single layer statistical language model.
Some of the sentences were randomly not bad sentences, which seemed amazing at the time!
That kind of thing (and Core Wars and Tierra etc) did lead me to getting a job at an artificial life startup at the end of the decade. But that was in turn about 10/15 years too early (no GPUs).
There's some lesson from this about timing, but honestly I've gained the most as a person when I did something that was fun, ethical and gained an audience. A tricky combination.
But of course, 10 years ago this wasn't obvious.
Or more reasonably, the same old thing : use Linux, hack a little, why not learn programming basics. But learn to own your technology, fight against centralization of technology. The same old RMS story.
IDK where the tech industry is going, if there will be jobs anymore or not, but what I'm sure (and what have been the case for the last 10-15 years anyway) is that for most tech jobs, having good technical knowledge beyond the basics is pretty useless and will probably not be recognized.
If you can, stay a computer geek if that's your thing, but don't make it your career choice, the Eldorado is behind us.
I’m no paulg, but if you’re reading this - and you’re 17 - just focus on getting into a university and having a good time that you won’t regret later. Play games/sports, make relationships, fall in love, explore.
Then vote for someone who will make the debts go away?
I mean, have you seen the options for people graduating right now? How people are behaving?
Or forget the data, look at how the story of the new future technology is being told. The people making it recognize that it has the potential to put swathes of white collar workers out of jobs, and they are openly talking/warning/PR-ing about it.
People in tech and SV, the places which have a underlying culture of near delusional optimism, are talking about trying to avoid being part of "the permanent underclass".
Gambling is up, and prediction markets are being treated as financial investments. Wall street bets is a thing, and outright speculative investments are the hope people have to get ahead.
This is happening in the USA, forget the weaker or smaller economies.
When people see the future as one massive zero sum game, with no way to win by building, then they are going to change how they plan their future.
Everybody goes to college nowadays and the average white collar has lots of debt and relatively minor financial benefits over a skilled trade worker.
edit: woah, so many people insulted by that. In my bubble and friends, me and another friend are the only people that make very good money compared to non-graduates. Plenty of others opened their shops, went into trades, one learned to tattoo fake eyelashes, one became a (successful) farmer and most make significantly more than the average law/chemist/mathematician/physics/architecture/languages graduates. Sure, the lowest salaries are to be found among the non-graduates too, but I don't see any evidence that graduates make that much more, and that graduating is worth it.
Some answers talking about how "formative college is", but my 25 years old friend with her own shop knows more about real life, business and economy than ivy league MBAs.
Its not good advice for everybody, but it is good advice for a lot of people. What if you want to be a doctor? What if you want to work in R & D? Not everyone enjoys working in a shop or a farm. Also, how old is your friend group? If they are mid twenties you are ignoring the greater scope for advancement in a lot of white collar careers.
> my 25 years old friend with her own shop knows more about real life, business and economy than ivy league MBAs.
Within the narrow limits relevant to her business. How much does she know about macro-economics or financial economics, or scaling up a business? I also suspect you are comparing her to people who went straight on from bachelors to MBA (which is a bad path - study business after having some experience IMO) and lack experience. How will she compare in 10 years time when those people also have real world experience?
Many people are surprised to learn how affordable elite colleges are if you genuinely need financial aid. I had no idea -- was pleasantly surprised when my alma mater took over 80% off of my tuition.
Where I live it's basically free and I still sometimes regret not going into the trades. But I suspect this feeling might mostly be a "grass is greener" thing.
If you’re talking about a place to mature, around others who are at a similar phase of life, also yes.
It is where most people meet their cofounders, for example, even if they don’t found anything until much later.
Plenty of blue collar workers make more than white collar ones, and have a huge debt free head start in life.
A plumber or electrician will make significantly more than the average bank employee or translator or nurse or teacher.
And they will also have an easier time starting a business as many trade workers are self employed and make much more than hired ones.
Financially, intellectually, socially, it’s a good idea - college is a formative period of life in American culture. That is more and more true the better the college gets. At the upper tier (Ivy League, etc.) you don’t really have to pay anything if your family isn’t already wealthy, and the connections and degree you’ll make more than pay for themselves.
Putting this in contrast with programming, I learned coding when I was 8, and it was incredibly stimulating to learn because you can quickly iterate and there were thousands of books and YouTube tutorials that dumb everything down and teach you fundamentals. All you needed was a $300 computer, and you can learn nearly anything you want, without being gatekept from this or that because you don't have enough vRAM / an sm_100 GPU.
As a platform engineer being based mainly out of Australia/Hong Kong, opportunities seem to be getting less unless targeting high frequency trading or banking.
It seems like building a startup with the help of some AI tools might be the best bet.
Better to start working with harnesses, evals, statistical analysis, etc. - where you don't need the huge hardware for pre-training etc.
There is very little reason for humans to get all too engrossed in this type of work now, today, with the hope of being good enough at it to command a high salary in 3-5 years. AI can already do it incredibly well, and they can do it persistently and doggedly 24 hours a day.
(Edit: And learn how honest business works)
I think the last one was seeing a skilled electronics repairman do surgery on a CT machine controller.
What I meant was that if you're rich you don't need any trade school or to run a business, you can just sit on your ass living off interest from capital. Only "poor" people need to work for a living.
Might be that these people are from outside the US as well, where things like "honest business" is very much possible today, probably most businesses I interact with AFK on a daily business are "honest businesses".
https://github.com/raiyanyahya/how-to-train-your-gpt
learning about llms is not useful so that you can make llms later, you want to learn about it so that you can work on next generation architectures. llms before long i imagine will be left in the dust by ebm / physics oriented models especially that can have an embodied understanding of the world. but a lot of things you learn about them are transferable by doing something like this
Learn how to make language models from scratch, yes. But learn how to use them, in the context of other machine learning tools, on very small hardware.
When the bubble bursts (and I still tend towards thinking it could burst rather than be deflated in a manageable way), the focus will be on uses of AI that are not like the hyperscalars' products.
People will still be interested in useful AI being added to small things — assistive technologies, home security, garden monitoring, their phones and smartwatches, robotics.
Instead of reductive, reactive make-an-anthropic-competitor advice like this, what about advising 17 year olds to focus on broad, integrated, helpful AI — or on going back through eighty years of history to look at AI projects that failed and reassess them?
2. LLM from 0 to Hero, and nanoGPT by Andrej Karpathy
The core technoology is pretty basic, developing a rudimentary understanding for why the individual parts work as well as they do is tricky.
I first studied them in 2011, and I was like what? Just a bunch of partial derivatives?
I keep looking at AI to check if now it's something else but it keeps being gradient descent.
Okay, it's great that you can perform miracles using gradient descent but that doesn't make it captivating in any way.
I mean first that is already what plenty of 17yo are actually doing, because that is what they do at school or in parascholar activities. There are already countless of such tutorials where you can do that in an afternoon.
The pointless part though is precisely why Amazon and others are hunting for rare books, all the low hanging fruits have been picked already so just training a bigger model will simply mean burning more energy and money. Sure training a small one for the basic principle is a great pedagogical thing, training another one, medium, then maybe a large one, is also good in term of learning the process and architecture, but one should not expect it to be useful out of that context.
Pure players are precisely doing everything they can to corner the market by making their own scale unreachable by others. Smaller players with access to lesser infrastructure are thus betting on different market, e.g. embedded systems.
17yos should definitely build their (L)LMs from scratch and whatever bigger model they can train for free, or for cheap, but they should not expect that to bring them any riches.
You should want to train a LLM from scratch as an intellectual curiosity itch that needs to be scratched.
The idea of learning one hot skill that has a pot of gold waiting at the end of it was a brief moment in time that came and went.
When I was 17, we would have said obviously support vector machines are the future. Neural networks overfit and don't work.
I don’t see why learning how LLMs work is a bad project for a 17 year old.
Optimizing your entire career and the next decade+ of your life on LLMs? Yeah, probably not ideal. It’s almost always a bad idea to make long term decisions based on current trendy things.
And since everyone is using this topic to give their ideal advice to 17 year olds, my advice as a mid-30s guy: seriously consider becoming highly skilled at a specific thing, and don’t be scared off by the idea that it’ll take 5-10-15 years to get there.
When you’re 17-25, the timescale of a decade seems infinite. But it’s really not, and a decade spent “exploring and keeping your options open” sometimes just ends up with you being pretty decent but not amazing at a lot of random things.
Sometimes I wish I had just become a carpenter, chef, electrician, etc. – a specific skill set that leads to mastery over time, rather than the endless exciting-new-thing hamster wheel of working in tech.
I have my opinion on this but I'd like to hear the HN opinion, I will just say one thing:
If you are starting with little knowledge, like a 17 year old would, letting an LLM explain it to you is a terrible idea.
I'm sure they'll get right on powering up their computer from the hamster wheel, Paul.
My heart goes to all the kids out there that didn't get the fair shake let alone fair access to tech that gets these condescending "learn to code/learn to LLM" bootstrappy talks from rich pricks that don't know what life really can be like for a lot of American kids out there.
Learning AI isn't like learning HTML in the 90s then expecting to get a job at a tech company building websites. You can't just "learn how to build LLMs" and expect a frontier lab to hire you so I'd argue this is rather bad advise.
Additionally, unlike web development in the 90s you cant really do anything interesting yourself... All of the interesting/useful stuff will require huge amounts of compute and data so there isn't even much point in learning to start your own thing either.
As someone whose built many of NNs from scratch (hand written code, long before the days of LLMs), it's more or less useless knowledge if I wanted to work in a frontier lab or do anything interesting in the field.
I also think anyone thinking about going into a field which is basically a crossover of CompSci and Maths is absolutely insane right now. Even if you think there is a place for CompSci and Maths post LLMs, there's almost no chance anything you learn today will be relevant to the skills required in say 5-10 years.
If the two current bottlenecks, for this LLM madness that could very well be a bubble, are processing capacity and accuracy (a second processing problem) then what comes next? Isn’t that where young people should be looking or are we just giving up on innovation?
Wait no it’s not, that was always happening.
What’s crazy is that people still believe in it.
Possibilities exist now that did not exist before. Those who do not exploit this are fools.
I do not see any past constructive experience as a waste of time.
And it's basically a weekend project to put transformers together in a ML library and train it.
The follow up comment,train it to play a game also doesn't make sense? Llms Sony really play games and there are better ml approaches to do that?
Even worse when they ask themselves.
Owner of Golf Club Company says I should dedicate my life to golf lmfao.
Sometimes if you want to be heard you must go to the public square, whatever the flags there.
The rest of the post is unintelligible: add some verbs at least.
I'd rather simply write another mnist implementation and check if I really like all that AI stuff at first place. Even then, before going into mature-on-the-way-to-dying tech (LLMs) I'd rather focus on fundamentals - good ols linear models, regressions, stat etc.
If I'd get a buck every time someone said something like this to me when I was in the 13-18 range, I wouldn't have a ton of money, but it's so very annoying when people tell you this.
Regardless if they're "gifted" or not, regardless if you believe in myths like that or not, let children explore what they want to explore, even if you don't understand what it is or why they want to explore that, just let people explore, regardless of age.
It was such a terrible experience being a young kid growing up, with so many adults spending hours trying to convince me to stop sitting in front of the computer so much doing whatever; "why are you even trying to learn that stuff, you have to go to school to understand anything of this" and so much other similar trash.
Sorry, not your fault and I'm borderline trauma-dumping now, but really sad to see this sort of gatekeeping on HN of all places, age is irrelevant to learning ANYTHING, in my humble opinion at least.
Kids, find anything interesting? Jump into it, ignore what adults tell you, and do whatever you feel like, you'll find your place eventually.
This also meant that by the time I was actually offered to take a programming class in school (junior year of HS), I had already been able to self-teach myself well beyond what that class was covering, thanks to just working on random projects that scratched an itch I had at the time, looking up anything I didn't know or understand, and internalizing those concepts over time.
In short though, I definitely agree, young kids and teens (and also, frankly, adults too!) should be encouraged to explore things that they have a passion for, without being told 'you need to go to school for this' or 'you cant understand this at your age'
There still will be varyy small number of outliers among youngsters who'd be able to extract tremensous value from such an excercise, but for most that'd be _IMO_ waste of of time, with illusion of understanding w/o actually having any.
Same! I just happened to disagree with your opinion, and frankly, I'd say trying to gatekeep what people learn is closer to "borderline irresponsible" compared to asking people to build/learn/do X.
> youngsters who'd be able to extract tremensous value from such an excercise
But they're youngsters, who are about "extracting value"? Life is about fun, not extraction, not value, not avoiding waste of time but literally enjoy what you do, nothing is more important (IMO).
Then who knows, doing fun stuff sometimes lead to useful stuff, like in my life. But if you only think about "extracting most value for time spent" or similar "optimization strategies", then you'd never discover this part of life.
This is, pardon, demagoguery. There is always "future fun" and "present fun" which a normal person would assign different nonzero weights (https://en.wikipedia.org/wiki/Discounted_utility). Besides, building a LLM _truly_ from the scratch, just using the famous 2017 paper and numpy manuals is not fun at all, esp. for a high schooler.
To you it isn't, is my entire point here. But why extrapolate what you think is fun, to others? Sure, I don't find that fun either (although useful), but who am I to say it isn't fun for others?
Yeah, if you see it as pointless, then I prefer not to. Have a enjoyable day!
Because I can?. JK. Because that was my experience, of someone who is 2.5 older than 17?
> For some (many?) people, a 'proper' understanding develops _after_ the exploration.
I am afraid you have a too confrontational attitude here, but I'll answer anyway: because I do not believe you can simply "explore" such complex topics like building an LLMs. You'd simply be unable to build LLM drom scratch, unless you'd call cargo-cult chaining magic numpy incantations you've taken from Karpathy's tutorials "exploring".
If I were in "exploratory" state of mins, I'd rather go from entirely different side - I'd try playing with LoRA-ing existing small LLMs, such as venerable 2 y.o. Mistral Nemo, to get "feeling" for what training is and how hyperameters influence the process.
That's what I would call exploring.
I too started exploring programming as a teen by cargo culting. Fooling around and getting results is what made it fun. Understanding came later.
Then it is not "building llm from scratch" in my book. Just mindlees following instructions. Could be educational yes, but only trivially useful, if you have no bloody idea what you are doing.
> Fooling around and getting results is what made it fun. Understanding came later.
It is not how LLMs are "built from scratch".
But you're learning.
Maybe just accept that not everyone starts from fundamental theory, and there are lots of people who start learning by fooling around.
It is sold by PG as something special though.
> Maybe just accept that not everyone starts from fundamental theory, and there are lots of people who start learning by fooling around.
Even then LLMs are strange thing to advice to play with, when there are so much more interesting and theoretically accesible for a 17 y.o. so I wonder why would you'd particularly single it out.
I attempted many projects at a young age that I was absolutely not equipped for. The result of the attempts more often than not left me equipped, every time it left me better off. This is terrible advice.
Transformers are difficult to understand even to people with strong ML background, let alone a teenager.
That 17yo would have already built many uncommon bases, and would build further.
That is, a 17yo with proper mentality.
Can you provide an example?
Either way, this isn't really advice for 17 year olds. Pg is thinking out loud about the pathways for founders.
>Whoa. I’m 19 and I trained a 100M language model from scratch. Did a v2 now with a new SFT experiment to see if I can get better results on same size.