How-to & Style

Home
Articles

Let's build GPT: from scratch, in code, spelled out. 121gamers 0 Views • 10 months ago

We build a Generatively Pretrained Transformer (GPT), following the paper "Attention is All You Need" and OpenAI's GPT-2 / GPT-3. We talk about connections to ChatGPT, which has taken the world by storm. We watch GitHub Copilot, itself a GPT, help us write a GPT (meta :D!) . I recommend people watch the earlier makemore videos to get comfortable with the autoregressive language modeling framework and basics of tensors and PyTorch nn, which we take for granted in this video.

Links:
- Google colab for the video: https://colab.research.google.....com/drive/1JMLa53HDu
- GitHub repo for the video: https://github.com/karpathy/ng-video-lecture
- Playlist of the whole Zero to Hero series so far: https://www.youtube.com/watch?v=VMj-3S1tku0&list=PLAqhIrjkxbuWI23v9cThsA9GvCAUhRvKZ
- nanoGPT repo: https://github.com/karpathy/nanoGPT
- my website: https://karpathy.ai
- my twitter: https://twitter.com/karpathy
- our Discord channel: https://discord.gg/3zy8kqD9Cp

Supplementary links:
- Attention is All You Need paper: https://arxiv.org/abs/1706.03762
- OpenAI GPT-3 paper: https://arxiv.org/abs/2005.14165
- OpenAI ChatGPT blog post: https://openai.com/blog/chatgpt/
- The GPU I'm training the model on is from Lambda GPU Cloud, I think the best and easiest way to spin up an on-demand GPU instance in the cloud that you can ssh to: https://lambdalabs.com . If you prefer to work in notebooks, I think the easiest path today is Google Colab.

Suggested exercises:
- EX1: The n-dimensional tensor mastery challenge: Combine the `Head` and `MultiHeadAttention` into one class that processes all the heads in parallel, treating the heads as another batch dimension (answer is in nanoGPT).
- EX2: Train the GPT on your own dataset of choice! What other data could be fun to blabber on about? (A fun advanced suggestion if you like: train a GPT to do addition of two numbers, i.e. a+b=c. You may find it helpful to predict the digits of c in reverse order, as the typical addition algorithm (that you're hoping it learns) would proceed right to left too. You may want to modify the data loader to simply serve random problems and skip the generation of train.bin, val.bin. You may want to mask out the loss at the input positions of a+b that just specify the problem using y=-1 in the targets (see CrossEntropyLoss ignore_index). Does your Transformer learn to add? Once you have this, swole doge project: build a calculator clone in GPT, for all of +-*/. Not an easy problem. You may need Chain of Thought traces.)
- EX3: Find a dataset that is very large, so large that you can't see a gap between train and val loss. Pretrain the transformer on this data, then initialize with that model and finetune it on tiny shakespeare with a smaller number of steps and lower learning rate. Can you obtain a lower validation loss by the use of pretraining?
- EX4: Read some transformer papers and implement one additional feature or change that people seem to use. Does it improve the performance of your GPT?

Chapters:
00:00:00 intro: ChatGPT, Transformers, nanoGPT, Shakespeare
baseline language modeling, code setup
00:07:52 reading and exploring the data
00:09:28 tokenization, train/val split
00:14:27 data loader: batches of chunks of data
00:22:11 simplest baseline: bigram language model, loss, generation
00:34:53 training the bigram model
00:38:00 port our code to a script
Building the "self-attention"
00:42:13 version 1: averaging past context with for loops, the weakest form of aggregation
00:47:11 the trick in self-attention: matrix multiply as weighted aggregation
00:51:54 version 2: using matrix multiply
00:54:42 version 3: adding softmax
00:58:26 minor code cleanup
01:00:18 positional encoding
01:02:00 THE CRUX OF THE VIDEO: version 4: self-attention
01:11:38 note 1: attention as communication
01:12:46 note 2: attention has no notion of space, operates over sets
01:13:40 note 3: there is no communication across batch dimension
01:14:14 note 4: encoder blocks vs. decoder blocks
01:15:39 note 5: attention vs. self-attention vs. cross-attention
01:16:56 note 6: "scaled" self-attention. why divide by sqrt(head_size)
Building the Transformer
01:19:11 inserting a single self-attention block to our network
01:21:59 multi-headed self-attention
01:24:25 feedforward layers of transformer block
01:26:48 residual connections
01:32:51 layernorm (and its relationship to our previous batchnorm)
01:37:49 scaling up the model! creating a few variables. adding dropout
Notes on Transformer
01:42:39 encoder vs. decoder vs. both (?) Transformers
01:46:22 super quick walkthrough of nanoGPT, batched multi-headed self-attention
01:48:53 back to ChatGPT, GPT-3, pretraining vs. finetuning, RLHF
01:54:32 conclusions

Corrections:
00:57:00 Oops "tokens from the _future_ cannot communicate", not "past". Sorry! :)
01:20:05 Oops I should be using the head_size for the normalization, not C

How to Build an LLM from Scratch | An Overview 121gamers 0 Views • 10 months ago

Want to learn more? I’m launching a 6-week live BootCamp for AI Builders.
👉 Learn more: https://maven.com/s/course/13437a45a7
Save 50% at checkout with the code FOUNDER50

This is the 6th video in a series on using large language models (LLMs) in practice. Here, I review key aspects of developing a foundation LLM based on the development of models such as GPT-3, Llama, Falcon, and beyond.

More Resources:
▶️ Series Playlist: https://www.youtube.com/playli....st?list=PLz-ep5RbHos Read more: https://towardsdatascience.com..../how-to-build-an-llm

[1] BloombergGPT: https://arxiv.org/pdf/2303.17564.pdf
[2] Llama 2: https://ai.meta.com/research/p....ublications/llama-2-
[3] LLM Energy Costs: https://www.statista.com/stati....stics/1384401/energy
[4] arXiv:2005.14165 [cs.CL]
[5] Falcon 180b Blog: https://huggingface.co/blog/falcon-180b
[6] arXiv:2101.00027 [cs.CL]
[7] Alpaca Repo: https://github.com/gururise/AlpacaDataCleaned
[8] arXiv:2303.18223 [cs.CL]
[9] arXiv:2112.11446 [cs.CL]
[10] arXiv:1508.07909 [cs.CL]
[11] SentencePience: https://github.com/google/sent....encepiece/tree/maste
[12] Tokenizers Doc: https://huggingface.co/docs/tokenizers/quicktour
[13] arXiv:1706.03762 [cs.CL]
[14] Andrej Karpathy Lecture: https://www.youtube.com/watch?v=kCc8FmEb1nY&t=5307s
[15] Hugging Face NLP Course: https://huggingface.co/learn/n....lp-course/chapter1/7
[16] arXiv:1810.04805 [cs.CL]
[17] arXiv:1910.13461 [cs.CL]
[18] arXiv:1603.05027 [cs.CV]
[19] arXiv:1607.06450 [stat.ML]
[20] arXiv:1803.02155 [cs.CL]
[21] arXiv:2203.15556 [cs.CL]
[22] Trained with Mixed Precision Nvidia: https://docs.nvidia.com/deeple....arning/performance/m
[23] DeepSpeed Doc: https://www.deepspeed.ai/training/
[24] https://paperswithcode.com/method/weight-decay
[25] https://towardsdatascience.com..../what-is-gradient-cl
[26] arXiv:2001.08361 [cs.LG]
[27] arXiv:1803.05457 [cs.AI]
[28] arXiv:1905.07830 [cs.CL]
[29] arXiv:2009.03300 [cs.CY]
[30] arXiv:2109.07958 [cs.CL]
[31] https://huggingface.co/blog/ev....aluating-mmlu-leader
[32] https://www.cs.toronto.edu/~hi....nton/absps/JMLRdropo

--
Homepage: https://shawhintalebi.com/
Book a call: https://calendly.com/shawhintalebi

Intro - 0:00
How much does it cost? - 1:30
4 Key Steps - 3:55
Step 1: Data Curation - 4:19
1.1: Data Sources - 5:31
1.2: Data Diversity - 7:45
1.3: Data Preparation - 9:06
Step 2: Model Architecture (Transformers) - 13:17
2.1: 3 Types of Transformers - 15:13
2.2: Other Design Choices - 18:27
2.3: How big do I make it? - 22:45
Step 3: Training at Scale - 24:20
3.1: Training Stability - 26:52
3.2: Hyperparameters - 28:06
Step 4: Evaluation - 29:14
4.1: Multiple-choice Tasks - 30:22
4.2: Open-ended Tasks - 32:59
What's next? - 34:31

Create Realistic Ai Influencer for Free | AI Model Instagram 2024 | Create Your Own Ai Influencer 121gamers 0 Views • 10 months ago

Create Realistic Ai Influencer for Free | AI Model Instagram 2024 | Create Your Own Ai Influencer
#aiinfluencer #aimodel #instagraminfluencer

Hello Friends! In this incredible video tutorial, you're going to explore realistic AI creation and its impact. We'll show you the process of making amazingly impressive AI influencer profiles, as well as videos that make you doubt the truth of. From gathering sources to making breathtaking video and even face-changing, this class will change your understanding of artificial intelligence. If you're an experienced techie or just interested in AI's capabilities, AI this video tutorial should be watched. Learn the hidden secrets behind AI and allow people to convey their condolences. Let's look at what's coming the next step for influencer culture and AI imagination!
How to create a realistic AI influencer for free using only AI tools. I have used AI tools like Tensor Art, Insight Face bot, and Google Colab to create AI Instagram models and earn money through Instagram channel.
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

Step to Create Ai Influencer 🔗Links:
👌STEP-01➡️
TensorArt: https://tensor.art/
Demo Model: https://tensor.art/images/700972640476656117?model_id=700914916103796439
👌STEP-02 ➡️
Discord: https://discord.com/channels/@me
Face Swap: https://discord.com/oauth2/authorize?client_id=1090660574196674713&permissions=274877945856&scope=bot
👌STEP-03 ➡️
Google Colab: https://colab.research.google.....com/gist/trickydaya/
👌STEP-04 ➡️
🖼️Free Webste to Download Images & Videos:
a) Freepik: https://www.freepik.com/
b) Pexels: https://www.pexels.com/
c) Pixabay: https://pixabay.com/
d) Coverr: https://coverr.co/
🖼️Paid Webste to Download Images, Videos & Templates:
a) Envatoelements: https://elements.envato.com/
b) Storyblocks: https://www.storyblocks.com/
c) Istocks: https://www.istockphoto.com/
d) Shutterstock: https://www.shutterstock.com/
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
👧To Create Realistic AI Influencers Timestamps:
00:00 - Introduction
00:48 - Create Realistic AI Influencer Roadmap
01:33 - Generate Realistic AI Influencer Using Tensor Art
03:48 - AI Generated Pictures
04:07 - How to Generate face swap Images from same AI Influencer's face
04:38 - Face Swap Using Insight Face Bot
05:18 - How to Download Pictures and Videos
06:10 - Generate Instagram Reels & Videos Using Google Colab AI
07:04 - Create Instagram Account
07:08 - Create AI Influencer's Bio & Post
07:19 - Conclusion
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
👍Aaditri Official Instagram ID: https://www.instagram.com/aaditritheai/

⚠ Disclaimer:
I don't recommend using videos of other people for deepfakes. Try to make them yourself; ask your girlfriend, sister, mother, or anyone on Fiverr to do it. It was shown in the tutorial to demonstrate the capabilities of AI.
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
👍Credit:
Instagram links: https://www.instagram.com/naina_avtr/?hl=en
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
🌐 Stay Connected:
▶YouTube: www.youtube.com/@theailearning
▶Twitter: https://twitter.com/theailearning
▶Facebook: https://www.facebook.com/theailearning
▶Instagram: https://www.instagram.com/theailearning
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
📺 Related Videos:
📹AI Infographic Generator form Text: https://youtu.be/fEtMYjXPItc
📹How to Create Realistic AI Influencer From Shakker AI: https://youtu.be/1Wp2ToHSNnA
📹How to Make Animated Cartoon Video Using AI 2024: https://youtu.be/n714HoPxV1Q
📹Create Viral Talking AI Avatar for Free: https://youtu.be/gL_YBXIPS4c
📹Best Ways To Turn Any Video To Animation AI 2024: https://youtu.be/-Re1ydSt3z4
📹We Create Realistic AI Influencer 2024 | Ai Consistent Characters & Clothes Change AI | DeFooocus AI: https://youtu.be/vRqwWknZHnY
📹Create Your Own Realistic Indian AI Influencer: https://youtu.be/1-E7fvhbIwM
📹Google Colab Deepfake Face Swap: https://youtu.be/yJlQ_ZXALlI
📹Text to Video AI Free Tool & Images to VIdeo AI 2024: https://youtu.be/LX0cIyQgxHo
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

📩 Business Inquiries: technicaldayanand@gmail.com

learning AI and ChatGPT isn’t that hard 121gamers 0 Views • 10 months ago

Create your own machine learning model RIGHT NOW:
💥💥START HERE: Create a FREE Oracle Cloud account: https://bit.ly/3WYnLtc
🧪LAB 1 - Data Extraction: https://bit.ly/3GCyPa4
🧪LAB 2 - Model Building with scikit-learn and AutoGluon: https://bit.ly/3Ik2gii
🧪LAB 3: Build a Neural Network: https://bit.ly/3ZaJHTZ

CHECKOUT ORACLE
---------------------------------------------------
-Developer.Oracle.Com: https://bit.ly/3WSCSV4
-Join the Oracle Slack community: https://bit.ly/odevrel_slack
-another fun Hand-on-Lab: Oracle RedBull Pit Strategy http://bit.ly/3y68YCg

Anyone can learn machine learning. You don’t need a fancy college degree nor do you need to be a math genius. No matter where you are in your journey, you can start learning machine learning and Artificial Intelligence. In this video, NetworkChuck will show you how you can get started with Machine Learning, an insanely lucrative path. So, don’t wait, start learning the technologies behind things like ChatGPT and OpenAI.

Learn more about Santiago (a REAL ML Engineer)
---------------------------------------------------
YouTube: https://www.youtube.com/@underfitted
Twitter: https://twitter.com/svpino
Website: https://www.svpino.com/

🔥🔥Join the NetworkChuck Academy!: https://ntck.co/NCAcademy

**Sponsored by Oracle Cloud

ALTERNATE TITLES:
- The Future is Here: How AI is Revolutionizing Our Lives
- The Power of AI: How Machine Learning is Changing the Game
- AI: The Next Frontier in Technology
- The Rise of AI: How Artificial Intelligence is Changing the World
- The AI Revolution: How Machine Learning is Transforming Industries
- AI for Beginners: A Guide to Understanding Artificial Intelligence
- AI 101: Everything You Need to Know About Artificial Intelligence
- Demystifying AI: Understanding the Basics of Machine Learning
- AI Made Easy: A Beginner's Guide to Artificial Intelligence
- Unlocking the Power of AI: A Comprehensive Guide to Machine Learning
- i taught my computer to tell me I suck at video games
- i taught my computer to play video games
- i used AI to win League of Legends
- chatgpt thinks I suck at video games
- i used machine learning to suck less at video games
- i taught my computer LoL (machine learning)
- my computer predicted i suck at video games (machine learning)
- you need to learn Machine Learning RIGHT NOW!!
- machine learning is easy
- even idiots can learn machine learning
- even dumb dumb stupids can learn machine learning
- i learned machine learning in 24 hours
- machine learning isn’t that hard (7 Steps)
- machine learning isn’t that hard (even I can do it)
- learning AI isn’t that hard
- you to need to learn AI RIGHT NOW!!
- **learning AI isn’t that hard**
-
- AI is Easier Than You Think
- Don't Fear AI: It's Easier Than You Think
- AI: It's Easier Than You Realize
- Take the Leap: Learning AI Isn't Hard
- Easily Learn AI - You Can Do It!
- Learning AI: Start Small and Achieve Big Results
- AI: It's Easier than You Think - Get Started Today
- AI: Learn It and Conquer It
- Don't Panic: AI Is Easier to Learn Than You Think
- AI: Uncover Its Secrets and Conquer Its Challenges

SUPPORT NETWORKCHUCK
---------------------------------------------------
➡️NetworkChuck membership: https://ntck.co/Premium
☕☕ COFFEE and MERCH: https://ntck.co/coffee

Check out my new channel: https://ntck.co/ncclips

🆘🆘NEED HELP?? Join the Discord Server: https://discord.gg/networkchuck

STUDY WITH ME on Twitch: https://bit.ly/nc_twitch

READY TO LEARN??
---------------------------------------------------
-Learn Python: https://bit.ly/3rzZjzz
-Get your CCNA: https://bit.ly/nc-ccna

FOLLOW ME EVERYWHERE
---------------------------------------------------
Instagram: https://www.instagram.com/networkchuck/
Twitter: https://twitter.com/networkchuck
Facebook: https://www.facebook.com/NetworkChuck/
Join the Discord server: http://bit.ly/nc-discord

AFFILIATES & REFERRALS
---------------------------------------------------
(GEAR I USE...STUFF I RECOMMEND)
My network gear: https://geni.us/L6wyIUj
Amazon Affiliate Store: https://www.amazon.com/shop/networkchuck
Buy a Raspberry Pi: https://geni.us/aBeqAL
Do you want to know how I draw on the screen?? Go to https://ntck.co/EpicPen and use code NetworkChuck to get 20% off!!
fast and reliable unifi in the cloud: https://hostifi.com/?via=chuck

#machinelearning #chatgpt #artificialintelligence