AI VIETNAM

All-in-One Course
(TA Session)

Instruction Tuning
Extra Class: LLMs

Dinh-Thang Duong – TA
Nguyen-Thuan Duong – TA

Year 2024
 AI VIETNAM
All-in-One Course
(TA Session) Objectives

In this lecture, we will discuss about:

1. How LLMs learn (training type)?
2. How to make LLMs adapt to a task?
1. Prompting (In-context learning)
2. Instruction Tuning
3. How to train LLMs on a single
(small) GPU?
4. Apply Instruction Tuning for multiple
choice math question solver task.

2
 AI VIETNAM
All-in-One Course
(TA Session)

Outline
Ø Introduction
Ø In-context Learning
Ø Instruction Tuning
Ø Parameter Efficient Fine-Tuning
Ø Evaluation
Ø Practices
Ø Question 3
 AI VIETNAM
All-in-One Course
(TA Session)

Introduction

4
 AI VIETNAM
All-in-One Course
(TA Session) Introduction
v Getting Started

Example: AI Chatbot using Large Language Models (LLMs)

5
 AI VIETNAM
All-in-One Course
(TA Session) Introduction
v Chatbot (ChatGPT)

Textual Description (Prompt):

Write a python function that receive an
image and plot it using matplotlib
library.

6
 AI VIETNAM
All-in-One Course
(TA Session) Introduction
v What are Large Language Models (LLMs)?

ChatGPT App:

1. User inputs a prompt (greetings,

task…).
2. ChatGPT (Chatbot) outputs a
appropriate response.

7
 AI VIETNAM
All-in-One Course
(TA Session) Introduction
v What are Large Language Models (LLMs)?

LLMs (Large Language Models): AI models (language models) that were trained on a very large corpus of text. This
made them capable of performing various NLP tasks with high precision.
8
 AI VIETNAM
All-in-One Course
(TA Session) Introduction
v What are Large Language Models (LLMs)?

LLMs are often pretrained on a

vast majority of data and
designed to be adaptable to a
wide variety of tasks
(Foundation models).

9
 AI VIETNAM
All-in-One Course
(TA Session) Introduction
v LLMs I/O
Output Text
Input Text

Write a python function

that receive an image and
plot it using matplotlib LLMs
library.

Given a “prompt”, LLMs can generate

an appropriate response.

10
 AI VIETNAM
All-in-One Course
(TA Session) Introduction
v Generative AI

11
https://www.jonstokes.com/p/getting-started-with-stable-diffusion
 AI VIETNAM
All-in-One Course
(TA Session) Introduction
v Generative AI Prompting

Prompting: Refers to a process of

providing an input, usually in the form of
text data, to a generative AI model to
generate a specific output.

12
https://medium.com/m/global-identity-2?redirectUrl=https%3A%2F%2Ftowardsdatascience.com%2Fpractical-prompt-engineering-74e96130abc4
 AI VIETNAM
All-in-One Course
(TA Session) Introduction
v Prompting in LLMs

With prompting, we can make LLMs do any task with just natural language (zero-shot capability)

13
 AI VIETNAM
All-in-One Course
(TA Session) Introduction
v LLMs problem

Question: How can we improve LLMs on

a specific task?

14
 AI VIETNAM
All-in-One Course
(TA Session) Introduction
v How to improve LLMs on specific tasks?

In-context learning

How to improve LLMs? Fine-tuning

Augmenting

15
 AI VIETNAM
All-in-One Course
(TA Session) Introduction
v How to improve LLMs on specific tasks?

In-context learning

How to improve LLMs? Fine-tuning

Augmenting

16
 AI VIETNAM
All-in-One Course
(TA Session)

In-context Learning

17
 AI VIETNAM
All-in-One Course
(TA Session) In-context learning
v Introduction

In-context learning (ICL) in LLMs is a technique

where task demonstrations are integrated into the
prompt in a natural language format. This
approach allows pre-trained LLMs to address new
tasks without fine-tuning the model.

18
https://towardsdatascience.com/in-context-learning-approaches-in-large-language-models-9c0c53b116a1
 AI VIETNAM
All-in-One Course
(TA Session) In-context learning
v Example

Make LLMs adapt to a task using instruction and examples

19
 AI VIETNAM
All-in-One Course
(TA Session) In-context learning
v In-context learning type

Zero/One/Few-shot
learning

In-context learning Chain-of-Thought

20
 AI VIETNAM
All-in-One Course
(TA Session) In-context learning
v Zero-shot learning

Zero-shot learning: Prompting LLMs to do a task

without any examples.

21
https://www.hopsworks.ai/dictionary/in-context-learning-icl
 AI VIETNAM
All-in-One Course
(TA Session) In-context learning
v One-shot learning

One-shot learning: Prompting LLMs to do a task with

an example.

22
https://www.hopsworks.ai/dictionary/in-context-learning-icl
 AI VIETNAM
All-in-One Course
(TA Session) In-context learning
v Few-shot learning

Few-shot learning: Prompting LLMs to do a task with

more than one example.

23
https://www.hopsworks.ai/dictionary/in-context-learning-icl
 AI VIETNAM
All-in-One Course
(TA Session) In-context learning
v Chain-of-Thought Prompting

Chain-of-Thought (CoT) prompting enables complex

reasoning capabilities through intermediate reasoning
steps. You can combine it with few-shot prompting to
get better results on more complex tasks that require
reasoning before responding.

24
 AI VIETNAM
All-in-One Course
(TA Session) In-context learning
v Zero-shot learning with CoT

25
https://www.promptingguide.ai/techniques/cot
 AI VIETNAM
All-in-One Course
(TA Session) In-context learning
v One-shot learning with CoT

26
https://www.promptingguide.ai/techniques/cot
 AI VIETNAM
All-in-One Course
(TA Session) In-context learning
v CoT variations

27
https://cobusgreyling.medium.com/the-anatomy-of-chain-of-thought-prompting-cot-b7489c925402
 AI VIETNAM
All-in-One Course
(TA Session)

Instruction Tuning

28
 AI VIETNAM
All-in-One Course
(TA Session) Instruction Tuning
v Training model in Machine Learning
Start with a randomized
1. Training weights and train the model
on a new task.

Start with a randomized

weights and train the model
Basic training type 2. Pre-training on a very large dataset -
wide range of tasks.

Start with a pre-trained

3. Fine-tuning weights and train the model
on a new task.
29
 AI VIETNAM
All-in-One Course
(TA Session) Instruction Tuning
v Training model in Machine Learning
Start with a randomized
1. Training weights and train the model
on a new task.

Start with a randomized

weights and train the model
Basic training type 2. Pre-training on a very large dataset -
wide range of tasks.

Start with a pre-trained

3. Fine-tuning weights and train the model
on a new task.
30
 AI VIETNAM
All-in-One Course
(TA Session) Instruction Tuning
v Fine-tuning

Fine-tuning

Machine
Training Learning Model
Dataset

Pre-trained Update
Weights Weights
Initialize (Knowledge)

31
 AI VIETNAM
All-in-One Course
(TA Session) Instruction Tuning
v Introduction

Instruction Tuning in LLMs is a training

method aimed at enhancing the model’s
ability to understand and execute natural
language instructions.

32
 AI VIETNAM
All-in-One Course
(TA Session) Instruction Tuning
v Training LLMs

33
https://www.lesswrong.com/posts/9asGWZ9vjmNDc4TeN/proposal-align-systems-earlier-in-training
 AI VIETNAM
All-in-One Course
(TA Session) Instruction Tuning
v Applications

34
https://intuitivetutorial.com/2023/06/18/large-language-models-in-deep-learning/
 AI VIETNAM
All-in-One Course
(TA Session) Instruction Tuning
v Applications

Fine-tuning on Inference on Task

Pretrained LLMs
Task A A

Fine-tuning to perform on a single

task (pretrain-finetune)

35
 AI VIETNAM
All-in-One Course
(TA Session) Instruction Tuning
v Use case: Multiple choice math question solver
Problem statement: Build a model that can choose the best answer (A, B, C or D) given a multiple choice math
question.

36
 AI VIETNAM
All-in-One Course
(TA Session) Instruction Tuning
v Use case: Multiple choice math question solver

37
 AI VIETNAM
All-in-One Course
(TA Session)

Parameter Efficient Fine-Tuning

38
 AI VIETNAM
All-in-One Course
(TA Session) Parameter Efficient Fine-tuning
v Problem Description

Model 1.5B parameters Model 7B parameters Model 175B parameters

(GPT-2) (LLaMa-7B) (GPT-3)

8-bits quantized 8-bits quantized 8-bits quantized

30GB @ 8 bits 140GB @ 8 bits 3500GB @ 8 bits

precision precision precision

A100 80GB OutOfMemoryError:

A100 40GB
A100 80GB Cuda out of memory

39
 AI VIETNAM
All-in-One Course
(TA Session) Parameter Efficient Fine-tuning
v PEFT type

Parameter-efficient Fine-tuning (PEFT) is a

technique used to improve the performance of pre-
trained language models on specific downstream
tasks. It involves reusing the pre-trained model’s
parameters and fine-tuning them on a smaller
dataset, which saves computational resources and
time compared to training the entire model from
scratch.

40
https://www.leewayhertz.com/parameter-efficient-fine-tuning/
 AI VIETNAM
All-in-One Course
(TA Session) Parameter Efficient Fine-tuning
v PEFT type

41
 AI VIETNAM
All-in-One Course
(TA Session) Parameter Efficient Fine-tuning
v LoRA: Low Rank Adaptation 𝑟 𝑘

𝑟 A

LoRA training 𝑑 B

ℎ = 𝑊𝑥 = 𝑊! 𝑥 + ∆𝑊𝑥
ℎ = 𝑊! 𝑥 + 𝐵𝐴𝑥
𝑘
𝐵 ∈ ℝ"×$ , 𝐴 ∈ ℝ$×%
𝑟 ≪ {𝑑, 𝑘}
𝑑

𝑇𝑟𝑎𝑖𝑛𝑎𝑏𝑙𝑒 𝑝𝑎𝑟𝑎𝑚𝑠 = 2×𝑑&'"() ×𝑟×𝐿? *'+,

∗ 𝐿" !"#$ is the number of linear layers applied to LoRA

42
 AI VIETNAM
All-in-One Course
(TA Session) Parameter Efficient Fine-tuning
v LoRA: Low Rank Adaptation

*
Pre-trained
Task A
weight

Frozen

*
Pre-trained
Task B
weight

Frozen

43
 AI VIETNAM
All-in-One Course
(TA Session) Parameter Efficient Fine-tuning
v QLoRA

QLoRA = Quantization + LoRA

Block-wise k-bit Quantization

k-bit NormalFloat Quantization

Double Quantization

44
 AI VIETNAM
All-in-One Course
(TA Session) Parameter Efficient Fine-tuning
v Performance of LoRA

Without LoRA With LoRA

45
 AI VIETNAM
All-in-One Course
(TA Session)

LLM Metrics

46
 AI VIETNAM
All-in-One Course
(TA Session) LLM Metrics
v How to evaluate LLMs?

How to efficiently evaluate the performance of LLMs?

47
 AI VIETNAM
All-in-One Course
(TA Session) LLM Metrics
v How to evaluate LLMs?

48
https://datasciencedojo.com/blog/evaluating-large-language-models-llms/
 AI VIETNAM
All-in-One Course
(TA Session) LLM Metrics
v BLEU score
BLEU (Bilingual Evaluation Understudy) is an algorithm for evaluating the quality of text which has been
machine-translated.

Reference I am a student of this university

1+1+1+1+1+1
𝑠𝑐𝑜𝑟𝑒 = =1
𝑙𝑒𝑛(𝑐𝑎𝑛𝑑𝑖𝑑𝑎𝑡𝑒)
Candidate I I a student student of

+1 +1 +1 +1 +1 +1

BLEU score is:

How many words from the candidate appear in the reference?
49
 AI VIETNAM
All-in-One Course
(TA Session) LLM Metrics
v BLEU score
Modified version

Reference I am a student of this university

1+1+1+1
𝑠𝑐𝑜𝑟𝑒 = = 0.66
𝑙𝑒𝑛(𝑐𝑎𝑛𝑑𝑖𝑑𝑎𝑡𝑒)

Candidate I I a student student of

+1 +0 +1 +1 +0 +1

Cons:
- don't consider semantic meaning
- don't consider sentence structure

50
 AI VIETNAM
All-in-One Course
(TA Session) LLM Metrics
v ROUGE score
ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a set of metrics for evaluating text generation
models (summarization or machine translation).

Reference 1 I am a student of this university

1+1+1+1
𝑠𝑐𝑜𝑟𝑒1 = = 0.57
+1 +0 +1 +1 +1 𝑙𝑒𝑛(𝑟𝑒𝑓𝑒𝑟𝑒𝑛𝑐𝑒)

Candidate I I a student student of Ninal score = max(score1, score2)

+1 +0 +1 +0 +1 1+1+1
𝑠𝑐𝑜𝑟𝑒2 = = 0.42
𝑙𝑒𝑛(𝑟𝑒𝑓𝑒𝑟𝑒𝑛𝑐𝑒)
Reference 2 I am a member of this university
51
 AI VIETNAM
All-in-One Course
(TA Session) LLM Metrics
v Other metrics?

Evaluate LLMs on datasets built specificly for some capabilities

52
https://msandbu.org/benchmarking-llms-and-what-is-the-best-llm/
 AI VIETNAM
All-in-One Course
(TA Session)

Quiz

53
 AI VIETNAM
All-in-One Course
(TA Session)

Practices

54
 AI VIETNAM
All-in-One Course
(TA Session) Practices
v Problem Description
Description: Build a Vietnamese Chatbot that can handle math problem using Large Language Models (LLMs).

55
 AI VIETNAM
All-in-One Course
(TA Session) Practices
v Problem Description
Description: Build a Vietnamese Chatbot that can handle math problem using Large Language Models (LLMs).

We will apply fine-tuning to improve pre-trained performance

56
 AI VIETNAM
All-in-One Course
(TA Session) Practices
v Coding Step 1: Install libraries

57
 AI VIETNAM
All-in-One Course
(TA Session) Practices
v Coding Step 2: Import libraries/modules

58
 AI VIETNAM
All-in-One Course
(TA Session) Practices
v Coding Step 3: Load pre-trained model

59
 AI VIETNAM
All-in-One Course
(TA Session) Practices
v Coding Step 3: Load pre-trained model

60
 AI VIETNAM
All-in-One Course
(TA Session) Practices
v Coding Step 4: Configurate LLMs

Generation configuration (Will affect the generation results)

61
 AI VIETNAM
All-in-One Course
(TA Session) Practices
v Fine-tuning LLMs

Fine-tuning

LLMs
Training
Dataset

Pre-trained Update
Weights Weights
Initialize (Knowledge)

62
 AI VIETNAM
All-in-One Course
(TA Session) Practices
v Coding Step 5: Download dataset

Fine-tune VinaLLaMA on
hllj/vi_grade_school_math_mc
q
(https://huggingface.co/datasets
/hllj/vi_grade_school_math_mcq)
, a Vietnamese elementary math
dataset.

63
 AI VIETNAM
All-in-One Course
(TA Session) Practices
v Coding Step 5: Download dataset

64
 AI VIETNAM
All-in-One Course
(TA Session) Practices
v Prompting format

<|im_start|>system
Bạn là một chuyên gia về toán. Bạn sẽ nhận câu hỏi trắc nghiệm kèm theo các lựa
chọn, hãy giải step by step nếu có và chọn phương án đúng.

<|im_start|>user
### Câu hỏi:
{question}
### Các lựa chọn:
{choices}
### Câu trả lời:

<|im_start|>assistant
{explanation}

65
 AI VIETNAM
All-in-One Course
(TA Session) Practices
v Coding Step 6: Create generate prompt function

66
 AI VIETNAM
All-in-One Course
(TA Session) Practices
v Coding Step 7: Create training samples

67
 AI VIETNAM
All-in-One Course
(TA Session) Practices
v Coding Step 7: Create training samples

In problems column, each sample has 1 list of problems:

1. Choices
2. Question
3. Explanation

68
 AI VIETNAM
All-in-One Course
(TA Session) Practices
v Coding Step 7: Create training samples

Extract question,
Call generate
Take a dictionary choices,
prompt function
explanation

No

Problems Yes
End of list ?
column

Add sample to list New prompt

69
 AI VIETNAM
All-in-One Course
(TA Session) Practices
v Coding Step 7: Create training samples

70
 AI VIETNAM
All-in-One Course
(TA Session) Practices
v Coding Step 8: Training

71
 AI VIETNAM
All-in-One Course
(TA Session) Practices
v Fine-tuning results

Before fine-tuning

72
 AI VIETNAM
All-in-One Course
(TA Session) Practices
v Fine-tuning results

After fine-tuning

73
 AI VIETNAM
All-in-One Course
(TA Session) Summary
In this lecture, we have discussed:
1. Training type
1. Pre-training
2. Fine-tuning
2. How to make LLMs adapt to a task?
1. Prompting (In-context learning)
1. One-shot learning: Prompting with 1 example.
2. Few-shot learning: Prompting with more than 1 example.
3. Chain-of-Thought: Prompting with reasoning.
2. Instruction Tuning: By supervised learning LLMs with instruction data.
3. How to train LLMs on a single (small) GPU?
1. Parameter Efficient Fine-Tuning
4. Apply Instruction Tuning for multiple choice math question solver task.
74
 AI VIETNAM
All-in-One Course
(TA Session) Question

?
75
76