train_rte_1744902658

This model is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct on the rte dataset. It achieves the following results on the evaluation set:

• Loss: 0.0769
• Num Input Tokens Seen: 98761256

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 123
• gradient_accumulation_steps: 4
• total_train_batch_size: 16
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• training_steps: 40000

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
0.092	1.4207	200	0.1188	496688
0.0953	2.8414	400	0.1078	991488
0.0855	4.2567	600	0.1014	1481464
0.1016	5.6774	800	0.0951	1979088
0.0674	7.0927	1000	0.0916	2468504
0.0675	8.5134	1200	0.0880	2963120
0.0855	9.9340	1400	0.0856	3459048
0.0859	11.3494	1600	0.0827	3951104
0.0635	12.7701	1800	0.0819	4445432
0.0663	14.1854	2000	0.0801	4938824
0.0538	15.6061	2200	0.0799	5433720
0.0367	17.0214	2400	0.0782	5925896
0.0789	18.4421	2600	0.0781	6422360
0.0351	19.8627	2800	0.0769	6914152
0.0685	21.2781	3000	0.0776	7403976
0.0507	22.6988	3200	0.0777	7902520
0.0508	24.1141	3400	0.0784	8394080
0.0394	25.5348	3600	0.0784	8884224
0.0448	26.9554	3800	0.0804	9382368
0.0344	28.3708	4000	0.0781	9872768
0.0462	29.7914	4200	0.0827	10366000
0.0458	31.2068	4400	0.0818	10867488
0.0274	32.6275	4600	0.0828	11358568
0.0295	34.0428	4800	0.0867	11852320
0.0279	35.4635	5000	0.0853	12343880
0.0396	36.8841	5200	0.0864	12837040
0.0394	38.2995	5400	0.0882	13329368
0.0303	39.7201	5600	0.0914	13828784
0.0265	41.1355	5800	0.0913	14315304
0.0345	42.5561	6000	0.0965	14806592
0.022	43.9768	6200	0.0976	15305208
0.0608	45.3922	6400	0.0974	15791608
0.0329	46.8128	6600	0.0993	16292464
0.0078	48.2282	6800	0.0991	16781768
0.0096	49.6488	7000	0.1037	17278560
0.0179	51.0642	7200	0.1071	17769384
0.024	52.4848	7400	0.1106	18262680
0.025	53.9055	7600	0.1095	18763936
0.0057	55.3209	7800	0.1153	19258096
0.0096	56.7415	8000	0.1177	19753648
0.0165	58.1569	8200	0.1222	20244128
0.0033	59.5775	8400	0.1243	20739208
0.0165	60.9982	8600	0.1350	21236872
0.0311	62.4135	8800	0.1331	21726944
0.0183	63.8342	9000	0.1417	22223288
0.0047	65.2496	9200	0.1453	22716672
0.0114	66.6702	9400	0.1469	23209088
0.0033	68.0856	9600	0.1570	23701520
0.0019	69.5062	9800	0.1625	24197944
0.0007	70.9269	10000	0.1689	24694272
0.0085	72.3422	10200	0.1779	25191256
0.0024	73.7629	10400	0.1801	25688288
0.0032	75.1783	10600	0.1853	26177720
0.0094	76.5989	10800	0.1942	26675248
0.0036	78.0143	11000	0.1949	27168496
0.0012	79.4349	11200	0.2083	27664360
0.0006	80.8556	11400	0.2168	28161984
0.0003	82.2709	11600	0.2216	28655448
0.0001	83.6916	11800	0.2328	29151808
0.0003	85.1070	12000	0.2439	29642952
0.0012	86.5276	12200	0.2450	30140536
0.0002	87.9483	12400	0.2557	30639808
0.0001	89.3636	12600	0.2625	31135048
0.0001	90.7843	12800	0.2692	31630256
0.0	92.1996	13000	0.2778	32121256
0.0001	93.6203	13200	0.2851	32618184
0.0	95.0357	13400	0.2871	33115432
0.0	96.4563	13600	0.2967	33609472
0.0	97.8770	13800	0.2957	34098712
0.0	99.2923	14000	0.3018	34590368
0.0	100.7130	14200	0.3042	35081248
0.0	102.1283	14400	0.3081	35571464
0.0	103.5490	14600	0.3173	36063824
0.0	104.9697	14800	0.3213	36557944
0.0	106.3850	15000	0.3214	37048560
0.0	107.8057	15200	0.3286	37543928
0.0	109.2210	15400	0.3324	38035968
0.0	110.6417	15600	0.3341	38526000
0.0	112.0570	15800	0.3375	39021440
0.0	113.4777	16000	0.3440	39519712
0.0	114.8984	16200	0.3470	40014440
0.0	116.3137	16400	0.3465	40509368
0.0	117.7344	16600	0.3544	41001000
0.0	119.1497	16800	0.3556	41492672
0.0	120.5704	17000	0.3608	41991984
0.0	121.9911	17200	0.3677	42486736
0.0	123.4064	17400	0.3618	42979888
0.0	124.8271	17600	0.3672	43473920
0.0	126.2424	17800	0.3735	43963728
0.0	127.6631	18000	0.3795	44457208
0.0	129.0784	18200	0.3762	44952664
0.0	130.4991	18400	0.3836	45446704
0.0	131.9198	18600	0.3844	45936552
0.0	133.3351	18800	0.3866	46426240
0.0	134.7558	19000	0.3884	46921256
0.0	136.1711	19200	0.3936	47412080
0.0	137.5918	19400	0.4000	47911024
0.0	139.0071	19600	0.4013	48404752
0.0	140.4278	19800	0.3979	48901416
0.0	141.8485	20000	0.4048	49400736
0.0	143.2638	20200	0.4050	49895752
0.0	144.6845	20400	0.4090	50380736
0.0	146.0998	20600	0.4114	50871288
0.0	147.5205	20800	0.4133	51360328
0.0	148.9412	21000	0.4221	51853696
0.0	150.3565	21200	0.4133	52348712
0.0	151.7772	21400	0.4201	52842992
0.0	153.1925	21600	0.4187	53335368
0.0	154.6132	21800	0.4241	53831240
0.0	156.0285	22000	0.4271	54320840
0.0	157.4492	22200	0.4315	54818304
0.0	158.8699	22400	0.4378	55310560
0.0	160.2852	22600	0.4331	55805192
0.0	161.7059	22800	0.4387	56294240
0.0	163.1212	23000	0.4402	56785216
0.0	164.5419	23200	0.4422	57277112
0.0	165.9626	23400	0.4469	57768960
0.0	167.3779	23600	0.4443	58259216
0.0	168.7986	23800	0.4450	58754552
0.0	170.2139	24000	0.4488	59250304
0.0	171.6346	24200	0.4542	59743752
0.0	173.0499	24400	0.4631	60240920
0.0	174.4706	24600	0.4591	60738488
0.0	175.8913	24800	0.4609	61232632
0.0	177.3066	25000	0.4604	61726896
0.0	178.7273	25200	0.4639	62220440
0.0	180.1426	25400	0.4690	62713544
0.0	181.5633	25600	0.4743	63208560
0.0	182.9840	25800	0.4737	63703320
0.0	184.3993	26000	0.4700	64195280
0.0	185.8200	26200	0.4720	64693448
0.0	187.2353	26400	0.4812	65180864
0.0	188.6560	26600	0.4797	65680024
0.0	190.0713	26800	0.4736	66173368
0.0	191.4920	27000	0.4879	66664968
0.0	192.9127	27200	0.4814	67157528
0.0	194.3280	27400	0.4878	67657848
0.0	195.7487	27600	0.4905	68154280
0.0	197.1640	27800	0.4967	68648760
0.0	198.5847	28000	0.4929	69145424
0.0	200.0	28200	0.4865	69634592
0.0	201.4207	28400	0.5011	70126824
0.0	202.8414	28600	0.4969	70621048
0.0	204.2567	28800	0.5014	71112744
0.0	205.6774	29000	0.5011	71609328
0.0	207.0927	29200	0.5018	72096488
0.0	208.5134	29400	0.5106	72590600
0.0	209.9340	29600	0.5025	73085400
0.0	211.3494	29800	0.5078	73578704
0.0	212.7701	30000	0.5055	74071832
0.0	214.1854	30200	0.5065	74558088
0.0	215.6061	30400	0.5114	75054720
0.0	217.0214	30600	0.5145	75550968
0.0	218.4421	30800	0.5194	76052048
0.0	219.8627	31000	0.5042	76544760
0.0	221.2781	31200	0.5109	77039312
0.0	222.6988	31400	0.5114	77536608
0.0	224.1141	31600	0.5141	78029096
0.0	225.5348	31800	0.5181	78521640
0.0	226.9554	32000	0.5107	79014704
0.0	228.3708	32200	0.5216	79509056
0.0	229.7914	32400	0.5178	80004760
0.0	231.2068	32600	0.5175	80498576
0.0	232.6275	32800	0.5124	80992160
0.0	234.0428	33000	0.5140	81484216
0.0	235.4635	33200	0.5206	81981536
0.0	236.8841	33400	0.5279	82469112
0.0	238.2995	33600	0.5172	82967264
0.0	239.7201	33800	0.5282	83460632
0.0	241.1355	34000	0.5240	83946936
0.0	242.5561	34200	0.5260	84438976
0.0	243.9768	34400	0.5288	84936992
0.0	245.3922	34600	0.5308	85424648
0.0	246.8128	34800	0.5231	85921552
0.0	248.2282	35000	0.5305	86414392
0.0	249.6488	35200	0.5272	86904424
0.0	251.0642	35400	0.5262	87399560
0.0	252.4848	35600	0.5229	87900568
0.0	253.9055	35800	0.5293	88391952
0.0	255.3209	36000	0.5363	88887288
0.0	256.7415	36200	0.5269	89375944
0.0	258.1569	36400	0.5282	89868176
0.0	259.5775	36600	0.5216	90365056
0.0	260.9982	36800	0.5223	90855096
0.0	262.4135	37000	0.5215	91348504
0.0	263.8342	37200	0.5219	91843280
0.0	265.2496	37400	0.5263	92339160
0.0	266.6702	37600	0.5266	92834936
0.0	268.0856	37800	0.5363	93329096
0.0	269.5062	38000	0.5160	93825960
0.0	270.9269	38200	0.5332	94316976
0.0	272.3422	38400	0.5265	94808456
0.0	273.7629	38600	0.5259	95304384
0.0	275.1783	38800	0.5236	95796256
0.0	276.5989	39000	0.5329	96293992
0.0	278.0143	39200	0.5310	96783960
0.0	279.4349	39400	0.5310	97275176
0.0	280.8556	39600	0.5310	97769584
0.0	282.2709	39800	0.5310	98266712
0.0	283.6916	40000	0.5310	98761256

Framework versions

• PEFT 0.15.1
• Transformers 4.51.3
• Pytorch 2.6.0+cu124
• Datasets 3.5.0
• Tokenizers 0.21.1