Hugging Face's logo

JetBrains
/
Mellum-4b-sft-python

Text Generation
Transformers
Safetensors
llama
code
Eval Results
text-generation-inference
Model card Files Community
dustalov commited on
Commit
b8c3555
·
verified ·
1 Parent(s): 9756c2f

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +137 -0
README.md CHANGED
@@ -10,6 +10,143 @@ tags:
10
  - code
11
  base_model:
12
  - JetBrains/Mellum-4b-base
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
13
  ---
14
 
15
  # Model Description
 
10
  - code
11
  base_model:
12
  - JetBrains/Mellum-4b-base
13
+ model-index:
14
+ - name: Mellum-4b-sft-python
15
+ results:
16
+ - task:
17
+ type: text-generation
18
+ dataset:
19
+ type: tianyang/repobench_python_v1.1
20
+ name: RepoBench 1.1 (Python)
21
+ metrics:
22
+ - name: EM
23
+ type: exact_match
24
+ value: 0.2837
25
+ verified: false
26
+ - name: EM ≤ 8k
27
+ type: exact_match
28
+ value: 0.2987
29
+ verified: false
30
+ - task:
31
+ type: text-generation
32
+ dataset:
33
+ type: tianyang/repobench_python_v1.1
34
+ name: RepoBench 1.1 (Python, 2k)
35
+ metrics:
36
+ - name: EM
37
+ type: exact_match
38
+ value: 0.2924
39
+ verified: false
40
+ - task:
41
+ type: text-generation
42
+ dataset:
43
+ type: tianyang/repobench_python_v1.1
44
+ name: RepoBench 1.1 (Python, 4k)
45
+ metrics:
46
+ - name: EM
47
+ type: exact_match
48
+ value: 0.3060
49
+ verified: false
50
+ - task:
51
+ type: text-generation
52
+ dataset:
53
+ type: tianyang/repobench_python_v1.1
54
+ name: RepoBench 1.1 (Python, 8k)
55
+ metrics:
56
+ - name: EM
57
+ type: exact_match
58
+ value: 0.2977
59
+ verified: false
60
+ - task:
61
+ type: text-generation
62
+ dataset:
63
+ type: tianyang/repobench_python_v1.1
64
+ name: RepoBench 1.1 (Python, 12k)
65
+ metrics:
66
+ - name: EM
67
+ type: exact_match
68
+ value: 0.2680
69
+ verified: false
70
+ - task:
71
+ type: text-generation
72
+ dataset:
73
+ type: tianyang/repobench_python_v1.1
74
+ name: RepoBench 1.1 (Python, 16k)
75
+ metrics:
76
+ - name: EM
77
+ type: exact_match
78
+ value: 0.2543
79
+ verified: false
80
+ - task:
81
+ type: text-generation
82
+ dataset:
83
+ type: gonglinyuan/safim
84
+ name: SAFIM
85
+ metrics:
86
+ - name: pass@1
87
+ type: pass@1
88
+ value: 0.4212
89
+ verified: false
90
+ - task:
91
+ type: text-generation
92
+ dataset:
93
+ type: gonglinyuan/safim
94
+ name: SAFIM (Algorithmic)
95
+ metrics:
96
+ - name: pass@1
97
+ type: pass@1
98
+ value: 0.3316
99
+ verified: false
100
+ - task:
101
+ type: text-generation
102
+ dataset:
103
+ type: gonglinyuan/safim
104
+ name: SAFIM (Control)
105
+ metrics:
106
+ - name: pass@1
107
+ type: pass@1
108
+ value: 0.3611
109
+ verified: false
110
+ - task:
111
+ type: text-generation
112
+ dataset:
113
+ type: gonglinyuan/safim
114
+ name: SAFIM (API)
115
+ metrics:
116
+ - name: pass@1
117
+ type: pass@1
118
+ value: 0.5710
119
+ verified: false
120
+ - task:
121
+ type: text-generation
122
+ dataset:
123
+ type: loubnabnl/humaneval_infilling
124
+ name: HumanEval Infilling (Single-Line)
125
+ metrics:
126
+ - name: pass@1
127
+ type: pass@1
128
+ value: 0.8045
129
+ verified: false
130
+ - task:
131
+ type: text-generation
132
+ dataset:
133
+ type: loubnabnl/humaneval_infilling
134
+ name: HumanEval Infilling (Multi-Line)
135
+ metrics:
136
+ - name: pass@1
137
+ type: pass@1
138
+ value: 0.4819
139
+ verified: false
140
+ - task:
141
+ type: text-generation
142
+ dataset:
143
+ type: loubnabnl/humaneval_infilling
144
+ name: HumanEval Infilling (Random Span)
145
+ metrics:
146
+ - name: pass@1
147
+ type: pass@1
148
+ value: 0.3768
149
+ verified: false
150
  ---
151
 
152
  # Model Description