This folder contains files to run JerryScript in mbed / for:

* Freedom-K64F (k64)
* Discovery-STM32F4 (stm32f4)
* Discovery-STM32F429ZI (stm32f429i)
* Nucleo-F401RE (nucleo)

####Yotta
You need to install yotta before proceeding. Please visit [Yotta docs page](http://yottadocs.mbed.com/#installing-on-linux).

####Cross-compiler
For cross-compilation the GCC 5.2.1 is suggested to be used. All the supported targets were tested with this version. If you don't have any GCC compiler installed, please visit [this](https://launchpad.net/gcc-arm-embedded/+download) page to download GCC 5.2.1.

####How to build a target
Navigate to your JerryScript root folder (after you cloned this repository into the targets folder) and use the following command:

```
make -f targets/mbed/Makefile.mbed board=$(TARGET)
```
Where the `$(TARGET)` is one of the following options: `k64f`, `stm32f4`, `stm32f429i` or `nucleo`.
This command will create a new folder for your target and build the jerryscript and mbed OS into that folder.

####How to build a completely new target
If you want to build a new target (which is not available in this folder) you have to modify the makefile.
You have to add the new board name to the `TARGET_LIST` and you have to add a new branch with the new `YOTTA_TARGET` and a new `TARGET_DIR` path (if it neccessary) to the if at the top in the Makefile (just as you see right now).
There is a little code snippet:

```
ifeq ($(TARGET), k64f)
  YOTTA_TARGET  = frdm-k64f-gcc
  TARGET_DIR   ?= /media/$(USER)/MBED
else ifeq ($(TARGET), stm32f4)
  YOTTA_TARGET  =
```

Basically, you can create a new target in this way (If the mbed OS support your board).

#####Let's get into the details!
1. The next rule is the `jerry` rule. This rule builds the JerryScript and copy the output files into the target libjerry folder. Two files will be generated at `targets/mbed/libjerry`:
  * libjerrycore.a
  * libfdlibm.a

  You can run this rule with the following command:
  - `make -f targets/mbed/Makefile.mbed board=$(TARGET) jerry`

2. The next rule is the `js2c`. This rule calls a `js2c.py` python script from the `jerryscript/targets/tools` and creates the JavaScript builtin file into the `targets/mbed/source/` folder. This file is the `jerry_targetjs.h`. You can run this rule with the follwoing command:

  - `make -f targets/mbed/Makefile.mbed board=$(TARGET) js2c`

3. The last rule is the `yotta`. This rule sets the yotta target and install the mbed-drivers module, install the dependencies for the mbed OS and finaly creates the mbed binary file. The binary file will be genrated at `targets/mbed/build/$(YOTTA_TARGET)/source/jerry.bin`. You can run this rule with the following command:

  - `make -f targets/mbed/Makefile.mbed board=$(TARGET) yotta`

4. Optional rule: `clean`. It removes the build folder from the mbed and jerry. You can run this rule with this command:

  - `make -f targets/mbed/Makefile.mbed board=$(TARGET) clean`

#####Flashing
When the build is finished you can flash the binary into your board if you want. In case of ST boards you have to install the `st-link` software. Please visit [this page](https://github.com/texane/stlink) to install STLink-v2.
You can flash your binary into your board with the following command:
```
make -f targets/mbed/Makefile.mbed board=$(TARGET) flash
```
The flash rule grabs the binary and copies it to the mounted board or use the STLink-v2 to flash.
When the status LED of the board stops blinking, press RESET button on the board to execute JerryScript led flashing sample program in js folder.

###Note
If you use an STM32F4 board your build will stop with missing header errors. To fix this error please visit to [this page](http://browser.sed.hu/blog/20160407/how-run-javascripts-jerryscript-mbed) and read about the fix in the `New target for STM32F4` block.


# JerryScript with mbed OS 5

TL;DR? jump straight to [quickstart](#quick-start)

## Introduction

This directory contains the necessary code to build JerryScript for devices
capable of running mbed OS 5. It has been tested with the following boards
so far:

- [Nordic Semiconductor NRF52 Development Kit](https://developer.mbed.org/platforms/Nordic-nRF52-DK/)
- [NXP Freedom K64F](https://developer.mbed.org/platforms/FRDM-K64F/)
- [STM NUCLEO F401RE](https://developer.mbed.org/platforms/ST-Nucleo-F401RE/)
- [Silicon Labs EFM32 Giant Gecko](https://developer.mbed.org/platforms/EFM32-Giant-Gecko/)

## Features

### Peripheral Drivers

Peripheral Drivers are intended as a 1-to-1 mapping to mbed C++ APIs, with a few
differences (due to differences between JavaScript and C++ like lack of operator
overloading).

- [DigitalOut](https://docs.mbed.com/docs/mbed-os-api-reference/en/5.1/APIs/io/DigitalOut/)
- [InterruptIn](https://docs.mbed.com/docs/mbed-os-api-reference/en/5.1/APIs/io/InterruptIn/)
- [I2C](https://docs.mbed.com/docs/mbed-os-api-reference/en/5.1/APIs/interfaces/digital/I2C/)
- setInterval and setTimeout using [mbed-event](https://github.com/ARMmbed/mbed-events)

## Dependencies

### mbed CLI

mbed CLI is used as the build tool for mbed OS 5. You can find out how to install
it in the [official documentation](https://docs.mbed.com/docs/mbed-os-handbook/en/5.1/dev_tools/cli/#installing-mbed-cli).

### arm-none-eabi-gcc

arm-none-eabi-gcc is the only currently tested compiler for jerryscript on mbed,
and instructions for building can be found as part of the mbed-cli installation
instructions above.

### make

make is used to automate the process of fetching dependencies, and making sure that
mbed-cli is called with the correct arguments.

### (optional) jshint

jshint is used to statically check your JavaScript code, as part of the build process.
This ensures that pins you are using in your code are available on your chosen target
platform.

## Quick Start

Once you have all of your dependencies installed, you can build the project as follows:

```bash
git clone https://github.com/Samsung/jerryscript
cd jerryscript/targets/mbedos5
make getlibs
# NRF52 Development Kit:
make BOARD=NRF52_DK
# FRDM K64F
make BOARD=K64F
```

The produced file (in .build/**[BOARD]**/GCC_ARM) can then be uploaded to your board, and will
run when you press reset.

If you make a modification to main.js, you can simply rerun make, and it will remember your
previous choice of board:

```bash
make
```

This is the latest version of the internal repository from Pebble Technology providing the software to run on Pebble watches. Proprietary source code has been removed from this repository and it will not compile as-is. This is for information only.


### About

This folder contains files to run JerryScript on
[STM32F4-Discovery board](http://www.st.com/content/st_com/en/products/evaluation-tools/product-evaluation-tools/mcu-eval-tools/stm32-mcu-eval-tools/stm32-mcu-discovery-kits/stm32f4discovery.html) with [NuttX](http://nuttx.org/)

### How to build

#### 1. Setting up the build environment for STM32F4-Discovery board

Clone JerryScript and NuttX into jerry-nuttx directory

```
mkdir jerry-nuttx
cd jerry-nuttx
git clone https://github.com/Samsung/jerryscript.git
git clone https://bitbucket.org/nuttx/nuttx.git
git clone https://bitbucket.org/nuttx/apps.git
git clone https://github.com/texane/stlink.git
```

The following directory structure is created after these commands

```
jerry-nuttx
  + apps
  + jerryscript
  |  + targets
  |      + nuttx-stm32f4
  + nuttx
  + stlink
```

#### 2. Adding JerryScript as an interpreter for NuttX

```
cd apps/interpreters
mkdir jerryscript
cp ../../jerryscript/targets/nuttx-stm32f4/* ./jerryscript/
```

#### 3. Configure NuttX

```
# assuming you are in jerry-nuttx folder
cd nuttx/tools

# configure NuttX USB console shell
./configure.sh stm32f4discovery/usbnsh

cd ..
# might require to run "make menuconfig" twice
make menuconfig
```

We must set the following options:

* Change `Build Setup -> Build Host Platform` from _Windows_ to _Linux_
* Enable `System Type -> FPU support`
* Enable `Library Routines -> Standard Math library`
* Enable `Application Configuration -> Interpreters -> JerryScript`

#### 4. Build JerryScript for NuttX

```
# assuming you are in jerry-nuttx folder
cd nuttx/
make
```

#### 5. Flashing

Connect Mini-USB for power supply and connect Micro-USB for `NSH` console.

To configure `stlink` utility for flashing, follow the instructions [here](https://github.com/texane/stlink#build-from-sources).

To flash,
```
# assuming you are in nuttx folder
sudo ../stlink/build/st-flash write nuttx.bin 0x8000000
```

### Running JerryScript

You can use `minicom` for terminal program, or any other you may like, but set
baud rate to `115200`.

```
sudo minicom --device=/dev/ttyACM0 --baud=115200
```

You may have to press `RESET` on the board and press `Enter` keys on the console
several times to make `nsh` prompt to appear.

If the prompt shows like this,
```
NuttShell (NSH)
               nsh>
                    nsh>
                         nsh>
```
please set `Add Carriage Ret` option by `CTRL-A` > `Z` > `U` at the console,
if you're using `minicom`.

Run `jerry` with javascript file(s)

```
NuttShell (NSH)
nsh> jerry full_path/any.js
```

Without argument it prints:
```
nsh> jerry
No input files, running a hello world demo:
Hello world 5 times from JerryScript
```


### About

This folder contains files to run JerryScript on RIOT-OS with STM32F4-Discovery board.

### How to build

#### 1. Preface

1, Directory structure

Assume `harmony` as the path to the projects to build.
The folder tree related would look like this.

```
harmony
  + jerryscript
  |  + targets
  |      + riot-stm32f4
  + RIOT
```

2, Target board

Assume [STM32F4-Discovery with BB](http://www.st.com/web/en/catalog/tools/FM116/SC959/SS1532/LN1199/PF255417)
as the target board.

#### 2. Prepare RIOT-OS

Follow [this](https://www.riot-os.org/#download) page to get the RIOT-OS source.

Follow the [Inroduction](https://github.com/RIOT-OS/RIOT/wiki/Introduction) wiki site and also check that you can flash the stm32f4-board.

#### 3. Build JerryScript for RIOT-OS

```
# assume you are in harmony folder
cd jerryscript
make -f ./targets/riot-stm32f4/Makefile.riot
```

This will generate the following libraries:
```
/build/bin/release.riotstm32f4/librelease.jerry-core.a
/build/bin/release.riotstm32f4/librelease.jerry-libm.lib.a
```

This will copy one library files to `targets/riot-stm32f4/bin` folder:
```
libjerrycore.a
```

This will create a hex file in the `targets/riot-stm32f4/bin` folder:
```
riot_jerry.elf
```

#### 4. Flashing

```
make -f ./targets/riot-stm32f4/Makefile.riot flash
```

For how to flash the image with other alternative way can be found here:
[Alternative way to flash](https://github.com/RIOT-OS/RIOT/wiki/Board:-STM32F4discovery#alternative-way-to-flash)

#### 5. Cleaning

To clean the build result:
```
make -f ./targets/riot-stm32f4/Makefile.riot clean
```

### 5. Running JerryScript Hello World! example

You may have to press `RESET` on the board after the flash.

You can use `minicom` for terminal program, and if the prompt shows like this:
```
main(): This is RIOT! (Version: ****)
                                     You are running RIOT on a(n) stm32f4discovery board.
                                                                                This board features a(n) stm32f4 MCU.
```
please set `Add Carriage Ret` option by `CTRL-A` > `Z` > `U` at the console, if you're using `minicom`.

Help will provide a list of commands:
```
> help
```

The `test` command will run the test example, which contains the following script code:
```
print ('Hello, World!');
```


### About

This folder contains files to integrate JerryScript with Zephyr RTOS to
run on a number of supported boards (like
[Arduino 101 / Genuino 101](https://www.arduino.cc/en/Main/ArduinoBoard101),
[Zephyr Arduino 101](https://www.zephyrproject.org/doc/board/arduino_101.html)).

### How to build

#### 1. Preface

1. Directory structure

Assume `harmony` as the path to the projects to build.
The folder tree related would look like this.

```
harmony
  + jerryscript
  |  + targets
  |      + zephyr
  + zephyr-project
```

2. Target boards/emulations

Following Zephyr boards were tested: qemu_x86, qemu_cortex_m3, arduino_101,
frdm_k64f.

#### 2. Prepare Zephyr

Follow [this](https://www.zephyrproject.org/doc/getting_started/getting_started.html) page to get
the Zephyr source and configure the environment.

If you just start with Zephyr, you may want to follow "Building a Sample
Application" section in the doc above and check that you can flash your
target board.

Remember to source the Zephyr environment as explained in the zephyr documenation:

```
cd zephyr-project
source zephyr-env.sh

export ZEPHYR_GCC_VARIANT=zephyr
export ZEPHYR_SDK_INSTALL_DIR=<sdk installation directory>
```

#### 3. Build JerryScript for Zephyr

The easiest way is to build and run on a QEMU emulator:

For x86 architecture:

```
make -f ./targets/zephyr/Makefile.zephyr BOARD=qemu_x86 qemu
```

For ARM (Cortex-M) architecture:

```
make -f ./targets/zephyr/Makefile.zephyr BOARD=qemu_cortex_m3 qemu
```

#### 4. Build for Arduino 101

```
# assume you are in harmony folder
cd jerryscript
make -f ./targets/zephyr/Makefile.zephyr BOARD=arduino_101_factory
```

This will generate the following libraries:
```
./build/arduino_101_factory/librelease-cp_minimal.jerry-core.a
./build/arduino_101_factory/librelease-cp_minimal.jerry-libm.lib.a
./build/arduino_101_factory/librelease.external-cp_minimal-entry.a
```

The final Zephyr image will be located here:
```
./build/arduino_101_factory/zephyr/zephyr.strip
```

#### 5. Flashing

Details on how to flash the image can be found here:
[Flashing image](https://www.zephyrproject.org/doc/board/arduino_101.html)
(or similar page for other supported boards).

To be able to use this demo in hardware you will need the serial console
which will be generating output to Pins 0 & 1.

You will need a 3.3v TTL to RS232, please follow the zephyr documentation on it.

Some examples of building the software

```
make -f ./targets/zephyr/Makefile.zephyr BOARD=<board> clean
```

- Not using a Jtag and having a factory stock Arduino 101.
You can follow the Zephyr instructions to flash using the dfu-util command
or use this helper:

```
make -f ./targets/zephyr/Makefile.zephyr BOARD=arduino_101_factory dfu-x86
```

Make sure you have the factory bootloader in your device to use this method or it will not flash.

- Using JTAG

There is a helper function to flash using the JTAG and Flywatter2

![alt tag](docs/arduino_101.jpg?raw=true "Example")

```
make -f ./targets/zephyr/Makefile.zephyr BOARD=arduino_101_factory flash
```

<warning> Careful if you flash the BOARD arduino_101, you will lose the bootloader
and you will have to follow the zephyr documentation to get it back from
the backup we all know you did at the setup. </warning>

#### 6. Serial terminal

Test command line in a serial terminal.

You should see something similar to this:
```
JerryScript build: Aug 12 2016 17:12:55
JerryScript API 1.0
Zephyr version 1.4.0
js>
```

Run the example javascript command test function
```
js> var test=0; for (t=100; t<1000; t++) test+=t; print ('Hi JS World! '+test);
Hi JS World! 494550
```

Try a more complex function:
```
js> function hello(t) {t=t*10;return t}; print("result"+hello(10.5));
```


Docker files
------------

This folder contains docker files that are used in testing nanopb automatically
on various platforms.

By default they take the newest master branch code from github.

To build tests for a single target, use for example:

    docker build ubuntu1804

To build tests for all targets, use:

    ./build_all.sh


Application-layer PULSEv2 Protocols
===================================


#### Apply patch to ESP8266 SDK

As `iram` is quite small to fit all the codes but linker tries to put it there.
To force JerryScript codes to be placed at the `irom` section,
need to change the order and tell the linker as below;

```
diff --git a/ld/eagle.app.v6.common.ld b/ld/eagle.app.v6.common.ld
index caf8e32..dadaceb 100644
--- a/ld/eagle.app.v6.common.ld
+++ b/ld/eagle.app.v6.common.ld
@@ -151,6 +151,21 @@ SECTIONS
   } >dram0_0_seg :dram0_0_bss_phdr
 /* __stack = 0x3ffc8000; */

+  .irom0.text : ALIGN(4)
+  {
+    _irom0_text_start = ABSOLUTE(.);
+    *(.irom0.literal .irom.literal .irom.text.literal .irom0.text .irom.text)
+    *(.literal.* .text.*)
+    _irom0_text_end = ABSOLUTE(.);
+
+    _jerry_text_start = ABSOLUTE(.);
+    *\libjerryentry.a:*(.text*)
+    *\libjerrycore.a:*(.text*)
+    *\libjerrylibm.a:*(.text*)
+    _jerry_text_end = ABSOLUTE(.);
+
+  } >irom0_0_seg :irom0_0_phdr
+
   .text : ALIGN(4)
   {
     _stext = .;
@@ -199,13 +214,6 @@ SECTIONS
     _lit4_end = ABSOLUTE(.);
   } >iram1_0_seg :iram1_0_phdr

-  .irom0.text : ALIGN(4)
-  {
-    _irom0_text_start = ABSOLUTE(.);
-    *(.irom0.literal .irom.literal .irom.text.literal .irom0.text .irom.text)
-    *(.literal.* .text.*)
-    _irom0_text_end = ABSOLUTE(.);
-  } >irom0_0_seg :irom0_0_phdr
 }

 /* get ROM code address */
diff --git a/ld/eagle.app.v6.ld b/ld/eagle.app.v6.ld
index 3e7ec1b..4a9ab5b 100644
--- a/ld/eagle.app.v6.ld
+++ b/ld/eagle.app.v6.ld
@@ -26,7 +26,7 @@ MEMORY
   dport0_0_seg :                       org = 0x3FF00000, len = 0x10
   dram0_0_seg :                        org = 0x3FFE8000, len = 0x14000
   iram1_0_seg :                        org = 0x40100000, len = 0x8000
-  irom0_0_seg :                        org = 0x40240000, len = 0x3C000
+  irom0_0_seg :                        org = 0x40240000, len = 0xB0000
 }

 INCLUDE "../ld/eagle.app.v6.common.ld"
```

Second file is to modify `irom` size so that it can hold all the codes and data.
This can be done by giving another `SPI_SIZE_MAP`.
For now, I'll use this manual modification.

#### Need to get setjmp / longjmp

Extract and copy from the SDK.

```
cd ~/harmony/jerryscript/targets/esp8266/libs
ar -xv $SDK_PATH/lib/libcirom.a lib_a-setjmp.o
```


#### Preparation

##### Accessories

You need,
* 3.3V power supply. You can use bread board power (+5V, +3.3V). I used LM317 like this;
  * Use [LM317](http://www.ti.com/lit/ds/symlink/lm317.pdf)
  * R1 = 330 Ohm, R2 = 545 Ohm (1K + 1.2K in parallel)
  * 5V 2A adaptor
* USB to RS-232 Serial + RS-232 Serial to Digital or USB-to-RS232 TTL converter

#### Toolchain

Reference [Toolchain](https://github.com/esp8266/esp8266-wiki/wiki/Toolchain) page.

I've slightly changed the step to use SDK from Espressif official SDK
(https://github.com/espressif/esp_iot_rtos_sdk)

##### Toolchain:

dependencies for x86
```
sudo apt-get install git autoconf build-essential gperf \
   bison flex texinfo libtool libncurses5-dev wget \
    gawk libc6-dev-i386 python-serial libexpat-dev
sudo mkdir /opt/Espressif
sudo chown $USER /opt/Espressif/
```

dependencies for x64
```
sudo apt-get install git autoconf build-essential gperf \
    bison flex texinfo libtool libncurses5-dev wget \
    gawk libc6-dev-amd64 python-serial libexpat-dev
sudo mkdir /opt/Espressif
sudo chown $USER /opt/Espressif/
```

crosstool-NG
```
cd /opt/Espressif
git clone -b lx106-g++-1.21.0 git://github.com/jcmvbkbc/crosstool-NG.git
cd crosstool-NG
./bootstrap && ./configure --prefix=`pwd` && make && make install
./ct-ng xtensa-lx106-elf
./ct-ng build
```
add path to environment file such as `.profile`
```
PATH=$PWD/builds/xtensa-lx106-elf/bin:$PATH
```

##### Espressif SDK: use Espressif official

```
cd /opt/Esprissif
git clone https://github.com/espressif/ESP8266_RTOS_SDK.git ESP8266_RTOS_SDK.git
ln -s ESP8266_RTOS_SDK.git ESP8266_SDK
cd ESP8266_SDK
git checkout -b jerry a2b413ad2996450fe2f173b6afab243f6e1249aa
```

We use SDK 1.2.0 version which has stdlib.h and others. Latest 1.3.0 version,
as of writing this document, doesn't have it.
(If anyone knows how to use latest version, please add an issue or send a PR.)

set two environment variables such as in .profile
```
export SDK_PATH=/opt/Espressif/ESP8266_SDK
export BIN_PATH=(to output folder path)
```

##### Xtensa libraries and headers:
```
cd /opt/Espressif/ESP8266_SDK
wget -O lib/libhal.a https://github.com/esp8266/esp8266-wiki/raw/master/libs/libhal.a
```

##### ESP image tool
```
cd /opt/Espressif
wget -O esptool_0.0.2-1_i386.deb https://github.com/esp8266/esp8266-wiki/raw/master/deb/esptool_0.0.2-1_i386.deb
sudo dpkg -i esptool_0.0.2-1_i386.deb
```

##### ESP upload tool
```
cd /opt/Espressif
git clone https://github.com/themadinventor/esptool esptool-py
sudo ln -s $PWD/esptool-py/esptool.py crosstool-NG/builds/xtensa-lx106-elf/bin/esptool.py
```

#### Test writing with Blinky example

##### Get the source

found one example that works with SDK V1.2 (which is based on FreeRTOS, as of writing)

* https://github.com/mattcallow/esp8266-sdk/tree/master/rtos_apps/01blinky

##### Compile

Read `2A-ESP8266__IOT_SDK_User_Manual_EN` document in
[this](http://bbs.espressif.com/viewtopic.php?f=51&t=1024) link.

It's configured 2048KB flash
```
BOOT=new APP=1 SPI_SPEED=80 SPI_MODE=QIO SPI_SIZE_MAP=5 make
BOOT=new APP=2 SPI_SPEED=80 SPI_MODE=QIO SPI_SIZE_MAP=5 make
```

or old way... this works not sure this is ok.
```
make BOOT=new APP=0 SPI_SPEED=80 SPI_MODE=QIO SPI_SIZE_MAP=2
```

##### Flashing

* power off ESP8266 board
* connect GPIO0 to GND, connect GPIO2 to VCC
* power on
* write

```
sudo /opt/Espressif/esptool-py/esptool.py \
  --port /dev/ttyUSB0 write_flash \
  0x00000 $SDK_PATH/bin/"boot_v1.4(b1).bin" \
  0x01000 $BIN_PATH/upgrade/user1.2048.new.5.bin \
  0x101000 $BIN_PATH/upgrade/user2.2048.new.5.bin \
  0x3FE000 $SDK_PATH/bin/blank.bin \
  0x3FC000 $SDK_PATH/bin/esp_init_data_default.bin
```
_change `/dev/ttyUSB1` to whatever you have._

or the old way...this works not sure this is ok.
```
cd $BIN_PATH
sudo /opt/Espressif/esptool-py/esptool.py \
--port /dev/ttyUSB0 write_flash \
0x00000 eagle.flash.bin 0x40000 eagle.irom0text.bin
```

* power off
* disconnect GPIO0 so that it is floating
* connect GPIO2 with serial of 470 Ohm + LED and to GND
* power On

LED should blink on and off every second


# Health Algorithms
## Step Counting
The step counting algorithm uses input from the accelerometer sensor to detect when the user is walking and how many steps have been taken. The accelerometer measures acceleration in each of 3 axes: x, y, and z. A perfectly still watch resting flat on a table will have 1G (1 “Gravity”) of acceleration in the z direction (due to gravity) and 0G’s in both the x and y axes. If you tilt the watch on its side for example, the z reading will go to 0 and then either x or y will show +/-1G (depending on which of the 4 sides you tilt it to). During watch movement, the x, y, and z readings will vary over time due to the watch’s changing orientation to gravity as well as the acceleration of the watch when it changes direction or speed. The pattern of these variations in the accelerometer readings over time can be used to detect if, and how fast, the user is stepping.

There are generally two dominant signals that show up in the accelerometer readings when a person is walking or running. The first is the signal due to your feet hitting the ground. This signal shows up as a spike in the accelerometer readings each time a foot hits the ground and will be more or less pronounced depending on the cushioning of your shoes, the type of flooring, etc. Another signal that can show up is from the arm swinging motion, and the strength of this will vary depending on the user’s walking style, whether their hand is in their pocket or not, whether they are carrying something, etc.

Of these two signals, the foot fall one is the most reliable since a user will not always be swinging their arms when walking. The goal of the step tracking algorithm is to isolate and detect this foot fall signal, while not getting confused by other signals (arm swings, random arm movements, etc.).

An overall outline of the approach taken by the stepping algorithm (glossing over the details for now) is as follows:

1. Separate the accelerometer sensor readings into 5 second epochs.
2. For each 5 second epoch, compute an FFT (Fast Fourier Transform) to get the energy of the signal at different frequencies (called the _spectral density_)
3. Examine the FFT output using a set of heuristics to identify the foot fall signal (if present) and its frequency.
4. The frequency of the foot fall signal (if present) is outputted as the number of steps taken in that epoch.

As an example, if the FFT of a 5 second epoch shows a significant amount of foot fall signal at a frequency of 2Hz, we can assume the person has walked 10 steps (2Hz x 5 seconds) in that epoch.

### Example Data
The following figure shows an example of the raw accelerometer data of a five-second epoch when a user is walking 10 steps. The x, y, and z axis signals are each shown in a different color. In this plot, there is a fairly evident five-cycle rhythm in the red and green axes, which happens to be the arm swing signal (for every 2 steps taken, only 1 full arm swing cycle occurs). The ten-cycle foot fall signal however is difficult to see in this particular sample because the arm swing is so strong.

![Raw accelerometer data](raw_accel_5s.png)

The spectral density of that same walk sample, showing the amount of energy present at different frequencies, is shown in the following figure (this particular plot was generated from a sample longer than 5 seconds, so will be less noisy than any individual 5 second epoch). Here, the spectral density of the x, y, and z axes as well as the combined signal are each plotted in a different color. The _combined_ signal is computed as the magnitude of the x, y, and z spectral density at each frequency:
	combined[f] = sqrt(x[f]^2 + y[f]^2 + z[f]^2)

_Note that the y axis on this plot is not simply Power, but rather “Power / Average Power”, where “Average Power” is the average power of that particular signal._

![Spectral Density](spectial_density.png)

You can see in the above spectral density plot that the dominant frequency in this example is 1Hz, corresponding to the 5 arm swings that occurred in these 5 seconds.

There are also several smaller peaks at the following frequencies:

- ~.25Hz: non-stepping signal, most likely random arm movements
- 2Hz: stepping frequency + 2nd harmonic of arm swing
- 3Hz: 3rd harmonic of arm swing
- 4Hz: 2nd harmonic of steps + 4th harmonic of arm swing
- 5Hz: 5th harmonic of arm swing
- 8Hz: 4th harmonic of steps

The logic used to pull out and identify the stepping signal from the spectral density output will be described later, but first we have to introduce the concept of VMC, or Vector Magnitude Counts.

### VMC

VMC, or Vector Magnitude Counts, is a measure of the overall amount of movement in the watch over time. When the watch is perfectly still, the VMC will be 0 and greater amounts of movement result in higher VMC numbers. Running, for example results in a higher VMC than walking.

The VMC computation in Pebble Health was developed in conjunction with the Stanford Wearables lab and has been calibrated to match the VMC numbers produced by the [Actigraph](http://www.actigraphcorp.com/product-category/activity-monitors/) wrist-worn device. The Actigraph is commonly used today for medical research studies. The Stanford Wearables lab will be publishing the VMC computation used in the Pebble Health algorithm and this transparency of the algorithm will enable the Pebble to be used for medical research studies as well.

VMC is computed using the formula below. Before the accelerometer readings are incorporated into this computation however, each axis’ signal is run through a bandpass filter with a design of 0.25Hz to 1.75Hz.

![](vmc_formula.png)

The following pseudo code summarizes the VMC calculation for N samples worth of data in each axis. The accelerometer is sampled at 25 samples per second in the Health algorithm, so the VMC calculation for 1 second’s worth of data would process 25 samples from each axis. The _bandpass\_filter_ method referenced in this pseudo code is a convolution filter with a frequency response of 0.25 to 1.75Hz:

	for each axis in x, y, z:
	  axis_sum[axis] = 0
	  for each sample in axis from 0 to N:
	    filtered_sample = bandpass_filter(sample,
	                                      filter_state)
	    axis_sum[axis] += abs(filtered_sample)

	VMC = scaling_factor * sqrt(axis_sum[x]^2
	                            + axis_sum[y]^2
	                            + axis_sum[z]^2)

The step algorithm makes use of a VMC that is computed over each 5-second epoch. In addition to this 5-second VMC, the algorithm also computes a VMC over each minute of data. It saves this 1-minute VMC to persistent storage and sends it to data logging as well so that it will eventually get pushed to the phone and saved to a server. The 1-minute VMC values stored in persistent storage can be accessed by 3rd party apps through the Health API. It is these 1-minute VMC values that are designed to match the Actigraph computations and are most useful to medical researcher studies.

### Step Identification

As mentioned above, accelerometer data is processed in chunks of 5 seconds (one epoch) at a time. For each epoch, we use the combined spectral density (FFT output) and the 5-second VMC as inputs to the step identification logic.

#### Generating the FFT output
The accelerometer is sampled at 25Hz, so each 5 second epoch comprises 125 samples in each axis. An FFT must have an input width which is a power of 2, so for each axis, we subtract the mean and then 0-extend to get 128 samples before computing the FFT for that axis.

An FFT of a real signal with 128 samples produces 128 outputs that represent 64 different frequencies. For each frequency, the FFT produces a real and an imaginary component (thus the 128 outputs for 64 different frequencies). The absolute value of the real and imaginary part denote the amplitude at a particular frequency, while the angle represents the phase of that frequency. It is the amplitude of each frequency that we are interested in, so we compute sqrt(real^2 + imag^2) of each frequency to end up with just 64 outputs.

Once the 64 values for each of the 3 axes have been computed, we combine them to get the overall energy at each frequency as follows:

	for i = 0 to 63:
	  energy[i] = sqrt(amp_x[i]^2 + amp_y[i]^2 + amp_z[i]^2)

In this final array of 64 elements, element 0 represents the DC component (0 Hz), element 1 represents a frequency of 1 cycle per epoch (1 / 5s = 0.2Hz), element 2 represents a frequency of 2 cycles per epoch (2 / 5s = 0.4Hz), etc. If the user is walking at a rate of 9 steps every 5 seconds, then a spike will appear at index 9 in this array (9 / 5s = 1.8Hz).

As an example, the following shows the FFT output of a user walking approximately 9 steps in an epoch (with very little arm swing):

![FFT of stepping epoch, 9 steps](fft_walking.png)

#### Determining the stepping frequency

Once the FFT output and VMC have been obtained, we search for the most likely stepping frequency. The naive approach is to simply locate the frequency with the highest amplitude among all possible stepping frequencies. That would work fine for the example just shown above where there is a clear peak at index 9 of the FFT, which happens to be the stepping frequency.

However, for some users the arm swinging signal can be as large or larger than the stepping signal, and happens to be at half the stepping frequency. If a user is walking at a quick pace, the arm swinging signal could easily be misinterpreted as the stepping signal of a slow walk. The following is the FFT of such an example. The stepping signal shows up at indices 9 and 10, but there is a larger peak at the arm-swing frequency at index 5.

![Stepping epoch with large arm-swing component](fft_arm_swing.png "FFT of arm-swing walk")

To deal with these possible confusions between arm-swing and stepping signals, the VMC is used to narrow down which range of frequencies the stepping is likely to fall in. Based on the VMC level, we search one of three different ranges of frequencies to find which frequency has the most energy and is the likely stepping frequency. When the VMC is very low, we search through a range of frequencies that represent a slow walk, and for higher VMCs we search through ranges of frequencies that represent faster walks or runs.

Once we find a stepping frequency candidate within the expected range, we further refine the choice by factoring in the harmonics of the stepping/arm swinging. Occasionally, a max signal in the stepping range does not represent the actual stepping rate - it might be off by one or two indices due to noise in the signal, or it might be very close in value to the neighboring frequency, making it hard to determine which is the optimal one to use. This is evident in the arm-swinging output shown above where the energy at index 9 is very close to the energy at index 10.

As mentioned earlier, we often see significant energy at the harmonics of both the arm-swinging and the stepping frequency. A harmonic is an integer multiple of the fundamental frequency (i.e. a stepping frequency of 2 Hz will result in harmonics at 4Hz, 6Hz, 8Hz, etc.). To further refine the stepping frequency choice, we evaluate all possible stepping frequencies near the first candidate (+/- 2 indices on each side) and add in the energy of the harmonics for each. For each evaluation, we add up the energy of that stepping frequency, the arm energy that would correspond to that stepping frequency (the energy at half the stepping frequency), and the 2nd thru 5th harmonics of both the stepping and arm-swinging frequencies. Among these 5 different candidate stepping frequencies, we then choose the one that ended up with the most energy overall.

At the end of this process, we have the most likely stepping frequency, **if** the user is indeed walking. The next step is to determine whether or not the user is in fact walking or not.

#### Classifying step vs non-step epochs

In order to classify an epoch as walking or non-walking, we compute and check a number of metrics from the FFT output.

The first such metric is the _walking score_ which is the sum of the energy in the stepping related frequencies (signal energy) divided by the sum of energy of all frequencies (total energy). The signal energy includes the stepping frequency, arm-swing frequency, and each of their harmonics. If a person is indeed walking, the majority of the signal will appear at these signal frequencies, yielding a high walking score.

The second constraint that the epoch must pass is that the VMC must be above a _minimum stepping VMC_ threshold. A higher threshold is used if the detected stepping rate is higher.

The third constraint that the epoch must pass is that the amount of energy in the very low frequency components must be relatively low. To evaluate this constraint, the amount of energy in the low frequency components (indices 0 through 4) is summed and then divided by the signal energy (computed above). If this ratio is below a set _low frequency ratio_ threshold, the constraint is satisfied. The example below is typical of many epochs that are non-stepping epochs - a large amount of the energy appears in the very low frequency area.

![Non-stepping epoch](fft_non_walk.png)

The fourth and final constraint that the epoch must pass is that the energy in the high frequencies must be relatively low. To evaluate this constraint, the amount of energy in the high frequency components (index 50 and above) is summed and then divided by the signal energy. If this ratio is below a set _high frequency ratio_ threshold, the constraint is satisfied. This helps to avoid counting driving epochs as stepping epochs. In many instances, the vibration of the engine in a car will show up as energy at these high frequencies as shown in the following diagram.

![Driving epoch](fft_driving.png)

#### Partial Epochs

If the user starts or ends a walk in the middle of an epoch, the epoch will likely not pass the checks for a full fledged stepping epoch and these steps will therefore not get counted. To adjust for this undercounting, the algorithm introduces the concept of _partial epochs_.

The required _walking score_ and _minimum VMC_ are lower for a partial epoch vs. a normal epoch and there are no constraints on the low or high frequency signal ratios. To detect if an epoch is a _partial epoch_ we only check that the _walking score_ is above the _partial epoch walking score_ threshold and that the VMC is above the _partial epoch minimum VMC_ threshold.

If we detect a partial epoch, and either the prior or next epoch were classified as a stepping epoch, we add in half the number of steps that were detected in the adjacent stepping epoch. This helps to average out the undercounting that would normally occur at the start and end of a walk. For a very short walk that is less than 2 epochs long though, there is still a chance that no steps at all would be counted.

----

## Sleep Tracking

The sleep tracking algorithm uses the minute-level VMC values and minute-level average orientation of the watch to determine if/when the user is sleeping and whether or not the user is in “restful” sleep.

The minute-level VMC was described above. It gives a measure of the overall amount of movement seen by the watch in each minute.

The average orientation is a quantized (currently 8 bits) indication of the 3-D angle of the watch. It is computed once per minute based on the average accelerometer reading seen in each of the 3 axes. The angle of the watch in the X-Y plane is computed and quantized into the lower 4 bits and the angle of that vector with the Z-axis is then quantized and stored in the upper 4 bits.

### Sleep detection

The following discussion uses the term _sleep minute_. To determine if a minute is a _sleep minute_, we perform a convolution of the VMC values around that minute (using the 4 minutes immediately before and after the given minute) to generate a _filtered VMC_ and compare the _filtered VMC_ value to a threshold. If the result is below a determined sleep threshold, we count it as a _sleep minute_.

A rough outline of the sleep algorithm is as follows.

1. Sleep is entered if there are at least 5 _sleep minutes_ in a row.
2. Sleep continues until there are at least 11 non-_sleep minutes_ in a row.
3. If there were at least 60 minutes between the above sleep enter and sleep exit times, it is counted as a valid sleep session.

There are some exceptions to the above rules however:

- After sleep has been entered, if we see any minute with an exceptionally high _filtered VMC_, we end the sleep session immediately.
- If it is early in the sleep session (the first 60 minutes), we require 14 non-_sleep minutes_ in a row to consider the user as awake instead of 11.
- If at least 80% of the minutes have slight movement in them (even if each one is not high enough to make it a non-_sleep minute_), we consider the user awake.
- If we detect that the watch was not being worn during the above time (see below), we invalidate the sleep session.

#### Restful sleep

Once we detect a sleep session using the above logic, we make another pass through that same data to see if there are any periods within that session that might be considered as _restful sleep_.

A _restful sleep minute_ is a minute where the _filtered VMC_ is below the _restful sleep minute_ threshold (this is lower than the normal _sleep minute_ threshold).

1. Restful sleep is entered if there are at least 20 _restful sleep minutes_ in a row.
2. Restful sleep continues until there is at least 1 minute that is not a _restful sleep minute_.

### Detecting not-worn

Without some additional logic in place, the above rules would think a user is in a sleep session if the watch is not being worn. This is because there would be no movement and the VMC values would all be 0, or at least very low.

Once we detect a possible sleep session, we run that same data through the “not-worn” detection logic to determine if the watch was not being worn during that time. This is a set of heuristics that are designed to distinguish not-worn from sleep.

The following description uses the term _not worn minute_. A _not worn minute_ is a minute where **either** of the following is true:

- The VMC (the raw VMC, not _filtered VMC_) is below the _not worn_ threshold and the average orientation is same as it was the prior minute
- The watch is charging

If we see **both** of the following, we assume the watch is not being worn:

1. There are at least 100 _not worn_ minutes in a row in the sleep session
2. The _not worn_ section from #1 starts within 20 minutes of the start of the candidate sleep session and ends within 10 minutes of the end of the candidate sleep session.

The 100 minute required run length for _not worn_ might seem long, but it is not uncommon to see valid restful sleep sessions for a user that approach 100 minutes in length.

The orientation check is useful for situations where a watch is resting on a table, but encounters an occasional vibration due to floor or table shaking. This vibration shows up as a non-zero VMC and can look like the occasional movements that are normal during sleep. During actual sleep however, it is more likely that the user will change positions and end up at a different orientation on the next minute.

----

## System Integration

The following sections discuss how the step and sleep tracking algorithms are integrated into the firmware.

### Code organization

The core of the Health support logic is implemented in the activity service, which is in the `src/fw/services/normal/activity` directory. The 3rd party API, which calls into the activity service, is implemented in `src/fw/applib/health_service.c.`

The activity service implements the step and sleep algorithms and all of the supporting logic required to integrate the algorithms into the system. It has the following directory structure:

	src/fw/services/normal/activity
	  activity.c
	  activity_insights.c
	  kraepelin/
	    kraepelin_algorithm.c
	    activity_algorithm_kraepelin.c

- **activity.c** This is the main module for the activity service. It implements the API for the activity service and the high level glue layer around the underlying step and sleep algorithms. This module contains only algorithm agnostic code and should require minimal changes if an alternative implementation for step or sleep tracking is incorporated in the future.
- **activity\_insights.c** This module implements the logic for generating Health timeline pins and notifications.
- **kraepelin** This subdirectory contains the code for the Kraepelin step and sleep algorithm, which is the name given to the current set of algorithms described in this document. This logic is broken out from the generic interface code in activity.c to make it easier to substitute in alternative algorithm implementations in the future if need be.
- **kraepelin\_algorithm.c** The core step and sleep algorithm code. This module is intended to be operating system agnostic and contains minimal calls to external functions. This module originated from open source code provided by the Stanford Wearables Lab.
- **kraepelin/activity\_algorightm\_kraepelin.c** This module wraps the core algorithm code found in `kraepelin_algorithm.c` to make it conform to the internal activity service algorithm API expected by activity.c. An alternative algorithm implementation would just need to implement this same API in order for it to be accessible from `activity.c`. This modules handles all memory allocations, persistent storage management, and other system integration functions for the raw algorithm code found in kraepelin\_algorithm.c.

The 3rd party Health API is implemented in `src/fw/applib/health_service.c`. The `health_service.c` module implements the “user land” logic for the Health API and makes calls into the activity service (which runs in privileged mode) to access the raw step and sleep data.

### Step Counting

The `activity.c` module asks the algorithm implementation `activity_algorithm_kraepelin.c` what accel sampling rate it requires and handles all of the logic required to subscribe to the accel service with that sampling rate. All algorithmic processing (both step and sleep) in the activity service is always done from the KernelBG task, so `activity.c` subscribes to the accel service from a KernelBG callback and provides the accel service a callback method which is implemented in `activity.c`.

When `activity.c’s` accel service callback is called, it simply passes the raw accel data onto the underlying algorithm’s accel data handler implemented `activity_algorithm_kraepelin.c`. This handler in turn calls into the core algorithm code in `kraepelin_algorithm.c` to execute the raw step algorithm code and increments total steps by the number of steps returned by that method. Since the step algorithm in `kraepelin_algorithm.c` is based on five-second epochs and the accel service callback gets called once a second (25 samples per second), the call into `kraepelin_algorithm.c` will only return a non-zero step count value once every 5 times it is called.

Whenever a call is made to `activity.c` to get the total number of steps accumulated so far, `activity.c` will ask the `activity_algorithm_kraepelin.c` module for that count. The `activity_algorithm_kraepelin.c` module maintains that running count directly and returns it without needing to call into the raw algorithm code.

At midnight of each day, `activity.c` will make a call into `activity_algorithm_kraeplin.c` to reset the running count of steps back to 0.

### Sleep processing

For sleep processing, the `activity_algorithm_kraepelin.c` module has a much bigger role than it does for step processing. The core sleep algorithm in `kraepelin_algorithm.c` simply expects an array of VMC and average orientation values (one each per minute) and from that it identifies where the sleep sessions are. It is the role of `activity_algorithm_kraepelin.c` to build up this array of VMC values for the core algorithm and it does this by fetching the stored VMC and orientation values from persistent storage. The `activity_algorithm_kraepelin.c` module includes logic that periodically captures the VMC and orientation for each minute from the core algorithm module and saves those values to persistent storage for this purpose as well as for retrieval by the 3rd party API call that can be used by an app or worker to fetch historical minute-level values.

Currently, `activity.c` asks `activity_algorithm_kraepelin.c` to recompute sleep every 15 minutes. When asked to recompute sleep, `activity_algorithm_kraepelin.c` fetches the last 36 hours of VMC and orientation data from persistent storage and passes that array of values to the core sleep algorithm. When we compute sleep for the current day, we include all sleep sessions that *end* after midnight of the current day, so they may have started sometime before midnight. Including 36 hours of minute data means that, if asked to compute sleep at 11:59pm for example, we can go as far back as a sleep session that started at 6pm the prior day.

To keep memory requirements to a minimum, we encode each minute VMC value into a single byte for purposes of recomputing sleep. The raw VMC values that we store in persistent storage are 16-bit values, so we take the square root of each 16-bit value to compress it into a single byte. The average orientation is also encoded as a single byte. The 36 hours of minute data therefore requires that an array of 36 \* 60 \* 2 (4320) bytes be temporarily allocated and passed to the core sleep algorithm logic.

The core sleep logic in `kraepelin_algorithm.c` does not have any concept of what timestamp corresponds to each VMC value in the array, it only needs to describe the sleep sessions in terms of indices into the array. It is the role of `activity_algorithm_kraepelin.c` to translate these indices into actual UTC time stamps for use by the activity service.

## Algorithm Development and Testing

There is a full set of unit tests in `tests/fw/services/activity` for testing the step and sleep algorithms. These tests run captured sample data through the `kraepelin_algorithm.c` algorithm code to verify the expected number of steps or sleep sessions.

### Step Algorithm Testing

For testing the step algorithm, raw accel data is fed into the step algorithm. This raw accel data is stored in files as raw tuples of x, y z, accelerometer readings and can be found in the `tests/fixtures/activity/step_samples` directory.

Although these files have the C syntax, they are not compiled but are read in and parsed by the unit tests at run-time. Each sample in each file contains meta-data that tells the unit test the expected number of steps for that sample, which is used to determine if the test passes or not.

To capture these samples, the activity service has a special mode that can be turned on for raw sample capture and the `Activity Demo` app has an item in its debug menu for turning on this mode. When this mode is turned on, the activity service saves raw samples to data logging, and at the same time, also captures the raw sample data to the Pebble logs as base64 encoded binary data. Capturing the accel data to the logs makes it super convenient to pull that data out of the watch simply by issuing a support request from the mobile app.

The `tools/activity/parse_activity_data_logging_records.py` script can be used to parse the raw accel samples out of a log file that was captured as part of a support request or from a binary file containing the data logging records captured via data logging. This tool outputs a text file, in C syntax, that can be used directly by the step tracking unit tests.

The unit test that processes all of the step samples in `tests/fixtures/activity/step_samples` insures that the number of steps computed by the algorithm for each sample is within the allowed minimum and maximum for that sample (as defined by the meta data included in each sample file). It also computes an overall error amount across all sample files and generates a nice summary report for reference purposes. When tuning the algorithm, these summary reports can be used to easily compare results for various potential changes.

### Sleep Algorithm Testing

For testing the sleep algorithm, minute-by-minute VMC values are fed into the algorithm code. The set of sample sleep files used by the unit tests are found in the `tests/fixtures/activity/sleep_samples` directory. As is the case for the step samples, these files are parsed by the unit tests at run-time even though they are in C syntax.

To capture these samples, the activity service has a special call that will result in a dump of the contents of the last 36 hours of minute data to the Pebble logs. The `Activity Demo` app has an item in its debug menu for triggering this call. When this call is made, the activity service will fetch the last 36 hours of minute data from persistent storage, base64 encode it, and put it into the Pebble logs so that it can be easily retrieved using a support request from the mobile app.

As is the case for step data, the `tools/activity/parse_activity_data_logging_records.py` script can also be used to extract the minute data out of a support request log file and will in turn generate a text file that can be directly parsed by the sleep algorithm unit tests.

Each sleep sample file contains meta data in it that provides upper and lower bounds for each of the sleep metrics that can be computed by the algorithm (total amount of sleep, total amount of restful sleep, sleep start time, sleep end time, etc.). These metrics are checked by the unit tests to determine if each sample passes.

Note that the minute-by-minute VMC values can always be captured up to 36 hours **after** a sleep issue has been discovered on the watch since the watch is always storing these minute statistics in persistent storage. In contrast, turning on capture of raw accel data for a step algorithm issue must be done before the user starts the activity since capturing raw accel data is too expensive (memory and power-wise) to leave on all the time.


# Welcome Contributors

We welcome contributions to enhance DeepClaude's capabilities and improve its performance. To report bugs, create a [GitHub issue](https://github.com/getasterisk/deepclaude/issues).

> Before contributing, read through the existing issues and pull requests to see if someone else is already working on something similar. That way you can avoid duplicating efforts.

To contribute, please follow these steps:

1. Fork the DeepClaude repository on GitHub.
2. Create a new branch for your feature or bug fix.
3. Make your changes and ensure that the code passes all tests.
4. Submit a pull request describing your changes and their benefits.

### Pull Request Guidelines
When submitting a pull request, please follow these guidelines:

1. **Title**: please include following prefixes:
   - `Feature:` for new features
   - `Fix:` for bug fixes
   - `Docs:` for documentation changes
   - `Refactor:` for code refactoring
   - `Improve:` for performance improvements
   - `Other:` for other changes

   for example:
      - `Feature: added new feature to the code`
      - `Fix: fixed the bug in the code`

2. **Description**: Provide a clear and detailed description of your changes in the pull request. Explain the problem you are solving, the approach you took, and any potential side effects or limitations of your changes.
3. **Documentation**: Update the relevant documentation to reflect your changes. This includes the README file, code comments, and any other relevant documentation.
4. **Dependencies**: If your changes require new dependencies, ensure that they are properly documented and added to the `Cargo.toml` or `package.json` files.
5. if the pull request does not meet the above guidelines, it may be closed without merging.

**Note**: Please ensure that you have the latest version of the code before creating a pull request. If you have an existing fork, just sync your fork with the latest version of the DeepClaude repository.

Please adhere to the coding conventions, maintain clear documentation, and provide thorough testing for your contributions.

CONTRIBUTING.md

A high-performance LLM inference API and Chat UI that integrates DeepSeek R1's CoT reasoning traces with Anthropic Claude models.


MIT License

Copyright (c) 2025 Mufeed VH  / Asterisk

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

LICENSE.md


DeepClaude 🐬🧠

Harness the power of DeepSeek R1's reasoning and Claude's creativity and code generation capabilities with a unified API and chat interface.

[![GitHub license](https://img.shields.io/github/license/getasterisk/deepclaude)](https://github.com/getasterisk/deepclaude/blob/main/LICENSE.md)
[![Rust](https://img.shields.io/badge/rust-v1.75%2B-orange)](https://www.rust-lang.org/)
[![API Status](https://img.shields.io/badge/API-Stable-green)](https://deepclaude.asterisk.so)

[Getting Started](#getting-started) •
[Features](#features) •
[API Usage](#api-usage) •
[Documentation](#documentation) •
[Self-Hosting](#self-hosting) •
[Contributing](#contributing)

## Table of Contents
- [Overview](#overview)
- [Features](#features)
- [Why R1 + Claude?](#why-r1--claude)
- [Getting Started](#getting-started)
  - [Prerequisites](#prerequisites)
  - [Installation](#installation)
  - [Configuration](#configuration)
- [API Usage](#api-usage)
  - [Basic Example](#basic-example)
  - [Streaming Example](#streaming-example)
- [Configuration Options](#configuration-options)
- [Self-Hosting](#self-hosting)
- [Security](#security)
- [Contributing](#contributing)
- [License](#license)
- [Acknowledgments](#acknowledgments)

## Overview

DeepClaude is a high-performance LLM inference API that combines DeepSeek R1's Chain of Thought (CoT) reasoning capabilities with Anthropic Claude's creative and code generation prowess. It provides a unified interface for leveraging the strengths of both models while maintaining complete control over your API keys and data.

## Features

🚀 **Zero Latency** - Instant responses with R1's CoT followed by Claude's response in a single stream, powered by a high-performance Rust API

🔒 **Private & Secure** - End-to-end security with local API key management. Your data stays private

⚙ **Highly Configurable** - Customize every aspect of the API and interface to match your needs

🌟 **Open Source** - Free and open-source codebase. Contribute, modify, and deploy as you wish

🤖 **Dual AI Power** - Combine DeepSeek R1's reasoning with Claude's creativity and code generation

🔑 **Managed BYOK API** - Use your own API keys with our managed infrastructure for complete control

## Why R1 + Claude?

DeepSeek R1's CoT trace demonstrates deep reasoning to the point of an LLM experiencing "metacognition" - correcting itself, thinking about edge cases, and performing quasi Monte Carlo Tree Search in natural language.

However, R1 lacks in code generation, creativity, and conversational skills. Claude 3.5 Sonnet excels in these areas, making it the perfect complement. DeepClaude combines both models to provide:

- R1's exceptional reasoning and problem-solving capabilities
- Claude's superior code generation and creativity
- Fast streaming responses in a single API call
- Complete control with your own API keys

## Getting Started

### Prerequisites

- Rust 1.75 or higher
- DeepSeek API key
- Anthropic API key

### Installation

1. Clone the repository:
```bash
git clone https://github.com/getasterisk/deepclaude.git
cd deepclaude
```

2. Build the project:
```bash
cargo build --release
```

### Configuration

Create a `config.toml` file in the project root:

```toml
[server]
host = "127.0.0.1"
port = 3000

[pricing]
# Configure pricing settings for usage tracking
```

## API Usage

See [API Docs](https://deepclaude.chat)

### Basic Example

```python
import requests

response = requests.post(
    "http://127.0.0.1:1337/",
    headers={
        "X-DeepSeek-API-Token": "<YOUR_DEEPSEEK_API_KEY>",
        "X-Anthropic-API-Token": "<YOUR_ANTHROPIC_API_KEY>"
    },
    json={
        "messages": [
            {"role": "user", "content": "How many 'r's in the word 'strawberry'?"}
        ]
    }
)

print(response.json())
```

### Streaming Example

```python
import asyncio
import json
import httpx

async def stream_response():
    async with httpx.AsyncClient() as client:
        async with client.stream(
            "POST",
            "http://127.0.0.1:1337/",
            headers={
                "X-DeepSeek-API-Token": "<YOUR_DEEPSEEK_API_KEY>",
                "X-Anthropic-API-Token": "<YOUR_ANTHROPIC_API_KEY>"
            },
            json={
                "stream": True,
                "messages": [
                    {"role": "user", "content": "How many 'r's in the word 'strawberry'?"}
                ]
            }
        ) as response:
            response.raise_for_status()
            async for line in response.aiter_lines():
                if line:
                    if line.startswith('data: '):
                        data = line[6:]
                        try:
                            parsed_data = json.loads(data)
                            if 'content' in parsed_data:
                                content = parsed_data.get('content', '')[0]['text']
                                print(content, end='',flush=True)
                            else:
                                print(data, flush=True)
                        except json.JSONDecodeError:
                            pass

if __name__ == "__main__":
    asyncio.run(stream_response())
```

## Configuration Options

The API supports extensive configuration through the request body:

```json
{
    "stream": false,
    "verbose": false,
    "system": "Optional system prompt",
    "messages": [...],
    "deepseek_config": {
        "headers": {},
        "body": {}
    },
    "anthropic_config": {
        "headers": {},
        "body": {}
    }
}
```

## Self-Hosting

DeepClaude can be self-hosted on your own infrastructure. Follow these steps:

1. Configure environment variables or `config.toml`
2. Build the Docker image or compile from source
3. Deploy to your preferred hosting platform

## Security

- No data storage or logged
- BYOK (Bring Your Own Keys) architecture
- Regular security audits and updates

## Contributing

We welcome contributions! Please see our [Contributing Guidelines](CONTRIBUTING.md) for details on:

- Code of Conduct
- Development process
- Submitting pull requests
- Reporting issues

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE.md) file for details.

## Acknowledgments

DeepClaude is a free and open-source project by [Asterisk](https://asterisk.so/). Special thanks to:

- DeepSeek for their incredible R1 model
- Anthropic for Claude's capabilities
- The open-source community for their continuous support

---

Made with ❤ by Asterisk

README.md


This is a [Next.js](https://nextjs.org) project bootstrapped with [`create-next-app`](https://nextjs.org/docs/app/api-reference/cli/create-next-app).

## Getting Started

First, run the development server:

```bash
npm run dev
# or
yarn dev
# or
pnpm dev
# or
bun dev
```

Open [http://localhost:3000](http://localhost:3000) with your browser to see the result.

You can start editing the page by modifying `app/page.tsx`. The page auto-updates as you edit the file.

This project uses [`next/font`](https://nextjs.org/docs/app/building-your-application/optimizing/fonts) to automatically optimize and load [Geist](https://vercel.com/font), a new font family for Vercel.

## Learn More

To learn more about Next.js, take a look at the following resources:

- [Next.js Documentation](https://nextjs.org/docs) - learn about Next.js features and API.
- [Learn Next.js](https://nextjs.org/learn) - an interactive Next.js tutorial.

You can check out [the Next.js GitHub repository](https://github.com/vercel/next.js) - your feedback and contributions are welcome!

## Deploy on Vercel

The easiest way to deploy your Next.js app is to use the [Vercel Platform](https://vercel.com/new?utm_medium=default-template&filter=next.js&utm_source=create-next-app&utm_campaign=create-next-app-readme) from the creators of Next.js.

Check out our [Next.js deployment documentation](https://nextjs.org/docs/app/building-your-application/deploying) for more details.

frontend/README.md


# g1: Using Llama-3.1 70b on Groq to create o1-like reasoning chains

[Video Demo](https://github.com/user-attachments/assets/db2a221f-f8eb-48c3-b5a7-8399c6300243)

This is an early prototype of using prompting strategies to improve the LLM's reasoning capabilities through o1-like reasoning chains. This allows the LLM to "think" and solve logical problems that usually otherwise stump leading models. Unlike o1, all the reasoning tokens are shown, and the app uses an open source model.

g1 is experimental and being open sourced to help inspire the open source community to develop new strategies to produce o1-like reasoning. This experiment helps show the power of prompting reasoning in visualized steps, not a comparison to or full replication of o1, which uses different techniques. OpenAI's o1 is instead trained with large-scale reinforcement learning to reason using Chain of Thought, achieving state-of-the-art performance on complex PhD-level problems.

g1 demonstrates the potential of prompting alone to overcome straightforward LLM logic issues like the Strawberry problem, allowing existing open source models to benefit from dynamic reasoning chains and an improved interface for exploring them.

### How it works

g1 powered by Llama3.1-70b creates reasoning chains, in principle a dynamic Chain of Thought, that allows the LLM to "think" and solve some logical problems that usually otherwise stump leading models.

At each step, the LLM can choose to continue to another reasoning step, or provide a final answer. Each step is titled and visible to the user. The system prompt also includes tips for the LLM. There is a full explanation under Prompt Breakdown, but a few examples are asking the model to “include exploration of alternative answers” and “use at least 3 methods to derive the answer”.

The reasoning ability of the LLM is therefore improved through combining Chain-of-Thought with the requirement to try multiple methods, explore alternative answers, question previous draft solutions, and consider the LLM’s limitations. This alone, without any training, is sufficient to achieve ~70% accuracy on the Strawberry problem (n=10, "How many Rs are in strawberry?"). Without prompting, Llama-3.1-70b had 0% accuracy and ChatGPT-4o had 30% accuracy.

### Examples

> [!IMPORTANT]
> g1 is not perfect, but it can perform significantly better than LLMs out-of-the-box. From initial testing, g1 accurately solves simple logic problems 60-80% of the time that usually stump LLMs. However, accuracy has yet to be formally evaluated. See examples below.

##### How many Rs are in strawberry?

Prompt: How many Rs are in strawberry?

Result:

![Strawberry example](examples/strawberry.png)

---

Prompt: Which is larger, .9 or .11?

Result:

![0.9 or 0.11 example](examples/math.png)

### Quickstart

To use the Streamlit UI, follow these instructions:

~~~
python3 -m venv venv
~~~

~~~
source venv/bin/activate
~~~

~~~
pip3 install -r requirements.txt
~~~

~~~
export GROQ_API_KEY=gsk...
~~~

~~~
streamlit run app.py
~~~

---

Alternatively, follow these additional instructions to use the Gradio UI:

~~~
cd gradio
~~~

~~~
pip3 install -r requirements.txt
~~~

~~~
python3 app.py
~~~

### Prompting Strategy

The prompt is as follows:

```
You are an expert AI assistant that explains your reasoning step by step. For each step, provide a title that describes what you're doing in that step, along with the content. Decide if you need another step or if you're ready to give the final answer. Respond in JSON format with 'title', 'content', and 'next_action' (either 'continue' or 'final_answer') keys. USE AS MANY REASONING STEPS AS POSSIBLE. AT LEAST 3. BE AWARE OF YOUR LIMITATIONS AS AN LLM AND WHAT YOU CAN AND CANNOT DO. IN YOUR REASONING, INCLUDE EXPLORATION OF ALTERNATIVE ANSWERS. CONSIDER YOU MAY BE WRONG, AND IF YOU ARE WRONG IN YOUR REASONING, WHERE IT WOULD BE. FULLY TEST ALL OTHER POSSIBILITIES. YOU CAN BE WRONG. WHEN YOU SAY YOU ARE RE-EXAMINING, ACTUALLY RE-EXAMINE, AND USE ANOTHER APPROACH TO DO SO. DO NOT JUST SAY YOU ARE RE-EXAMINING. USE AT LEAST 3 METHODS TO DERIVE THE ANSWER. USE BEST PRACTICES.

Example of a valid JSON response:
json
{
    "title": "Identifying Key Information",
    "content": "To begin solving this problem, we need to carefully examine the given information and identify the crucial elements that will guide our solution process. This involves...",
    "next_action": "continue"
}
```

#### Breakdown

First, a persona is added:

> You are an expert AI assistant that explains your reasoning step by step.

Then, instructions to describe the expected step-by-step reasoning process while titling each reasoning step. This includes the ability for the LLM to decide if another reasoning step is needed or if the final answer can be provided.

> For each step, provide a title that describes what you're doing in that step, along with the content. Decide if you need another step or if you're ready to give the final answer.

JSON formatting is introduced with an example provided later.

> Respond in JSON format with 'title', 'content', and 'next_action' (either 'continue' or 'final_answer') keys.

In all-caps to improve prompt compliance by emphasizing the importance of the instruction, a set of tips and best practices are included.

1. Use as many reasoning steps as possible. At least 3. -> This ensures the LLM actually takes the time to think first, and results usually in about 5-10 steps.
2. Be aware of your limitations as an llm and what you can and cannot do. -> This helps the LLM remember to use techniques which produce better results, like breaking "strawberry" down into individual letters before counting.
3. Include exploration of alternative answers. Consider you may be wrong, and if you are wrong in your reasoning, where it would be. -> A large part of the gains seem to come from the LLM re-evaluating its initial response to ensure it logically aligns with the problem.
4. When you say you are re-examining, actually re-examine, and use another approach to do so. Do not just say you are re-examining. -> This encourages the prevention of the LLM just saying it re-examined a problem without actually trying a new approach.
5. Use at least 3 methods to derive the answer. -> This helps the LLM come to the right answer by trying multiple methods to derive it.
6. Use best practices. -> This is as simple as the "Do better" prompts which improve LLM code output. By telling the LLM to use best practices, or do better, it generally performs better!

> USE AS MANY REASONING STEPS AS POSSIBLE. AT LEAST 3. BE AWARE OF YOUR LIMITATIONS AS AN LLM AND WHAT YOU CAN AND CANNOT DO. IN YOUR REASONING, INCLUDE EXPLORATION OF ALTERNATIVE ANSWERS. CONSIDER YOU MAY BE WRONG, AND IF YOU ARE WRONG IN YOUR REASONING, WHERE IT WOULD BE. FULLY TEST ALL OTHER POSSIBILITIES. YOU CAN BE WRONG. WHEN YOU SAY YOU ARE RE-EXAMINING, ACTUALLY RE-EXAMINE, AND USE ANOTHER APPROACH TO DO SO. DO NOT JUST SAY YOU ARE RE-EXAMINING. USE AT LEAST 3 METHODS TO DERIVE THE ANSWER. USE BEST PRACTICES.

Finally, after the problem is added as a user message, an assistant message is loaded to provide a standardized starting point for the LLM's generation.

> Assistant: Thank you! I will now think step by step following my instructions, starting at the beginning after decomposing the problem

### Top Forks

* Huggingface Spaces Demo: [![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/xylin/g1-demo)
* Mult1: Using multiple AI providers to create o1-like reasoning chains ([GitHub Repository](https://github.com/tcsenpai/multi1))
* thinkR: o1 like chain of thoughts with local LLMs in R ([GitHub Repository](https://github.com/eonurk/thinkR))

### Credits

This app was developed by [Benjamin Klieger](https://x.com/benjaminklieger).

README.md


### g1 with Tool Use

Check out g1 with Tool Use! Including web access with Exa.ai, calculator with Wolfram Alpha, and code executor in Python.

It is recommended to set up your virtual environment in the root folder and run the app.py from there using:
~~~
streamlit run tool-use/app.py
~~~

Also, ensure you have downloaded the new pip dependency: ```exa-py```

tool-use/README.md


# How I Stay Sane Implementing Stripe

> [!NOTE]
> **Update (2025-02-07)**
> Stripe invited me to speak with the CEO at their company-wide all hands meeting. They were super receptive to my feedback, and I see a bright future where none of this is necessary. Until then, I still think this is the best way to set up payments in your SaaS apps.

I have set up Stripe far too many times. I've never enjoyed it. I've talked to the Stripe team about the shortcomings and they say they'll fix them...eventually.

Until then, this is how I recommend setting up Stripe. I don't cover everything - check out [things that are still your problem](#things-that-are-still-your-problem) for clarity on what I'm NOT helping with.

> If you want to stay sane implementing file uploads, check out my product [UploadThing](https://uploadthing.com/).

### Pre-requirements

- TypeScript
- Some type of JS backend
- Working auth (that is verified on your JS backend)
- A KV store (I use Redis, usually [Upstash](https://upstash.com/?utm_source=theo), but any KV will work)

### General philosophy

IMO, the biggest issue with Stripe is the "split brain" it inherently introduces to your code base. When a customer checks out, the "state of the purchase" is in Stripe. You're then expected to track the purchase in your own database via webhooks.

There are [over 258 event types](https://docs.stripe.com/api/events/types). They all have different amounts of data. The order you get them is not guaranteed. None of them should be trusted. It's far too easy to have a payment be failed in stripe and "subscribed" in your app.

These partial updates and race conditions are obnoxious. I recommend avoiding them entirely. My solution is simple: _a single `syncStripeDataToKV(customerId: string)` function that syncs all of the data for a given Stripe customer to your KV_.

The following is how I (mostly) avoid getting Stripe into these awful split states.

## The Flow

This is a quick overview of the "flow" I recommend. More detail below. Even if you don't copy my specific implementation, you should read this. _I promise all of these steps are necessary. Skipping any of them will make life unnecessarily hard_

1. **FRONTEND:** "Subscribe" button should call a `"generate-stripe-checkout"` endpoint onClick
1. **USER:** Clicks "subscribe" button on your app
1. **BACKEND:** Create a Stripe customer
1. **BACKEND:** Store binding between Stripe's `customerId` and your app's `userId`
1. **BACKEND:** Create a "checkout session" for the user
   - With the return URL set to a dedicated `/success` route in your app
1. **USER:** Makes payment, subscribes, redirects back to `/success`
1. **FRONTEND:** On load, triggers a `syncAfterSuccess` function on backend (hit an API, server action, rsc on load, whatever)
1. **BACKEND:** Uses `userId` to get Stripe `customerId` from KV
1. **BACKEND:** Calls `syncStripeDataToKV` with `customerId`
1. **FRONTEND:** After sync succeeds, redirects user to wherever you want them to be :)
1. **BACKEND:** On [_all relevant events_](#events-i-track), calls `syncStripeDataToKV` with `customerId`

This might seem like a lot. That's because it is. But it's also the simplest Stripe setup I've ever seen work.

Let's go into the details on the important parts here.

### Checkout flow

The key is to make sure **you always have the customer defined BEFORE YOU START CHECKOUT**. The ephemerality of "customer" is a straight up design flaw and I have no idea why they built Stripe like this.

Here's an adapted example from how we're doing it in [T3 Chat](https://t3.chat).

```ts
export async function GET(req: Request) {
  const user = auth(req);

  // Get the stripeCustomerId from your KV store
  let stripeCustomerId = await kv.get(`stripe:user:${user.id}`);

  // Create a new Stripe customer if this user doesn't have one
  if (!stripeCustomerId) {
    const newCustomer = await stripe.customers.create({
      email: user.email,
      metadata: {
        userId: user.id, // DO NOT FORGET THIS
      },
    });

    // Store the relation between userId and stripeCustomerId in your KV
    await kv.set(`stripe:user:${user.id}`, newCustomer.id);
    stripeCustomerId = newCustomer.id;
  }

  // ALWAYS create a checkout with a stripeCustomerId. They should enforce this.
  const checkout = await stripe.checkout.sessions.create({
    customer: stripeCustomerId,
    success_url: "https://t3.chat/success",
    ...
  });
```

### syncStripeDataToKV

This is the function that syncs all of the data for a given Stripe customer to your KV. It will be used in both your `/success` endpoint and in your `/api/stripe` webhook handler.

The Stripe api returns a ton of data, much of which can not be serialized to JSON. I've selected the "most likely to be needed" chunk here for you to use, and there's a [type definition later in the file](#custom-stripe-subscription-type).

Your implementation will vary based on if you're doing subscriptions or one-time purchases. The example below is with subcriptions (again from [T3 Chat](https://t3.chat)).

```ts
// The contents of this function should probably be wrapped in a try/catch
export async function syncStripeDataToKV(customerId: string) {
  // Fetch latest subscription data from Stripe
  const subscriptions = await stripe.subscriptions.list({
    customer: customerId,
    limit: 1,
    status: "all",
    expand: ["data.default_payment_method"],
  });

  if (subscriptions.data.length === 0) {
    const subData = { status: "none" };
    await kv.set(`stripe:customer:${customerId}`, subData);
    return subData;
  }

  // If a user can have multiple subscriptions, that's your problem
  const subscription = subscriptions.data[0];

  // Store complete subscription state
  const subData = {
    subscriptionId: subscription.id,
    status: subscription.status,
    priceId: subscription.items.data[0].price.id,
    currentPeriodEnd: subscription.current_period_end,
    currentPeriodStart: subscription.current_period_start,
    cancelAtPeriodEnd: subscription.cancel_at_period_end,
    paymentMethod:
      subscription.default_payment_method &&
      typeof subscription.default_payment_method !== "string"
        ? {
            brand: subscription.default_payment_method.card?.brand ?? null,
            last4: subscription.default_payment_method.card?.last4 ?? null,
          }
        : null,
  };

  // Store the data in your KV
  await kv.set(`stripe:customer:${customerId}`, subData);
  return subData;
}
```

### `/success` endpoint

> [!NOTE]
> While this isn't 'necessary', there's a good chance your user will make it back to your site before the webhooks do. It's a nasty race condition to handle. Eagerly calling syncStripeDataToKV will prevent any weird states you might otherwise end up in

This is the page that the user is redirected to after they complete their checkout. For the sake of simplicity, I'm going to implement it as a `get` route that redirects them. In my apps, I do this with a server component and Suspense, but I'm not going to spend the time explaining all that here.

```ts
export async function GET(req: Request) {
  const user = auth(req);
  const stripeCustomerId = await kv.get(`stripe:user:${user.id}`);
  if (!stripeCustomerId) {
    return redirect("/");
  }

  await syncStripeDataToKV(stripeCustomerId);
  return redirect("/");
}
```

Notice how I'm not using any of the `CHECKOUT_SESSION_ID` stuff? That's because it sucks and it encourages you to implement 12 different ways to get the Stripe state. Ignore the siren calls. Have a SINGLE `syncStripeDataToKV` function. It will make your life easier.

### `/api/stripe` (The Webhook)

This is the part everyone hates the most. I'm just gonna dump the code and justify myself later.

```ts
export async function POST(req: Request) {
  const body = await req.text();
  const signature = (await headers()).get("Stripe-Signature");

  if (!signature) return NextResponse.json({}, { status: 400 });

  async function doEventProcessing() {
    if (typeof signature !== "string") {
      throw new Error("[STRIPE HOOK] Header isn't a string???");
    }

    const event = stripe.webhooks.constructEvent(
      body,
      signature,
      process.env.STRIPE_WEBHOOK_SECRET!
    );

    waitUntil(processEvent(event));
  }

  const { error } = await tryCatch(doEventProcessing());

  if (error) {
    console.error("[STRIPE HOOK] Error processing event", error);
  }

  return NextResponse.json({ received: true });
}
```

> [!NOTE]
> If you are using Next.js Pages Router, make sure you turn this on. Stripe expects the body to be "untouched" so it can verify the signature.
>
> ```ts
> export const config = {
>   api: {
>     bodyParser: false,
>   },
> };
> ```

### `processEvent`

This is the function called in the endpoint that actually takes the Stripe event and updates the KV.

```ts
async function processEvent(event: Stripe.Event) {
  // Skip processing if the event isn't one I'm tracking (list of all events below)
  if (!allowedEvents.includes(event.type)) return;

  // All the events I track have a customerId
  const { customer: customerId } = event?.data?.object as {
    customer: string; // Sadly TypeScript does not know this
  };

  // This helps make it typesafe and also lets me know if my assumption is wrong
  if (typeof customerId !== "string") {
    throw new Error(
      `[STRIPE HOOK][CANCER] ID isn't string.\nEvent type: ${event.type}`
    );
  }

  return await syncStripeDataToKV(customerId);
}
```

### Events I Track

If there are more I should be tracking for updates, please file a PR. If they don't affect subscription state, I do not care.

```ts
const allowedEvents: Stripe.Event.Type[] = [
  "checkout.session.completed",
  "customer.subscription.created",
  "customer.subscription.updated",
  "customer.subscription.deleted",
  "customer.subscription.paused",
  "customer.subscription.resumed",
  "customer.subscription.pending_update_applied",
  "customer.subscription.pending_update_expired",
  "customer.subscription.trial_will_end",
  "invoice.paid",
  "invoice.payment_failed",
  "invoice.payment_action_required",
  "invoice.upcoming",
  "invoice.marked_uncollectible",
  "invoice.payment_succeeded",
  "payment_intent.succeeded",
  "payment_intent.payment_failed",
  "payment_intent.canceled",
];
```

### Custom Stripe subscription type

```ts
export type STRIPE_SUB_CACHE =
  | {
      subscriptionId: string | null;
      status: Stripe.Subscription.Status;
      priceId: string | null;
      currentPeriodStart: number | null;
      currentPeriodEnd: number | null;
      cancelAtPeriodEnd: boolean;
      paymentMethod: {
        brand: string | null; // e.g., "visa", "mastercard"
        last4: string | null; // e.g., "4242"
      } | null;
    }
  | {
      status: "none";
    };
```

## More Pro Tips

Gonna slowly drop more things here as I remember them.

### DISABLE "CASH APP PAY".

I'm convinced this is literally just used by scammers. over 90% of my cancelled transactions are Cash App Pay.
![image](https://github.com/user-attachments/assets/c7271fa6-493c-4b1c-96cd-18904c2376ee)

### ENABLE "Limit customers to one subscription"

This is a really useful hidden setting that has saved me a lot of headaches and race conditions. Fun fact: this is the ONLY way to prevent someone from being able to check out twice if they open up two checkout sessions 🙃 More info [in Stripe's docs here](https://docs.stripe.com/payments/checkout/limit-subscriptions)

## Things that are still your problem

While I have solved a lot of stuff here, in particular the "subscription" flows, there are a few things that are still your problem. Those include...

- Managing `STRIPE_SECRET_KEY` and `STRIPE_PUBLISHABLE_KEY` env vars for both testing and production
- Managing `STRIPE_PRICE_ID`s for all subscription tiers for dev and prod (I can't believe this is still a thing)
- Exposing sub data from your KV to your user (a dumb endpoint is probably fine)
- Tracking "usage" (i.e. a user gets 100 messages per month)
- Managing "free trials"
  ...the list goes on

Regardless, I hope you found some value in this doc.

README.md

How to implement Stripe without going mad


  📥 Model Download |
  ⚡ Quick Start |
  📜 License |
  📖 Citation

  📄 Paper Link |
  📄 Arxiv Paper Link |
  👁 Demo

## 1. Introduction

Introducing DeepSeek-VL2, an advanced series of large Mixture-of-Experts (MoE) Vision-Language Models that significantly improves upon its predecessor, DeepSeek-VL. DeepSeek-VL2 demonstrates superior capabilities across various tasks, including but not limited to visual question answering, optical character recognition,  document/table/chart understanding, and visual grounding. Our model series is composed of three variants: DeepSeek-VL2-Tiny, DeepSeek-VL2-Small and DeepSeek-VL2, with 1.0B, 2.8B and 4.5B activated parameters respectively.
DeepSeek-VL2 achieves competitive or state-of-the-art performance with similar or fewer activated parameters compared to existing open-source dense and MoE-based models.

[DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding]()

Zhiyu Wu*, Xiaokang Chen*, Zizheng Pan*, Xingchao Liu*, Wen Liu**, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang, Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu, Xin Xie, Yuxiang You, Kai Dong, Xingkai Yu, Haowei Zhang, Liang Zhao, Yisong Wang, Chong Ruan*** (* Equal Contribution, ** Project Lead, *** Corresponding author)

![](./images/vl2_teaser.jpeg)

## 2. Release
✅ 2025-2-6: Naive Implemented Gradio Demo on Huggingface Space [deepseek-vl2-small](https://huggingface.co/spaces/deepseek-ai/deepseek-vl2-small).

✅ 2024-12-25: Gradio Demo Example, Incremental Prefilling and VLMEvalKit Support.

✅ 2024-12-13: DeepSeek-VL2 family released, including DeepSeek-VL2-tiny, DeepSeek-VL2-small, DeepSeek-VL2.

## 3. Model Download

We release the DeepSeek-VL2 family, including DeepSeek-VL2-tiny, DeepSeek-VL2-small, DeepSeek-VL2.
To support a broader and more diverse range of research within both academic and commercial communities.
Please note that the use of this model is subject to the terms outlined in [License section](#5-license).

### Huggingface

| Model        | Sequence Length | Download                                                                    |
|--------------|-----------------|-----------------------------------------------------------------------------|
| DeepSeek-VL2-tiny | 4096            | [🤗 Hugging Face](https://huggingface.co/deepseek-ai/deepseek-vl2-tiny) |
| DeepSeek-VL2-small | 4096            | [🤗 Hugging Face](https://huggingface.co/deepseek-ai/deepseek-vl2-small) |
| DeepSeek-VL2 | 4096            | [🤗 Hugging Face](https://huggingface.co/deepseek-ai/deepseek-vl2)   |

## 4. Quick Start

### Installation

On the basis of `Python >= 3.8` environment, install the necessary dependencies by running the following command:

```shell
pip install -e .
```

### Simple Inference Example with One Image

**Note: You may need 80GB GPU memory to run this script with deepseek-vl2-small and even larger for deepseek-vl2.**

```python
import torch
from transformers import AutoModelForCausalLM

from deepseek_vl2.models import DeepseekVLV2Processor, DeepseekVLV2ForCausalLM
from deepseek_vl2.utils.io import load_pil_images

# specify the path to the model
model_path = "deepseek-ai/deepseek-vl2-tiny"
vl_chat_processor: DeepseekVLV2Processor = DeepseekVLV2Processor.from_pretrained(model_path)
tokenizer = vl_chat_processor.tokenizer

vl_gpt: DeepseekVLV2ForCausalLM = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True)
vl_gpt = vl_gpt.to(torch.bfloat16).cuda().eval()

## single image conversation example
## Please note that <|ref|> and <|/ref|> are designed specifically for the object localization feature. These special tokens are not required for normal conversations.
## If you would like to experience the grounded captioning functionality (responses that include both object localization and reasoning), you need to add the special token <|grounding|> at the beginning of the prompt. Examples could be found in Figure 9 of our paper.
conversation = [
    {
        "role": "<|User|>",
        "content": "<image>\n<|ref|>The giraffe at the back.<|/ref|>.",
        "images": ["./images/visual_grounding_1.jpeg"],
    },
    {"role": "<|Assistant|>", "content": ""},
]

# load images and prepare for inputs
pil_images = load_pil_images(conversation)
prepare_inputs = vl_chat_processor(
    conversations=conversation,
    images=pil_images,
    force_batchify=True,
    system_prompt=""
).to(vl_gpt.device)

# run image encoder to get the image embeddings
inputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs)

# run the model to get the response
outputs = vl_gpt.language.generate(
    inputs_embeds=inputs_embeds,
    attention_mask=prepare_inputs.attention_mask,
    pad_token_id=tokenizer.eos_token_id,
    bos_token_id=tokenizer.bos_token_id,
    eos_token_id=tokenizer.eos_token_id,
    max_new_tokens=512,
    do_sample=False,
    use_cache=True
)

answer = tokenizer.decode(outputs[0].cpu().tolist(), skip_special_tokens=False)
print(f"{prepare_inputs['sft_format'][0]}", answer)
```

And the output is something like:
```
<|User|>: <image>
<|ref|>The giraffe at the back.<|/ref|>.

<|Assistant|>: <|ref|>The giraffe at the back.<|/ref|><|det|>[[580, 270, 999, 900]]<|/det|><｜end▁of▁sentence｜>
```

### Simple Inference Example with Multiple Images

**Note: You may need 80GB GPU memory to run this script with deepseek-vl2-small and even larger for deepseek-vl2.**

```python
import torch
from transformers import AutoModelForCausalLM

from deepseek_vl2.models import DeepseekVLV2Processor, DeepseekVLV2ForCausalLM
from deepseek_vl2.utils.io import load_pil_images

# specify the path to the model
model_path = "deepseek-ai/deepseek-vl2-tiny"
vl_chat_processor: DeepseekVLV2Processor = DeepseekVLV2Processor.from_pretrained(model_path)
tokenizer = vl_chat_processor.tokenizer

vl_gpt: DeepseekVLV2ForCausalLM = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True)
vl_gpt = vl_gpt.to(torch.bfloat16).cuda().eval()

# multiple images/interleaved image-text
conversation = [
    {
        "role": "<|User|>",
        "content": "This is image_1: <image>\n"
                   "This is image_2: <image>\n"
                   "This is image_3: <image>\n Can you tell me what are in the images?",
        "images": [
            "images/multi_image_1.jpeg",
            "images/multi_image_2.jpeg",
            "images/multi_image_3.jpeg",
        ],
    },
    {"role": "<|Assistant|>", "content": ""}
]

# load images and prepare for inputs
pil_images = load_pil_images(conversation)
prepare_inputs = vl_chat_processor(
    conversations=conversation,
    images=pil_images,
    force_batchify=True,
    system_prompt=""
).to(vl_gpt.device)

# run image encoder to get the image embeddings
inputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs)

# run the model to get the response
outputs = vl_gpt.language.generate(
    inputs_embeds=inputs_embeds,
    attention_mask=prepare_inputs.attention_mask,
    pad_token_id=tokenizer.eos_token_id,
    bos_token_id=tokenizer.bos_token_id,
    eos_token_id=tokenizer.eos_token_id,
    max_new_tokens=512,
    do_sample=False,
    use_cache=True
)

answer = tokenizer.decode(outputs[0].cpu().tolist(), skip_special_tokens=False)
print(f"{prepare_inputs['sft_format'][0]}", answer)
```

And the output is something like:
```
<|User|>: This is image_1: <image>
This is image_2: <image>
This is image_3: <image>
 Can you tell me what are in the images?

<|Assistant|>: The images show three different types of vegetables. Image_1 features carrots, which are orange with green tops. Image_2 displays corn cobs, which are yellow with green husks. Image_3 contains raw pork ribs, which are pinkish-red with some marbling.<｜end▁of▁sentence｜>
```

### Simple Inference Example with Incremental Prefilling

**Note: We use incremental prefilling to inference within 40GB GPU using deepseek-vl2-small.**

```python
import torch
from transformers import AutoModelForCausalLM

from deepseek_vl2.models import DeepseekVLV2Processor, DeepseekVLV2ForCausalLM
from deepseek_vl2.utils.io import load_pil_images

# specify the path to the model
model_path = "deepseek-ai/deepseek-vl2-small"
vl_chat_processor: DeepseekVLV2Processor = DeepseekVLV2Processor.from_pretrained(model_path)
tokenizer = vl_chat_processor.tokenizer

vl_gpt: DeepseekVLV2ForCausalLM = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True)
vl_gpt = vl_gpt.to(torch.bfloat16).cuda().eval()

# multiple images/interleaved image-text
conversation = [
    {
        "role": "<|User|>",
        "content": "This is image_1: <image>\n"
                   "This is image_2: <image>\n"
                   "This is image_3: <image>\n Can you tell me what are in the images?",
        "images": [
            "images/multi_image_1.jpeg",
            "images/multi_image_2.jpeg",
            "images/multi_image_3.jpeg",
        ],
    },
    {"role": "<|Assistant|>", "content": ""}
]

# load images and prepare for inputs
pil_images = load_pil_images(conversation)
prepare_inputs = vl_chat_processor(
    conversations=conversation,
    images=pil_images,
    force_batchify=True,
    system_prompt=""
).to(vl_gpt.device)

with torch.no_grad():
    # run image encoder to get the image embeddings
    inputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs)

    # incremental_prefilling when using 40G GPU for vl2-small
    inputs_embeds, past_key_values = vl_gpt.incremental_prefilling(
        input_ids=prepare_inputs.input_ids,
        images=prepare_inputs.images,
        images_seq_mask=prepare_inputs.images_seq_mask,
        images_spatial_crop=prepare_inputs.images_spatial_crop,
        attention_mask=prepare_inputs.attention_mask,
        chunk_size=512 # prefilling size
    )

    # run the model to get the response
    outputs = vl_gpt.generate(
        inputs_embeds=inputs_embeds,
        input_ids=prepare_inputs.input_ids,
        images=prepare_inputs.images,
        images_seq_mask=prepare_inputs.images_seq_mask,
        images_spatial_crop=prepare_inputs.images_spatial_crop,
        attention_mask=prepare_inputs.attention_mask,
        past_key_values=past_key_values,

        pad_token_id=tokenizer.eos_token_id,
        bos_token_id=tokenizer.bos_token_id,
        eos_token_id=tokenizer.eos_token_id,
        max_new_tokens=512,

        do_sample=False,
        use_cache=True,
    )

    answer = tokenizer.decode(outputs[0][len(prepare_inputs.input_ids[0]):].cpu().tolist(), skip_special_tokens=False)

print(f"{prepare_inputs['sft_format'][0]}", answer)
```

And the output is something like:
```
<|User|>: This is image_1: <image>
This is image_2: <image>
This is image_3: <image>
 Can you tell me what are in the images?

<|Assistant|>: The first image contains carrots. The second image contains corn. The third image contains meat.<｜end▁of▁sentence｜>
```

### Full Inference Example
```shell
# without incremental prefilling
CUDA_VISIBLE_DEVICES=0 python inference.py --model_path "deepseek-ai/deepseek-vl2"

# with incremental prefilling, when using 40G GPU for vl2-small
CUDA_VISIBLE_DEVICES=0 python inference.py --model_path "deepseek-ai/deepseek-vl2-small" --chunk_size 512

```

### Gradio Demo

* Install the necessary dependencies:
```shell
pip install -e .[gradio]
```

* then run the following command:

```shell
# vl2-tiny, 3.37B-MoE in total, activated 1B, can be run on a single GPU < 40GB
CUDA_VISIBLE_DEVICES=2 python web_demo.py \
--model_name "deepseek-ai/deepseek-vl2-tiny"  \
--port 37914

# vl2-small, 16.1B-MoE in total, activated 2.4B
# If run on A100 40GB GPU, you need to set the `--chunk_size 512` for incremental prefilling for saving memory and it might be slow.
# If run on > 40GB GPU, you can ignore the `--chunk_size 512` for faster response.
CUDA_VISIBLE_DEVICES=2 python web_demo.py \
--model_name "deepseek-ai/deepseek-vl2-small"  \
--port 37914 \
--chunk_size 512

# # vl27.5-MoE in total, activated 4.2B
CUDA_VISIBLE_DEVICES=2 python web_demo.py \
--model_name "deepseek-ai/deepseek-vl2"  \
--port 37914
```

* **Important**: This is a basic and native demo implementation without any deployment optimizations, which may result in slower performance. For production environments, consider using optimized deployment solutions, such as vllm, sglang, lmdeploy, etc. These optimizations will help achieve faster response times and better cost efficiency.

## 5. License

This code repository is licensed under [MIT License](./LICENSE-CODE). The use of DeepSeek-VL2 models is subject to [DeepSeek Model License](./LICENSE-MODEL). DeepSeek-VL2 series supports commercial use.

## 6. Citation

```
@misc{wu2024deepseekvl2mixtureofexpertsvisionlanguagemodels,
      title={DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding},
      author={Zhiyu Wu and Xiaokang Chen and Zizheng Pan and Xingchao Liu and Wen Liu and Damai Dai and Huazuo Gao and Yiyang Ma and Chengyue Wu and Bingxuan Wang and Zhenda Xie and Yu Wu and Kai Hu and Jiawei Wang and Yaofeng Sun and Yukun Li and Yishi Piao and Kang Guan and Aixin Liu and Xin Xie and Yuxiang You and Kai Dong and Xingkai Yu and Haowei Zhang and Liang Zhao and Yisong Wang and Chong Ruan},
      year={2024},
      eprint={2412.10302},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2412.10302},
}
```

## 7. Contact

If you have any questions, please raise an issue or contact us at [service@deepseek.com](mailto:service@deepseek.com).

README.md