This directory contains data needed by Bison.

# Directory Content
## Skeletons
Bison skeletons: the general shapes of the different parser kinds, that are
specialized for specific grammars by the bison program.

Currently, the supported skeletons are:

- yacc.c
  It used to be named bison.simple: it corresponds to C Yacc
  compatible LALR(1) parsers.

- lalr1.cc
  Produces a C++ parser class.

- lalr1.java
  Produces a Java parser class.

- glr.c
  A Generalized LR C parser based on Bison's LALR(1) tables.

- glr.cc
  A Generalized LR C++ parser.  Actually a C++ wrapper around glr.c.

These skeletons are the only ones supported by the Bison team.  Because the
interface between skeletons and the bison program is not finished, *we are
not bound to it*.  In particular, Bison is not mature enough for us to
consider that "foreign skeletons" are supported.

## m4sugar
This directory contains M4sugar, sort of an extended library for M4, which
is used by Bison to instantiate the skeletons.

## xslt
This directory contains XSLT programs that transform Bison's XML output into
various formats.

- bison.xsl
  A library of routines used by the other XSLT programs.

- xml2dot.xsl
  Conversion into GraphViz's dot format.

- xml2text.xsl
  Conversion into text.

- xml2xhtml.xsl
  Conversion into XHTML.

# Implementation Notes About the Skeletons

"Skeleton" in Bison parlance means "backend": a skeleton is fed by the bison
executable with LR tables, facts about the symbols, etc. and they generate
the output (say parser.cc, parser.hh, location.hh, etc.).  They are only in
charge of generating the parser and its auxiliary files, they do not
generate the XML output, the parser.output reports, nor the graphical
rendering.

The bits of information passing from bison to the backend is named
"muscles".  Muscles are passed to M4 via its standard input: it's a set of
m4 definitions.  To see them, use `--trace=muscles`.

Except for muscles, whose names are generated by bison, the skeletons have
no constraint at all on the macro names: there is no technical/theoretical
limitation, as long as you generate the output, you can do what you want.
However, of course, that would be a bad idea if, say, the C and C++
skeletons used different approaches and had completely different
implementations.  That would be a maintenance nightmare.

Below, we document some of the macros that we use in several of the
skeletons.  If you are to write a new skeleton, please, implement them for
your language.  Overall, be sure to follow the same patterns as the existing
skeletons.

## Symbols

### `b4_symbol(NUM, FIELD)`
In order to unify the handling of the various aspects of symbols (tag, type
name, whether terminal, etc.), bison.exe defines one macro per (token,
field), where field can `has_id`, `id`, etc.: see
`prepare_symbols_definitions()` in `src/output.c`.

The macro `b4_symbol(NUM, FIELD)` gives access to the following FIELDS:

- `has_id`: 0 or 1
  Whether the symbol has an `id`.

- `id`: string (e.g., `exp`, `NUM`, or `TOK_NUM` with api.token.prefix)
  If `has_id`, the name of the token kind (prefixed by api.token.prefix if
  defined), otherwise empty.  Guaranteed to be usable as a C identifier.
  This is used to define the token kind (i.e., the enum used by the return
  value of yylex).  Should be named `token_kind`.

- `tag`: string
  A human readable representation of the symbol.  Can be `'foo'`,
  `'foo.id'`, `'"foo"'` etc.

- `code`: integer
  The token code associated to the token kind `id`.
  The external number as used by yylex.  Can be ASCII code when a character,
  some number chosen by bison, or some user number in the case of `%token
  FOO <NUM>`.  Corresponds to `yychar` in `yacc.c`.

- `is_token`: 0 or 1
  Whether this is a terminal symbol.

- `kind_base`: string (e.g., `YYSYMBOL_exp`, `YYSYMBOL_NUM`)
  The base of the symbol kind, i.e., the enumerator of this symbol (token or
  nonterminal) which is mapped to its `number`.

- `kind`: string
  Same as `kind_base`, but possibly with a prefix in some languages.  E.g.,
  EOF's `kind_base` and `kind` are `YYSYMBOL_YYEOF` in C, but are
  `S_YYEMPTY` and `symbol_kind::S_YYEMPTY` in C++.

- `number`: integer
  The code associated to the `kind`.
  The internal number (computed from the external number by yytranslate).
  Corresponds to yytoken in yacc.c.  This is the same number that serves as
  key in b4_symbol(NUM, FIELD).

  In bison, symbols are first assigned increasing numbers in order of
  appearance (but tokens first, then nterms).  After grammar reduction,
  unused nterms are then renumbered to appear last (i.e., first tokens, then
  used nterms and finally unused nterms).  This final number NUM is the one
  contained in this field, and it is the one used as key in `b4_symbol(NUM,
  FIELD)`.

  The code of the rule actions, however, is emitted before we know what
  symbols are unused, so they use the original numbers.  To avoid confusion,
  they actually use "orig NUM" instead of just "NUM".  bison also emits
  definitions for `b4_symbol(orig NUM, number)` that map from original
  numbers to the new ones.  `b4_symbol` actually resolves `orig NUM` in the
  other case, i.e., `b4_symbol(orig 42, tag)` would return the tag of the
  symbols whose original number was 42.

- `has_type`: 0, 1
  Whether has a semantic value.

- `type_tag`: string
  When api.value.type=union, the generated name for the union member.
  yytype_INT etc. for symbols that has_id, otherwise yytype_1 etc.

- `type`: string
  If it has a semantic value, its type tag, or, if variant are used,
  its type.
  In the case of api.value.type=union, type is the real type (e.g. int).

- `slot`: string
  If it has a semantic value, the name of the union member (i.e., bounces to
  either `type_tag` or `type`).  It would be better to fix our mess and
  always use `type` for the true type of the member, and `type_tag` for the
  name of the union member.

- `has_printer`: 0, 1
- `printer`: string
- `printer_file`: string
- `printer_line`: integer
- `printer_loc`: location
  If the symbol has a printer, everything about it.

- `has_destructor`, `destructor`, `destructor_file`, `destructor_line`, `destructor_loc`
  Likewise.

### `b4_symbol_value(VAL, [SYMBOL-NUM], [TYPE-TAG])`
Expansion of $$, $1, $<TYPE-TAG>3, etc.

The semantic value from a given VAL.
- `VAL`: some semantic value storage (typically a union).  e.g., `yylval`
- `SYMBOL-NUM`: the symbol number from which we extract the type tag.
- `TYPE-TAG`, the user forced the `<TYPE-TAG>`.

The result can be used safely, it is put in parens to avoid nasty precedence
issues.

### `b4_lhs_value(SYMBOL-NUM, [TYPE])`
Expansion of `$$` or `$<TYPE>$`, for symbol `SYMBOL-NUM`.

### `b4_rhs_data(RULE-LENGTH, POS)`
The data corresponding to the symbol `#POS`, where the current rule has
`RULE-LENGTH` symbols on RHS.

### `b4_rhs_value(RULE-LENGTH, POS, SYMBOL-NUM, [TYPE])`
Expansion of `$<TYPE>POS`, where the current rule has `RULE-LENGTH` symbols
on RHS.

<!--

Local Variables:
mode: markdown
fill-column: 76
ispell-dictionary: "american"
End:

Copyright (C) 2002, 2008-2015, 2018-2021 Free Software Foundation, Inc.

This file is part of GNU Bison.

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see <https://www.gnu.org/licenses/>.

-->


The Python C API
================

The C API is divided into these sections:

1. ``Include/``: Limited API
2. ``Include/cpython/``: CPython implementation details
3. ``Include/cpython/``, names with the ``PyUnstable_`` prefix: API that can
   change between minor releases
4. ``Include/internal/``, and any name with ``_`` prefix: The internal API

Information on changing the C API is available `in the developer guide`_

.. _in the developer guide: https://devguide.python.org/c-api/

Perforator is a cluster-wide continuous profiling tool designed for large data centers


# logging (with profiling features)

See [`./demo`](./demo/app.py) application for using example.

## Полезные ссылки

**Спецификация Trace Event Format**

- https://docs.google.com/document/d/1CvAClvFfyA5R-PhYUmn5OOQtYMH4h6I0nSsKchNAySU

**UI для просмотра трейсов:**

- Chrome DevTools, вкладка Performance (кнопка Load profile), [руководство](https://developer.chrome.com/docs/devtools/performance/reference/)
- https://ui.perfetto.dev/
- <chrome://tracing/> в Chromium браузерах, включая [Yandex Browser](browser://tracing/)


# nots builder

Внутренний инструмент, выполняющий сборку Frontend-проектов (узлов сборки) в рамках сборщика `ya make`.

## Собирает папку `node_modules`

Запускает `pnpm install`, используя скаченные заранее архивы пакетов.

В результате ожидается архив `node_modules.tar`.

_Скоро выпилим_

## Запускает сборщики типа `tsc`, `webpack`, `next`, `vite`

В соответствии с типом таргета (`TS_TSC`, `TS_WEBPACK`, `TS_NEXT`, `TS_VITE` макросами в `ya.make`) для непосредственной сборки проекта.

Пакеты должны быть установлены в качестве зависимостей проекта, обычно в `devDependencies`.

В результате ожидается архив `<module_name>.output.tar`.

## Релиз

`builder` собирается в бинарники, которые используются внутри сборки уже собранными.
Маппинг собранных под разные платформы ресурсы описан в [resources.json](./resources.json).

Временно релиз настроен на ручной запуск, причем в двух местах: при запуске релиза и при влитии результата в транк.

При изменениях в builder в рамках PR нужно:

0. Нужно пошарить PR на `robot-nots` **ВАЖНО!**
1. Запустить ручной action `devtools/frontend_build_platform/nots/builder: Build nots-builder (from PR)`
2. Будет собран builder из ветки и новые ресурсы будут прописаны в [resources.json](./resources.json)
3. С новым коммитом будет перезапущены все tier0 проекты на TS макросах - проверка, что изменения билдера ничего не сломали
4. "Релиз" новой версии - это влитие ветки.

{% note warning "ПОМНИ!" %}

При дополнительных коммитах в PR нужно вручную запускать action

```
devtools/frontend_build_platform/nots/builder: Build nots-builder (from PR)
```

{% endnote %}

Можно не делать проверку в рамках своего ПР, а после влития запустить
[релиз](https://a.yandex-team.ru/projects/?frontend_build_platform/ci/releases/timeline?dir=devtools%2Ffrontend_build_platform%2Fnots%2Fbuilder&id=release-nots),
который:

1. Соберет новые ресурсы
2. Обновит [resources.json](./resources.json)
3. Сделает PR в транк
4. Будет ждать влития

## Локальный запуск

Чтобы отключить запуск предсобаранного бинарника и запускать с учетом изменений в ветке, то нужно указать переменную:

```shell
ya make -DTS_USE_PREBUILT_NOTS_TOOL=no --host-platform-flag=TS_USE_PREBUILT_NOTS_TOOL=no
```

Для удобства можно [настроить алиас](https://docs.yandex-team.ru/yatool/usage/options#primer-otklyuchenie-predpostroennyh-tulov) в `junk/<username>/ya.conf`:

```toml
[[include]]
path = "${YA_ARCADIA_ROOT}/devtools/frontend_build_platform/nots/builder/builder.ya.conf"
```

После чего будут доступны алиасы `-n` и `--no-prebuilt-nots`:

```shell
ya make -n
ya make --no-prebuilt-nots
```

## Запуск

Обычно запускается внутри сборки `ya make`, как команда (`.CMD`).

Ручной запуск сложен, т.к. требует определенный набор переданных в аргументах подготовленных директорий.
Если очень уж хочется, то можно бросить исключение из builder (упасть).
Тогда сборка завершится с ошибкой, а при этом сборочные папки не удаляются, в лог пишется команда запуска builder со всем аргументами.
Таким образом его можно поправить и перезапустить с теми же параметрами в режиме дебага, например.

## Отладка

Можно включить логирование с опцией `-DTS_LOG=yes`, тогда `builder` напишет в консоль переданные аргументы, то, как он их распарсил.

А также покажет команду для запуска самого сборщика (можно будет скопировать и выполнить самому).

### Отладка с брекпоинтами

Настоящий дебаг в IDE делается так:

1. Настраивается окружение с помощью команды `ya ide venv` из папки `cli` (в принципе полезно, в codenv должно быть уже настроено);
2. Это окружение активируется в IDE;
3. Запустить `builder` с опцией `-DTS_LOG`, скопировать все аргументы командной строки из выхлопа;
4. Настроить запуск `main.py` в IDE в режиме дебага и передать все эти аргументы.

## Профилирование

Есть возможность сгенерировать файлы с трейсами для отладки производительности на всех этапах сборки (внутри `builder`)

Для этого нужно запустить `ya make` с опцией `-DTS_TRACE` (т.е. установкой переменной `TS_TRACE=yes`).

В выходной архив добавится папка `.traces` с файлом в `Trace Event Format`, который можно открыть в Chrome Devtools или в https://ui.perfetto.dev/.

Подробнее, о том, как читать трейсинг тут: https://a.yandex-team.ru/arcadia/devtools/frontend_build_platform/libraries/logging/README.md

После запуска из `ya tool nots build` архивы распакуются и трейсы следует искать по таким путям:

- Для `create-node-modules`: `./node_modules/.traces/builder_create-node-modules.trace.json`
- Для прочих команд: `./.traces/builder_<command>.trace.json`


# Architecture overview

This page describes Perforator's architecture.

## Intro
Perforator consists of several components. There are four different kinds of components:
1. Perforator agent which is running on each node of the profiled cluster
2. A few microservices to collect, store, analyze, and symbolize profiles & executable files from the agents. We run our microservices inside the same cluster Perforator profiles.
3. A web-based user interface.
4. A set of databases: ClickHouse to store profile metadata, PostgreSQL to store binary metadata, and S3-compatible store to store raw profiles and binaries.

Such design has been proven to be effective enough to scale to tens of thousands of nodes and petabytes of profiles.

## High-level schema

![Perforator schema](../../../_assets/perforator-architecture.excalidraw.svg)

## Components overview

### Agent
The Perforator agent is the core of our profiling infrastructure. The agent runs on each node inside the cluster, collects profiles using eBPF, aggregates, compresses and sends them to the storage via gRPC in the pprof-compatible format.

Agent connects to kubelet to identify running pods on a node. Also agent tracks all the processes on a node and analyzes running executables. To profile, the agent uses [perf_events Linux API](https://man7.org/linux/man-pages/man2/perf_event_open.2.html) to trigger eBPF program on each perf event like "1M CPU cycles" or "1K major pagefaults". The eBPF program collects info about thread it was invoked on like thread / process id, thread / process name, userspace & kernelspace call stack and so on. The program sends collected samples to the user-space part of the agent via eBPF perfbuf API.

![Agent schema](../../../_assets/agent-schema.light.excalidraw.svg)

Agent collects samples from the eBPF program in memory and periodically sends them to the storage over gRPC. By default, for each pod an agent will send a profile every minute. We call this profile, consisting of the samples of one workload over one minute, an *atomic profile*.

In order to generate human-readable profile, the agent should map addresses of executable instructions to source code lines. That process, called *symbolization*, is compute-intensive. If the same code is executed on thousands of nodes, the agent should run the same symbolization on each node, which is proven to be really expensive. So we took another approach.

In addition to profiles, agent uploads executable files found on the profiled nodes. This can sound scary, but with careful synchronization we guarantee that each binary file is uploaded only once. The binaries are uploaded through storage microservice to S3 and PostgreSQL storages, and can be post-processed later to generate symbolized profiles. Binary files are identified using *build-id*: unique identifier of the compiled binary, which is often computed as a hash of meaningful ELF sections. Some executables do not contain build-id, so we compute so-called *pseudo build-id*: hash based on a few random portions of executable text.

{% note info %}

While in theory we can support external artifact registries, this feature is not available now. Feel free to discuss and contribute.

{% endnote %}

### Databases
Perforator uses three different storages. All the other microservices are stateless.
- Raw atomic profiles, binaries, and some post-processed artifacts of binaries (GSYM data) are stored in the S3-compatible storage. S3 is cheap and scales well.
- For each atomic profile we write a row to the ClickHouse cluster. This allows us to quickly find interesting profiles using complex selectors, because ClickHouse is blazing fast. See [ClickHouse table structure](https://github.com/yandex/perforator/tree/main/perforator/cmd/migrate/migrations/clickhouse) for details.
- For each executable file we write a row to the PostgreSQL cluster to synchronize uploads and track binary TTL. We have found that there is not too many executable files and they can be easily stored in an PostgreSQL. However, it is quite easy to use your custom SQL database instead (probably a few hundred lines of Golang). See the source code for details.

For more details about database structure see [database reference](../../reference/database.md).

### Storage proxy
Storage is a simple gRPC server that proxies upload requests from agents to ClickHouse, PostgreSQL and S3. Storage proxy is stateless, despite the name, and can (and should!) safely be replicated. We run hundreds of storage pods in our largest cluster. Storage can optionally provide agent authorization using mTLS.

### Symbolizer
Symbolizer (or *proxy* in the source code) is a main user-facing service. Symbolizer allows user to list profiles or services, build merged profiles that span multiple atomic profiles matching one selector and so on. Symbolizer consumes a noticeable amount of resources: symbolization is heavy. We are working to optimize this. Likely the symbolizer itself is often running in the same cluster we profile and can be easily self-profiled.

Symbolizer provides two variants of the same API: raw gRPC interface and HTTP RPC using beautiful [grpc-gateway](https://github.com/grpc-ecosystem/grpc-gateway/). The gRPC interface can be used by CLIs and automated services like profile quality monitors while HTTP API is used by our web interface.


# Fetch Profile

Perforator CLI allows you to fetch profiles from the Perforator server.

## Collect a flamegraph for a service over 30 minutes from 10 samples and start viewer on localhost:9000

```console
perforator fetch --format=flamegraph -s "now-30m" --service "redis-master" -m 10 --serve ":9000""
```

## Collect a flamegraph from a pod for the last 30 minutes and start viewer on localhost:9000

```console
perforator fetch -s "now-30m" --pod-id "mongodb-statefulset-2" -m 10 --serve ":9000"
```

## Collect a flamegraph for an executable over 30 minutes from 10 samples

To identify the executable, use the BuildID. You can find the BuildID using the following command:

```console
readelf -n <path_to_binary>
```

```console
perforator fetch --format=flamegraph -s "now-30m" --build-id "abacaba" -m 10 --serve ":9000"
```

## Collect a pprof profile for an arbitrary selector

```console
perforator fetch --format=pprof -s "now-30m" --selector
'{node_id="example.org|worker-us-east1-b-1", timestamp>="now-30m"}' -m 10 -o profile.pprof
```

## Collect a flamegraph filtered by TLS string variable value

Before collecting a profile, you need to mark TLS variables in your code using one of `Y_PERFORATOR_THREAD_LOCAL` macros.

```console
perforator fetch --format=flamegraph -s "now-30m" --selector {node_id="worker-us-east1-b-1", "tls.TString_KEY"="VALUE"}
```


# Install Perforator CLI

Build `perforator/cmd/cli` from the root of the repository.

# Configure Perforator CLI

* Set `PERFORATOR_ENDPOINT` environment variable to specify the Perforator server URL once. Otherwise you need to use `--url` flag for each command.
* Set `PERFORATOR_SECURE` to enable or disable TLS. By default, TLS is enabled.

```console
export PERFORATOR_ENDPOINT="https://perforator.example.com"
export PERFORATOR_SECURE=true
```


# List Profiles by Selector

Perforator CLI allows you to list profiles by selector.

## View the list of profiles by host for the last 15 minutes

```console
perforator list profiles --node-id "worker-us-east1-b-1" -s "now-15m"
```

## View the list of profiles by service for the last 30 minutes

```console
perforator list profiles --service "kafka-broker" -s "now-30m""
```

## View the list of profiles by abstract selector

```console
perforator list profiles --selector '{node_id="example.org|worker-us-east1-b-1", timestamp>="now-30m"}'
```


# List Services

Perforator CLI allows you to list profiled services from the Perforator server.

## View the complete list of services sorted by name

```console
perforator list services
```

## View the list of services filtered by regex and sorted by number of profiles

```console
perforator list services -r "perforator" --order-by "profiles"
```

## View the list of services that had profiles in the last hour

```console
perforator list services --max-stale-age "1h"
```


# Force profile saving using Microscope

Microscope is a way to save all profiles from a specific selector bypassing sampling (for example `{host="worker-us-east1-b-1"}`).

## Save profiles from a host {#save-profiles-from-a-host}

Create a microscope to save profiles from the entire host for 1 hour starting from the current moment:

```console
perforator microscope create --node-id "worker-us-east1-b-1" --duration "1h" --start-time "now"
```

After this command, minute-by-minute profiles will start being saved from the host `worker-us-east1-b-1`. You can view the list of profiles from the node for the last 15 minutes using this command:

```console
perforator list profiles --node-id "worker-us-east1-b-1" -s "now-15m"
```

## Save profiles from a pod

Create a microscope to save profiles from a pod for 15 minutes, starting in 30 minutes:

```console
perforator microscope create --pod-id "perforator-storage-production-73" --duration "15m" --start-time "now+30m"
```

View the profiles from the pod for the last 15 minutes:

```console
perforator list profiles --pod-id "perforator-storage-production-73" -s "now-15m"
```

## List created microscopes

View your created microscopes:

```console
perforator microscope list
```


# Collecting a Local Profile

Perforator can collect ad-hoc profiles on a local machine.

## Prerequisites

To work, you need:
1. A recent Linux kernel. Minimum 5.4, preferably 5.15. You can check the kernel version using `uname -r`.
2. Full root access. Perforator requires `CAP_SYS_ADMIN` because it runs an eBPF program capable of reading any state from the kernel or userspace. (run with `sudo`)

## Collect profile of a process and save to a pprof file

```console
perforator record --format pprof -p <pid> --duration 1m --output profile.pprof
```

View `profile.pprof` file.

## Start a subprocess and collect its flamegraph

```console
perforator record --duration 1m -o ./flame.svg -- ls
```

View `flame.svg` file.

## Collect profile of a process and serve a flamegraph on localhost:9000

```console
perforator record -p <pid> --duration 1m --serve ":9000"
```

View the flamegraph at `http://localhost:9000` in your browser.

## Collect profile of a whole system and serve a flamegraph on localhost:9000

```console
perforator record -a --duration 1m --serve ":9000"
```

View the flamegraph at `http://localhost:9000` in your browser.

## Collect profile of a whole system and save flamegraph SVG to file.

```console
perforator record -a --duration 1m --output flame.svg
```

View `flame.svg` file.


# JVM-based language support

This page documents support for Java and other JVM-based languages.

## Requirements

Following requirements must be met:

* HotSpot JVM version 17 or newer is used.

* JVM is running with `-XX:+PreserveFramePointer` flag.

* Perfmap-based symbolization is enabled for the JVM process, and [`java` option](../perfmap.md#configuration-java) is enabled as well.

## Automatic perfmap generation for JVM {#jvm}

When enabled, Perforator can automatically instruct a JVM process to generate perfmap. Internally, agent will use [JDK Attach API](https://docs.oracle.com/en/java/javase/21/docs/api/jdk.attach/module-summary.html) to periodically execute equivalent to the following command
```bash
jcmd ${PID} Compiler.perfmap
```

{% note warning %}

Attach API is an OpenJDK extension. It may be unavailable in other implementations of the JVM.

{% endnote %}


# Languages supported by Perforator

Perforator supports different kinds of programming languages and language runtimes. This page describes the current status of language support.

There are different tiers of support for different languages and runtimes:
- Tier 0: C++. This is our primary focus. Perforator has some advanced features for C++ that are not yet available for other languages.
- Tier 1: Native languages like Rust, C, Go. These should work fine without any configuration in most cases (the only requirement is debug symbols availability).
- Tier 2: Python and Java. We have experimental support for these, which have known limitations and runtime requirements.

## Native languages
Support for native (compiled to machine code) languages is the most important target for the Perforator. We strive to implement this as good as possible without complex requirements on profiled binaries like `-fno-omit-frame-pointer` switch. The only requirements are:
- The presence of debugging symbols either in production builds or separate debug info providers like debuginfod. It is simply not possible to map raw instructions to source code without debug symbols.
- The presence of `.eh_frame` section which is generated by default by modern compilers. This section is used by exception runtimes in some popular native languages, but executables without exceptions also have this section unless switch `-fno-asynchronous-unwind-tables` is specified.

Incomplete list of the native languages we tested Perforator on successfully:

### C & C++
Initially Perforator was developed to profile C++, our main language for high-performance services inside Yandex. We profile thousands of C & C++ binaries in different configurations.

### Golang
Golang is a very easy language for profiling due to the always-available frame pointers [since 2016](https://github.com/golang/go/issues/15840). The only requirement for profiling is debug symbols availability.

### Rust
We have not tested Rust profiling as extensively as C++, but a few tests show that Rust is profiled correctly. The only difference from C / C++ is Rust-specific function name mangling which is not correctly parsed in some cases. We are going to address this issue soon.

## Interpreted languages
### Python
Perforator supports Python with some additional requirements on the runtime. See [this page](./python/profiling.md) for more info. As for now, other interpreted languages are not supported. Please file an issue on GitHub if you have a language you think is worth supporting.

## JIT-compiled languages
### Java
We have an experimental implementation of the [Java](./java/) profiler which requires users to set some flags for the JVM runtime. We are working on zero-configuration Java support.

## Other languages
Probably a lot of less popular native languages can be profiled out of the box using either frame pointers or `.eh_frame`. We have seen valid profiles from Zig applications, for example. The best way to check whether your language/application is supported is to profile it.

Many interpreted & JIT-compiled languages can be profiled using [perf-pid.map](https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/Documentation/jit-interface.txt?h=perf-tools-next) JIT interface initially developed for perf. Perforator natively supports perfmaps, so when you enable them on the application side it should just work. Search for words like "perf", "frame pointers" and "perf-pid.map" in your language's documentations. For example, [NodeJS](https://nodejs.org/en/learn/diagnostics/poor-performance/using-linux-perf), [PHP](https://wiki.php.net/rfc/jit), and [Python](https://docs.python.org/3/c-api/perfmaps.html) can be profiled like this. See [perfmap reference](../perfmap.md) for more details.

## Language support policy
While we obviously cannot support every language, we are open to contributions and suggestions. Please file an issue on GitHub to describe your use case.


JSON writer with no external dependencies, producing output
where HTML special characters are always escaped.

Use it like this:

    #include <library/cpp/json/writer/json.h>
    ...

    NJsonWriter::TBuf json;
    json.BeginList()
        .WriteString("<script>")
        .EndList();
    Cout << json.Str(); // output: ["\u003Cscript\u003E"]

For compatibility with legacy formats where object keys
are not quoted, use CompatWriteKeyWithoutQuotes:

    NJsonWriter::TBuf json;
    json.BeginObject()
        .CompatWriteKeyWithoutQuotes("r").WriteInt(1)
        .CompatWriteKeyWithoutQuotes("n").WriteInt(0)
    .EndObject();
    Cout << json.Str(); // output: {r:1,n:0}


Эта библиотека содержит две раплизации InitContext для TLogBackendCreator.

TLogBackendCreatorInitContextYConf работает с YandexConfig (library/cpp/yconf).

TLogBackendCreatorInitContextConfig работает с NConfig::TConfig (library/cpp/config).


----------- YourBenchmarkName ---------------
 samples:       403
 iterations:    100576
 iterations hr:    101K
 run time:      5.016924443
 per iteration: 119265.0829 (119K) cycles

samples – сколько раз была вызвана функция с бенчмарком
iterations – сколько всего итераций было сделано. при каждом вызове функции бенчмарка может быть разное кол-во итераций
run time – скольку времени исполнялся бенчмарк
per iteration – сколько времени (процессорных тактов) ушло на одну итерацию, в идеале это то сколько работает функция для которой ты написал бенчмарк


# Gtest support in Arcadia

Gtest wrapper that reports results exactly how Arcadia CI wants it.

How to use:

- use `GTEST` in your `ya.make`;
- include `gtest.h` from this library. Don't include `<gtest/gtest.h>` and `<gmock/gmock.h>` directly because then you'll not get our extensions, including pretty printers for util types;
- write tests and enjoy.


# Extensions for Gtest and Gmock

Extensions that enable better support of util types in gtest and gmock: pretty printers, matchers, some convenience macros.

If you're using `GTEST`, include `library/cpp/testing/gtest/gtest.h` and it will automatically enable these extensions. This is the preferred way to include gtest and gmock as opposed to including gtest, gmock and extensions directly. It eliminates chances of forgetting to include extensions.


# Glue for `GTEST` macro

Provides `main` function. This library is automatically linked into any test binary that uses `GTEST`. This way, you don't have to implement `main` yourself every time you write a test target.


# Hook for google benchmark and gtest

Y_TEST_HOOK_BEFORE_INIT - вызывается перед инициализацией соответствующего фреймворка
Y_TEST_HOOK_BEFORE_RUN - вызывается перед запуском тестов
Y_TEST_HOOK_AFTER_RUN - вызывается всегда после завершения выполнения тестов,
                        если этап инициализации был успешным

## Примеры:

```
Y_TEST_HOOK_BEFORE_INIT(SetupMyApp) {
  // ваш код для выполнения перед инициализацией фреймворка
}

Y_TEST_HOOK_BEFORE_RUN(InitMyApp) {
  // ваш код для выполнения перед запуском тестов
}

Y_TEST_HOOK_AFTER_RUN(CleanMyApp) {
  // ваш код для выполнения после завершения тестов
}
```

## Тесты:
тесты лаунчерах соотвествующих фреймворков (gtest, gbenchmark и unittest)


See:
https://docs.yandex-team.ru/arcadia-cpp/cookbook/concurrency


# [library/cpp/yt/string]

## Structure

This library provides ways of printing data structures as well as some helpers methods to work with strings.

### `TStringBuilder` [library/cpp/yt/string/string_builder.h]

String formatter with dynamic buffer which supports strings, chars and arbitrary Format expressions (see below).
```cpp
TString HelloWorld()
{
    TStringBuilder builder;
    builder.AppendString("Hello,"); // <- Dynamic allocation of max(minSize, str.len()) bytes.
    builder.AppendChar(' ');
    builder.AppendFormat("World %v!", 42); // See Format section below
    return builder.Flush(); // Hello, World 42!
}
```

### `TRawFormatter` [library/cpp/yt/string/raw_formatter.h]

String formatter with static buffer which is stored on stack frame. Supports strings, chars, numbers and guids.
```cpp
TString HelloWorld(TGuid guid) // guid = "1-2-3-4"
{
    TRawFormatter<42> builder; // <- Buffer size is set right away. Never allocates.
    builder.AppendString("Hello");
    builder.AppendChar(' ');
    builder.AppendString("World ");
    builder.AppendGuid(guid);
    builder.AppendChar(' ');
    builder.AppendNumber(42);
    builder.AppendChar('!');
    return TString(builder.GetBuffer()); // Hello World 1-2-3-4 42!
}
```

Attempt to append string which results in buffer overflow truncates the string
```cpp
TString LongMessage()
{
    TRawFormatter<7> builder;
    builder.AppendString("Hello World!");
    return TString(builder.GetBuffer()); // Hello W
}
```

### `Format` [library/cpp/yt/string/format.h]

Universal way of generating strings in a fashion similar to `printf` with flags support.
```cpp
Format("Hello, World %d!", 42); // Hello, World 42!
```

Currently all std flags are supported via fallback to `printf` (Note: this is subject to change. We might remove support of the majority of flags in the future to reduce complexity on the user side). We additionally support "Universal" conversion specifier -- "v" which prints value in a certain default way.
```cpp
Format("Hello, World %v!", 42); // Hello, World 42!

Format("Value is %v", "MyValue"); // Value is MyValue

Format("Vector is %v", std::vector{1, 2, 3}); // Vector is [1, 2, 3]
```

"l" specifier can be applied to enums and bools to emit them as their lowercase versions:
```cpp
DEFINE_ENUM(EMyEnum,
((MyValue1)     (42))
((AnotherValue) (41))
);

Format("%v", true); // True
Format("%v", EMyEnum::MyValue1); // MyValue1

Format("%lv", true); // true
Format("%lv", EMyEnum::MyValue); // my_value1
```

"q" and "Q" specifiers wrap output into quotations marks ' and " respectively. If the same quotation marks are detected inside the formattable value, they are replaced by their "\\"-version:
```cpp
Format("%Qv", true); // "True"
Format("%qv", true); // 'true'
Format("%Qv", "\"Hello World\""); // "\"Hello World\""

// std::array{"MyValue1", "AnotherValue"}
auto names = TEnumTraits<EMyEnum>::GetDomainNames();
Format("%Qv", names); // "[\"MyValue1\", \"AnotherValue\"]"
```

`FormatterWrapper` allows conditional writes into the string:
```cpp
NYT::Format(
    "Value is %v%v",
    42,
    MakeFormatterWrapper([&] (auto* builder) {
        If (PossiblyMissingInfo_) {
            builder->AppendString(", PossiblyMissingInfo: ");
            FormatValue(builder, PossiblyMissingInfo_, "v");
        }
    }));
```

`FormatVector` allows treating range of values as a generator coroutine returning values for each placeholder:
```cpp
FormatVector("One: %v, Two: %v, Three: %v", {1, 2, 3})
// One: 1, Two: 2, Three: 3
```

### Customising Format

By default type is not Formattable:
```cpp
struct TMyStruct
{ };

static_assert(!CFormattable<TMyStruct>);
Format("%v", TMyStruct{}); // <- Results in CE
```
Compiler error looks like this:
```cpp
ROOT/library/cpp/yt/string/unittests/format_ut.cpp:46:36: error: call to consteval function 'NYT::TBasicStaticFormat<NYT::(anonymous namespace)::TMyStruct>::TBasicStaticFormat<char[3]>' is not a constant expression
[[maybe_unused]] auto val = Format("%v", TMyStruct{});
                                   ^
ROOT/library/cpp/yt/string/format_string-inl.h:38:17: note: non-constexpr function 'CrashCompilerClassIsNotFormattable<NYT::(anonymous namespace)::TMyStruct>' cannot be used in a constant expression
                CrashCompilerClassIsNotFormattable<std::tuple_element_t<Idx, TTuple>>();
                ^
ROOT/library/cpp/yt/string/format_string-inl.h:36:10: note: in call to '&[] {
    if (!CFormattable<std::tuple_element_t<0UL, TTuple>>) {
        CrashCompilerClassIsNotFormattable<std::tuple_element_t<0UL, TTuple>>();
    }
}->operator()()'
...
```

First line contains the source location where the error occured. Second line contains the function name `CrashCompilerClassIsNotFormattable<NYT::(anonymous namespace)::TMyStruct>` which name is the error and template argument is the errorneos type. There are some more lines which would contain incomprehensible garbage --- don't bother reading it. Other compiler errors generated by static analyser (see below) follow the same structure.

In order to support printing custom type, one must create an overload of `FormatValue` function. If everything is done correctly, concept `CFormattable<T>` should be satisfied and the value printed accordingly.

```cpp
struct TMyStruct
{ };

void FormatValue(TStringBuilderBase* builder, const TMyStruct& /*val*/, TStringBuf /*spec*/)
{
    builder->AppendString(TStringBuf("TMyStruct"));
}

static_assert(CFormattable<TMyStruct>);
Format("%v", TMyStruct{}); // "TMyStruct"
```

First argument is already known builder (technically, the part of builder which can be written into, but not flushed). Second argument is the value to be formatted and the `spec` is the set of flags to be applied during the formatting. Spec must not be empty or contain the introductory symbol '%'!
```cpp
struct TMyPair
{
    int Key;
    TString Value;
};

void FormatValue(TStringBuilderBase* builder, const TMyPair& pair, TStringBuf spec)
{
    // We shall support an extra flag -- 'k' which forces pair to be printed differently
    bool concat = false;

    for (auto c : spec) {
        concat |= (c == 'k');
    }

    if (concat) {
        builder->AppendFormat("%v_%v", Key, Value);
    } else {
        builder->AppendFormat("{%v: %v}", Key, Value);
    }
};

// Required for static analysis (see section below)
// If you don't add extra specifiers you can ignore this part.
template <>
struct NYT::TFormatArg<TMyPair>
    : public NYT::TFormatArgBase
{
    static constexpr auto ConversionSpecifiers = TFormatArgBase::ConversionSpecifiers;

    static constexpr auto FlagSpecifiers =
        TFormatArgBase::ExtendConversion</*Hot*/ true, 1, std::array{'k'}>();
};

Format("%v", TMyPair{42, "Hello"});
// spec is "v"
// output is {42: Hello}

Format("%kv", TMyPair{42, "Hello"});
// spec is "kv"
// output is 42_Hello
```

`TRuntimeFormat` is required if you want to use a non-constexpr value as a format string:
```cpp
cosntexpr TStringBuf fmtGood1 = "Hello, %v";
const char* fmtBad = "Hello, %v";
TRuntimeFormat fmtGood2{fmtBad};

Format(fmtGood1, "World!"); // Hello, World
Format(fmdBad, "World!"); // CE --- call to consteval function is not constexpr
Format(fmtGood2, "World!"); // Hello, World
```

### Static analysis (since clang-16)

If format string can bind to `TFormatString` (that is, it is a constexpr string_view or a literal) then static analysis on supplied args is performed.

#### How to disable

Per-file: `#define YT_DISABLE_FORMAT_STATIC_ANALYSIS` (see [library/cpp/yt/string/format_string.h] for up to date macro name).

#### What is checked

Static analyser checks if the number of specifier sequences matches the number of arguments supplied. Validity of specifier sequences if checked per argument (that specifier sequence is either "%%" or starts with "%", ends with one of the conversion specifiers and contains only the flag specifiers in the middle). Lists of conversion specifiers and flags specifiers are customisation points (see [library/cpp/yt/string/format_arg.h]).

#### Customising static analysis

We have already seen that `TMyPair` additionally required specialization of `TFormatArg` struct in order to work. Unless you want to change the list of allowed specifiers, default definition of `TFormatArg<T>` will be sufficient. We want to add an extra flag specifier for `TMyPair` and thus we must specialize `NYT::TFormatArg<TMyPair>`:

```cpp
template <>
struct NYT::TFormatArg<TMyPair>
    : public NYT::TFormatArgBase // Contains default sets of specifiers and some convenience tools for customization.
{
    // Technically not required as it is present in base. Here written for exposition.
    static constexpr auto ConversionSpecifiers = TFormatArgBase::ConversionSpecifiers;

    // Adds 'k' flag to the list of default specifiers
    // 'Hot' means that we prepend specifier since we expect
    // it to be used frequently. This speeds up the compilation a little.
    static constexpr auto FlagSpecifiers =
        TFormatArgBase::ExtendConversion</*Hot*/ true, 1, std::array{'k'}>();
};
```

Now we are able to print the value as format analyser is aware of the new flag 'k'. If we wanted to, we could remove the rest of the default specifiers provided by `TFormatArgBase`, since most of them might not make any sence for your type.

You can use `TFormatArg` + `FormatValue` to fully support format decorators:
```cpp
template <class T>
struct TDecorator
{
    T Value;
};

template <class T>
struct NYT::TFormatArg<TDecorator<T>>
    : public NYT::TFormatArgBase
{
    static constexpr auto ConversionSpecifiers = TFormatArg<T>::ConversionSpecifiers;
    static constexpr auto FlagSpecifiers =
        TFormatArgBase::ExtendConversion</*Hot*/ true, 1, std::array{'D'}, /*TFrom*/ T>::ExtendConversion();
};

template <class T>
void FormatValue(NYT::TStringBuilderBase* builder, const TDecorator<T>& value, TStringBuf spec)
{
    bool append = (spec[0] == 'D');

    if (append) {
        builder->AppendString("TDecorator value: ");
        FormatValue(builder, value.Value, TStringBuf(&spec[1], spec.size() - 1));
        return;
    }

    FormatValue(builder, value.Value, spec);
}

Format("Testing: %v", TDecorator{TMyPair{42, "Hello"}});
// Testing: {42, Hello}
Format("Testing: %Dv", TDecorator{TMyPair{42, "Hello"}});
// Testing: TDecorator value: {42, Hello}
Format("Testing: %Dkv", TDecorator{TMyPair{42, "Hello"}});
// Testing: TDecorator value: 42_Hello
```

### ToString auto generation

For names inside namespaces enclosing `NYT` and `std` we automatically generate overload of `ToString` which uses `FormatValue` function if there is such a function. In examples below we assume that `CFormattable` holds true for each type:
```cpp
auto val = ToString(NYT::TMyPair{42, "Hello"}}); // Works since TMyPair comes from namespace NYT;

auto val = ToString(std::optional{NYT::TMyPair{42, "Hello"}}}); // Works since optional comes from namespace std

auto val = ToString(NYT::NOrm::::NClient::NObjects::TObjectKey{}); // Works since NOrm::::NClient::NObjects enclose namespace NYT

auto val = ToString(NMyNs::TMyType{}); // Falls back to util ToString because NMyNs encloses neither std nor NYT. Fate is unknown.

auto val = ToString(NMyNS::TMyContainer<NYT::TMyPair>{}); // Falls back to util ToString because NMyNs encloses neither std nor NYT (we don't care about template parameters). Fate is unknown.

auto val = NYT::ToString(NMyNs::TMyType{}); // Works.

auto val = NYT::ToString(NMyNS::TMyContainer<NYT::TMyPair>{}); // Also works.

{
    using ::ToString; // Irrelevant since NYT::ToString is more constrained.
    using NYT::ToString;
    auto val = ToString(NMyNS::TMyContainer<NYT::TMyPair>{}); // Also works.
}
```

### Implementing `FormatValue` via `ToString`

One thing you may attempt to do is to use already defined `ToString` method to implement `FormatValue`. There are two cases for this:
    1. You have an overload of `ToString` visible from the inside of `FormatValue` which is not the util default overload. In this case you are fine.
    2. You rely on util `ToString` overload. In this case you hit an infinite recursion loop if name of your type comes from `std` or `NYT` namespaces. We strongly recommend that you stop relying on util `ToString` and simply write `FormatValue` from scratch. Should this be impossible, use `NYT::ToStringIgnoringFormatValue(const T&)` which implements default `ToString` mechanism (via `operator <<`) from util. This method has a different name hence it will break the recursion loop.

### Format extensions.

There are some types from util and other cpp libraries which one might want to print, but we don't need them in the yt project (allegedly). Some of these dependencies are located in [library/cpp/yt/string/format_extensions]. If you don't care about granularity, simply include "library/cpp/yt/string/format_extensions/all.h" to enable support for every currently known dependency.


# xerrors
Drop-in replacement для `errors` и `golang.org/x/xerrors`, а так же функционально для `github.com/pkg/errors`.

### Локальный запуск тестов со стандартным тулчейном

Так как тесты используют фреймы стэка, необходимо тримить абсолютные пути записываемые в бинарник теста. Для этого необходимо использовать аргумент `trimpath` - `go test -v -race -gcflags=-trimpath=<absolute_path_to_arcadia_mount>`. В противном случае ваши абсолютные пути будут пролезать во фреймы и тесты будут фейлиться.


Official Helm chart.
See https://perforator.tech/docs/en/guides/helm-chart for details.


<!--
IMPORTANT: read this when updating release notes.

Here is the format used:

```
# Version/Unreleased

Fixes:

+ Description (PR or commit[1])

Enhancements:

+ Description (PR or commit)

Internal changes:

+ Description (PR or commit)
```

1: If you are sending patch to GitHub, specify PR. Otherwise (if you are sending patch to internal monorepo), leave unset and then specify Git commit.
-->

# 0.2.7

Fixes:
+ Fix perforator chart not properly importing perforator-dev dependency.

# 0.2.6

Enhancements:
+ Add perforator-dev subchart, that allows to deploy postgresql, clickhouse, minio databases (78ff60b)
+ Allow enabling/disabling certain perforator microservices (#36)
+ Allow to deploy extra objects (#36)
+ Upgrade perforator to [v0.0.4](https://github.com/yandex/perforator/releases/tag/v0.0.4)

Fixes:
+ Fix storage service declaration (#36)
+ Fix priority class setting placement (#36)
+ Fix password secretes were deployed after the migration jobs, which in some cases caused chart installation failure (972f4a1)

# 0.2.5

Enhancements:
+ Allow customizing environment variables for all components (a9e002e)
+ Upgrade perforator to [v0.0.3](https://github.com/yandex/perforator/releases/tag/v0.0.3) (21e0f8a)

# 0.2.4

Fixes:
+ Fix mount extra volumes (#30) (#30)

Enhancements:
+ Add default images for all components (677ca2b)
+ (experimental) Add support for automated migrations (see `.databases.{postgresql,clickhouse}.migrations` for details) (1d38dff)

Internal changes:
+ Change auto-generated CA subject (75f12f2)

# 0.2.3

Fixes:
+ Fix CA certificate recreating on helm upgrade (6b9207e)

Enhancements:
+ Support custom CA for agent-storage communication (6b9207e)

# 0.2.2

Enhancements:
+ Support custom CA for ClickHouse, PostgresSQL and object storage (#25)

# 0.2.1

Fixes:
+ Fix dnsPolicy overrides (#16)
+ Fix duplicated spec.tls in ingresses (#17)

# 0.2.0

Fixes:
+ Use prefix match for ingresses (e492fd8)

Enhancements:
+ Always recreate pods on helm upgrade (4ccce88)

# 0.1.0

Initial release


# Changelog

## [0.3.0](https://github.com/googleapis/google-cloud-go/compare/auth/v0.2.2...auth/v0.3.0) (2024-04-23)

### Features

* **auth/httptransport:** Add ability to customize transport ([#10023](https://github.com/googleapis/google-cloud-go/issues/10023)) ([72c7f6b](https://github.com/googleapis/google-cloud-go/commit/72c7f6bbec3136cc7a62788fc7186bc33ef6c3b3)), refs [#9812](https://github.com/googleapis/google-cloud-go/issues/9812) [#9814](https://github.com/googleapis/google-cloud-go/issues/9814)

### Bug Fixes

* **auth/credentials:** Error on bad file name if explicitly set ([#10018](https://github.com/googleapis/google-cloud-go/issues/10018)) ([55beaa9](https://github.com/googleapis/google-cloud-go/commit/55beaa993aaf052d8be39766afc6777c3c2a0bdd)), refs [#9809](https://github.com/googleapis/google-cloud-go/issues/9809)

## [0.2.2](https://github.com/googleapis/google-cloud-go/compare/auth/v0.2.1...auth/v0.2.2) (2024-04-19)

### Bug Fixes

* **auth:** Add internal opt to skip validation on transports ([#9999](https://github.com/googleapis/google-cloud-go/issues/9999)) ([9e20ef8](https://github.com/googleapis/google-cloud-go/commit/9e20ef89f6287d6bd03b8697d5898dc43b4a77cf)), refs [#9823](https://github.com/googleapis/google-cloud-go/issues/9823)
* **auth:** Set secure flag for gRPC conn pools ([#10002](https://github.com/googleapis/google-cloud-go/issues/10002)) ([14e3956](https://github.com/googleapis/google-cloud-go/commit/14e3956dfd736399731b5ee8d9b178ae085cf7ba)), refs [#9833](https://github.com/googleapis/google-cloud-go/issues/9833)

## [0.2.1](https://github.com/googleapis/google-cloud-go/compare/auth/v0.2.0...auth/v0.2.1) (2024-04-18)

### Bug Fixes

* **auth:** Default gRPC token type to Bearer if not set ([#9800](https://github.com/googleapis/google-cloud-go/issues/9800)) ([5284066](https://github.com/googleapis/google-cloud-go/commit/5284066670b6fe65d79089cfe0199c9660f87fc7))

## [0.2.0](https://github.com/googleapis/google-cloud-go/compare/auth/v0.1.1...auth/v0.2.0) (2024-04-15)

### Breaking Changes

In the below mentioned commits there were a few large breaking changes since the
last release of the module.

1. The `Credentials` type has been moved to the root of the module as it is
   becoming the core abstraction for the whole module.
2. Because of the above mentioned change many functions that previously
   returned a `TokenProvider` now return `Credentials`. Similarly, these
   functions have been renamed to be more specific.
3. Most places that used to take an optional `TokenProvider` now accept
   `Credentials`. You can make a `Credentials` from a `TokenProvider` using the
   constructor found in the `auth` package.
4. The `detect` package has been renamed to `credentials`. With this change some
   function signatures were also updated for better readability.
5. Derivative auth flows like `impersonate` and `downscope` have been moved to
   be under the new `credentials` package.

Although these changes are disruptive we think that they are for the best of the
long-term health of the module. We do not expect any more large breaking changes
like these in future revisions, even before 1.0.0. This version will be the
first version of the auth library that our client libraries start to use and
depend on.

### Features

* **auth/credentials/externalaccount:** Add default TokenURL ([#9700](https://github.com/googleapis/google-cloud-go/issues/9700)) ([81830e6](https://github.com/googleapis/google-cloud-go/commit/81830e6848ceefd055aa4d08f933d1154455a0f6))
* **auth:** Add downscope.Options.UniverseDomain ([#9634](https://github.com/googleapis/google-cloud-go/issues/9634)) ([52cf7d7](https://github.com/googleapis/google-cloud-go/commit/52cf7d780853594291c4e34302d618299d1f5a1d))
* **auth:** Add universe domain to grpctransport and httptransport ([#9663](https://github.com/googleapis/google-cloud-go/issues/9663)) ([67d353b](https://github.com/googleapis/google-cloud-go/commit/67d353beefe3b607c08c891876fbd95ab89e5fe3)), refs [#9670](https://github.com/googleapis/google-cloud-go/issues/9670)
* **auth:** Add UniverseDomain to DetectOptions ([#9536](https://github.com/googleapis/google-cloud-go/issues/9536)) ([3618d3f](https://github.com/googleapis/google-cloud-go/commit/3618d3f7061615c0e189f376c75abc201203b501))
* **auth:** Make package externalaccount public ([#9633](https://github.com/googleapis/google-cloud-go/issues/9633)) ([a0978d8](https://github.com/googleapis/google-cloud-go/commit/a0978d8e96968399940ebd7d092539772bf9caac))
* **auth:** Move credentials to base auth package ([#9590](https://github.com/googleapis/google-cloud-go/issues/9590)) ([1a04baf](https://github.com/googleapis/google-cloud-go/commit/1a04bafa83c27342b9308d785645e1e5423ea10d))
* **auth:** Refactor public sigs to use Credentials ([#9603](https://github.com/googleapis/google-cloud-go/issues/9603)) ([69cb240](https://github.com/googleapis/google-cloud-go/commit/69cb240c530b1f7173a9af2555c19e9a1beb56c5))

### Bug Fixes

* **auth/oauth2adapt:** Update protobuf dep to v1.33.0 ([30b038d](https://github.com/googleapis/google-cloud-go/commit/30b038d8cac0b8cd5dd4761c87f3f298760dd33a))
* **auth:** Fix uint32 conversion ([9221c7f](https://github.com/googleapis/google-cloud-go/commit/9221c7fa12cef9d5fb7ddc92f41f1d6204971c7b))
* **auth:** Port sts expires fix ([#9618](https://github.com/googleapis/google-cloud-go/issues/9618)) ([7bec97b](https://github.com/googleapis/google-cloud-go/commit/7bec97b2f51ed3ac4f9b88bf100d301da3f5d1bd))
* **auth:** Read universe_domain from all credentials files ([#9632](https://github.com/googleapis/google-cloud-go/issues/9632)) ([16efbb5](https://github.com/googleapis/google-cloud-go/commit/16efbb52e39ea4a319e5ee1e95c0e0305b6d9824))
* **auth:** Remove content-type header from idms get requests ([#9508](https://github.com/googleapis/google-cloud-go/issues/9508)) ([8589f41](https://github.com/googleapis/google-cloud-go/commit/8589f41599d265d7c3d46a3d86c9fab2329cbdd9))
* **auth:** Update protobuf dep to v1.33.0 ([30b038d](https://github.com/googleapis/google-cloud-go/commit/30b038d8cac0b8cd5dd4761c87f3f298760dd33a))

## [0.1.1](https://github.com/googleapis/google-cloud-go/compare/auth/v0.1.0...auth/v0.1.1) (2024-03-10)

### Bug Fixes

* **auth/impersonate:** Properly send default detect params ([#9529](https://github.com/googleapis/google-cloud-go/issues/9529)) ([5b6b8be](https://github.com/googleapis/google-cloud-go/commit/5b6b8bef577f82707e51f5cc5d258d5bdf90218f)), refs [#9136](https://github.com/googleapis/google-cloud-go/issues/9136)
* **auth:** Update grpc-go to v1.56.3 ([343cea8](https://github.com/googleapis/google-cloud-go/commit/343cea8c43b1e31ae21ad50ad31d3b0b60143f8c))
* **auth:** Update grpc-go to v1.59.0 ([81a97b0](https://github.com/googleapis/google-cloud-go/commit/81a97b06cb28b25432e4ece595c55a9857e960b7))

## 0.1.0 (2023-10-18)

### Features

* **auth:** Add base auth package ([#8465](https://github.com/googleapis/google-cloud-go/issues/8465)) ([6a45f26](https://github.com/googleapis/google-cloud-go/commit/6a45f26b809b64edae21f312c18d4205f96b180e))
* **auth:** Add cert support to httptransport ([#8569](https://github.com/googleapis/google-cloud-go/issues/8569)) ([37e3435](https://github.com/googleapis/google-cloud-go/commit/37e3435f8e98595eafab481bdfcb31a4c56fa993))
* **auth:** Add Credentials.UniverseDomain() ([#8654](https://github.com/googleapis/google-cloud-go/issues/8654)) ([af0aa1e](https://github.com/googleapis/google-cloud-go/commit/af0aa1ed8015bc8fe0dd87a7549ae029107cbdb8))
* **auth:** Add detect package ([#8491](https://github.com/googleapis/google-cloud-go/issues/8491)) ([d977419](https://github.com/googleapis/google-cloud-go/commit/d977419a3269f6acc193df77a2136a6eb4b4add7))
* **auth:** Add downscope package ([#8532](https://github.com/googleapis/google-cloud-go/issues/8532)) ([dda9bff](https://github.com/googleapis/google-cloud-go/commit/dda9bff8ec70e6d104901b4105d13dcaa4e2404c))
* **auth:** Add grpctransport package ([#8625](https://github.com/googleapis/google-cloud-go/issues/8625)) ([69a8347](https://github.com/googleapis/google-cloud-go/commit/69a83470bdcc7ed10c6c36d1abc3b7cfdb8a0ee5))
* **auth:** Add httptransport package ([#8567](https://github.com/googleapis/google-cloud-go/issues/8567)) ([6898597](https://github.com/googleapis/google-cloud-go/commit/6898597d2ea95d630fcd00fd15c58c75ea843bff))
* **auth:** Add idtoken package ([#8580](https://github.com/googleapis/google-cloud-go/issues/8580)) ([a79e693](https://github.com/googleapis/google-cloud-go/commit/a79e693e97e4e3e1c6742099af3dbc58866d88fe))
* **auth:** Add impersonate package ([#8578](https://github.com/googleapis/google-cloud-go/issues/8578)) ([e29ba0c](https://github.com/googleapis/google-cloud-go/commit/e29ba0cb7bd3888ab9e808087027dc5a32474c04))
* **auth:** Add support for external accounts in detect ([#8508](https://github.com/googleapis/google-cloud-go/issues/8508)) ([62210d5](https://github.com/googleapis/google-cloud-go/commit/62210d5d3e56e8e9f35db8e6ac0defec19582507))
* **auth:** Port external account changes ([#8697](https://github.com/googleapis/google-cloud-go/issues/8697)) ([5823db5](https://github.com/googleapis/google-cloud-go/commit/5823db5d633069999b58b9131a7f9cd77e82c899))

### Bug Fixes

* **auth/oauth2adapt:** Update golang.org/x/net to v0.17.0 ([174da47](https://github.com/googleapis/google-cloud-go/commit/174da47254fefb12921bbfc65b7829a453af6f5d))
* **auth:** Update golang.org/x/net to v0.17.0 ([174da47](https://github.com/googleapis/google-cloud-go/commit/174da47254fefb12921bbfc65b7829a453af6f5d))


# auth

This module is currently EXPERIMENTAL and under active development. It is not
yet intended to be used.


# Changes

## [1.1.7](https://github.com/googleapis/google-cloud-go/compare/iam/v1.1.6...iam/v1.1.7) (2024-03-14)

### Bug Fixes

* **iam:** Update protobuf dep to v1.33.0 ([30b038d](https://github.com/googleapis/google-cloud-go/commit/30b038d8cac0b8cd5dd4761c87f3f298760dd33a))

## [1.1.6](https://github.com/googleapis/google-cloud-go/compare/iam/v1.1.5...iam/v1.1.6) (2024-01-30)

### Bug Fixes

* **iam:** Enable universe domain resolution options ([fd1d569](https://github.com/googleapis/google-cloud-go/commit/fd1d56930fa8a747be35a224611f4797b8aeb698))

## [1.1.5](https://github.com/googleapis/google-cloud-go/compare/iam/v1.1.4...iam/v1.1.5) (2023-11-01)

### Bug Fixes

* **iam:** Bump google.golang.org/api to v0.149.0 ([8d2ab9f](https://github.com/googleapis/google-cloud-go/commit/8d2ab9f320a86c1c0fab90513fc05861561d0880))

## [1.1.4](https://github.com/googleapis/google-cloud-go/compare/iam/v1.1.3...iam/v1.1.4) (2023-10-26)

### Bug Fixes

* **iam:** Update grpc-go to v1.59.0 ([81a97b0](https://github.com/googleapis/google-cloud-go/commit/81a97b06cb28b25432e4ece595c55a9857e960b7))

## [1.1.3](https://github.com/googleapis/google-cloud-go/compare/iam/v1.1.2...iam/v1.1.3) (2023-10-12)

### Bug Fixes

* **iam:** Update golang.org/x/net to v0.17.0 ([174da47](https://github.com/googleapis/google-cloud-go/commit/174da47254fefb12921bbfc65b7829a453af6f5d))

## [1.1.2](https://github.com/googleapis/google-cloud-go/compare/iam/v1.1.1...iam/v1.1.2) (2023-08-08)

### Documentation

* **iam:** Minor formatting ([b4349cc](https://github.com/googleapis/google-cloud-go/commit/b4349cc507870ff8629bbc07de578b63bb889626))

## [1.1.1](https://github.com/googleapis/google-cloud-go/compare/iam/v1.1.0...iam/v1.1.1) (2023-06-20)

### Bug Fixes

* **iam:** REST query UpdateMask bug ([df52820](https://github.com/googleapis/google-cloud-go/commit/df52820b0e7721954809a8aa8700b93c5662dc9b))

## [1.1.0](https://github.com/googleapis/google-cloud-go/compare/iam/v1.0.1...iam/v1.1.0) (2023-05-30)

### Features

* **iam:** Update all direct dependencies ([b340d03](https://github.com/googleapis/google-cloud-go/commit/b340d030f2b52a4ce48846ce63984b28583abde6))

## [1.0.1](https://github.com/googleapis/google-cloud-go/compare/iam/v1.0.0...iam/v1.0.1) (2023-05-08)

### Bug Fixes

* **iam:** Update grpc to v1.55.0 ([1147ce0](https://github.com/googleapis/google-cloud-go/commit/1147ce02a990276ca4f8ab7a1ab65c14da4450ef))

## [1.0.0](https://github.com/googleapis/google-cloud-go/compare/iam/v0.13.0...iam/v1.0.0) (2023-04-04)

### Features

* **iam:** Promote to GA ([#7627](https://github.com/googleapis/google-cloud-go/issues/7627)) ([b351906](https://github.com/googleapis/google-cloud-go/commit/b351906a10e17a02d7f7e2551bc1585fd9dc3742))

## [0.13.0](https://github.com/googleapis/google-cloud-go/compare/iam/v0.12.0...iam/v0.13.0) (2023-03-15)

### Features

* **iam:** Update iam and longrunning deps ([91a1f78](https://github.com/googleapis/google-cloud-go/commit/91a1f784a109da70f63b96414bba8a9b4254cddd))

## [0.12.0](https://github.com/googleapis/google-cloud-go/compare/iam/v0.11.0...iam/v0.12.0) (2023-02-17)

### Features

* **iam:** Migrate to new stubs ([a61ddcd](https://github.com/googleapis/google-cloud-go/commit/a61ddcd3041c7af4a15109dc4431f9b327c497fb))

## [0.11.0](https://github.com/googleapis/google-cloud-go/compare/iam/v0.10.0...iam/v0.11.0) (2023-02-16)

### Features

* **iam:** Start generating proto stubs ([970d763](https://github.com/googleapis/google-cloud-go/commit/970d763531b54b2bc75d7ff26a20b6e05150cab8))

## [0.10.0](https://github.com/googleapis/google-cloud-go/compare/iam/v0.9.0...iam/v0.10.0) (2023-01-04)

### Features

* **iam:** Add REST client ([06a54a1](https://github.com/googleapis/google-cloud-go/commit/06a54a16a5866cce966547c51e203b9e09a25bc0))

## [0.9.0](https://github.com/googleapis/google-cloud-go/compare/iam/v0.8.0...iam/v0.9.0) (2022-12-15)

### Features

* **iam:** Rewrite iam sigs and update proto import ([#7137](https://github.com/googleapis/google-cloud-go/issues/7137)) ([ad67fa3](https://github.com/googleapis/google-cloud-go/commit/ad67fa36c263c161226f7fecbab5221592374dca))

## [0.8.0](https://github.com/googleapis/google-cloud-go/compare/iam/v0.7.0...iam/v0.8.0) (2022-12-05)

### Features

* **iam:** Start generating and refresh some libraries ([#7089](https://github.com/googleapis/google-cloud-go/issues/7089)) ([a9045ff](https://github.com/googleapis/google-cloud-go/commit/a9045ff191a711089c37f1d94a63522d9939ce38))

## [0.7.0](https://github.com/googleapis/google-cloud-go/compare/iam/v0.6.0...iam/v0.7.0) (2022-11-03)

### Features

* **iam:** rewrite signatures in terms of new location ([3c4b2b3](https://github.com/googleapis/google-cloud-go/commit/3c4b2b34565795537aac1661e6af2442437e34ad))

## [0.6.0](https://github.com/googleapis/google-cloud-go/compare/iam/v0.5.0...iam/v0.6.0) (2022-10-25)

### Features

* **iam:** start generating stubs dir ([de2d180](https://github.com/googleapis/google-cloud-go/commit/de2d18066dc613b72f6f8db93ca60146dabcfdcc))

## [0.5.0](https://github.com/googleapis/google-cloud-go/compare/iam/v0.4.0...iam/v0.5.0) (2022-09-28)

### Features

* **iam:** remove ListApplicablePolicies ([52dddd1](https://github.com/googleapis/google-cloud-go/commit/52dddd1ed89fbe77e1859311c3b993a77a82bfc7))

## [0.4.0](https://github.com/googleapis/google-cloud-go/compare/iam/v0.3.0...iam/v0.4.0) (2022-09-06)

### Features

* **iam:** start generating apiv2 ([#6605](https://github.com/googleapis/google-cloud-go/issues/6605)) ([a6004e7](https://github.com/googleapis/google-cloud-go/commit/a6004e762f782869cd85688937475744f7b17e50))

## [0.3.0](https://github.com/googleapis/google-cloud-go/compare/iam/v0.2.0...iam/v0.3.0) (2022-02-23)

### Features

* **iam:** set versionClient to module version ([55f0d92](https://github.com/googleapis/google-cloud-go/commit/55f0d92bf112f14b024b4ab0076c9875a17423c9))

## [0.2.0](https://github.com/googleapis/google-cloud-go/compare/iam/v0.1.1...iam/v0.2.0) (2022-02-14)

### Features

* **iam:** add file for tracking version ([17b36ea](https://github.com/googleapis/google-cloud-go/commit/17b36ead42a96b1a01105122074e65164357519e))

### [0.1.1](https://www.github.com/googleapis/google-cloud-go/compare/iam/v0.1.0...iam/v0.1.1) (2022-01-14)

### Bug Fixes

* **iam:** run formatter ([#5277](https://www.github.com/googleapis/google-cloud-go/issues/5277)) ([8682e4e](https://www.github.com/googleapis/google-cloud-go/commit/8682e4ed57a4428a659fbc225f56c91767e2a4a9))

## v0.1.0

This is the first tag to carve out iam as its own module. See
[Add a module to a multi-module repository](https://github.com/golang/go/wiki/Modules#is-it-possible-to-add-a-module-to-a-multi-module-repository).


# IAM API

[![Go Reference](https://pkg.go.dev/badge/cloud.google.com/go/iam.svg)](https://pkg.go.dev/cloud.google.com/go/iam)

Go Client Library for IAM API.

## Install

```bash
go get cloud.google.com/go/iam
```

## Stability

The stability of this module is indicated by SemVer.

However, a `v1+` module may have breaking changes in two scenarios:

* Packages with `alpha` or `beta` in the import path
* The GoDoc has an explicit stability disclaimer (for example, for an experimental feature).

## Go Version Support

See the [Go Versions Supported](https://github.com/googleapis/google-cloud-go#go-versions-supported)
section in the root directory's README.

## Authorization

See the [Authorization](https://github.com/googleapis/google-cloud-go#authorization)
section in the root directory's README.

## Contributing

Contributions are welcome. Please, see the [CONTRIBUTING](https://github.com/GoogleCloudPlatform/google-cloud-go/blob/main/CONTRIBUTING.md)
document for details.

Please note that this project is released with a Contributor Code of Conduct.
By participating in this project you agree to abide by its terms. See
[Contributor Code of Conduct](https://github.com/GoogleCloudPlatform/google-cloud-go/blob/main/CONTRIBUTING.md#contributor-code-of-conduct)
for more information.


# Internal

This directory contains internal code for cloud.google.com/go packages.

## .repo-metadata-full.json

`.repo-metadata-full.json` contains metadata about the packages in this repo. It
is generated by `internal/gapicgen/generator`. It's processed by external tools
to build lists of all of the packages.

Don't make breaking changes to the format without consulting with the external
tools.

One day, we may want to create individual `.repo-metadata.json` files next to
each package, which is the pattern followed by some other languages. External
tools would then talk to pkg.go.dev or some other service to get the overall
list of packages and use the `.repo-metadata.json` files to get the additional
metadata required. For now, `.repo-metadata-full.json` includes everything.

### Updating OwlBot SHA

You may want to manually update the which version of the post-processor will be
used -- to do this you need to update the SHA in the OwlBot lock file.

See the [postprocessor/README](postprocessor/README.md) for detailed
instructions.

*Note*: OwlBot will eventually open a pull request to update this value if it
discovers a new version of the container.


# Changes

## [0.5.6](https://github.com/googleapis/google-cloud-go/compare/longrunning/v0.5.5...longrunning/v0.5.6) (2024-03-14)

### Bug Fixes

* **longrunning:** Update protobuf dep to v1.33.0 ([30b038d](https://github.com/googleapis/google-cloud-go/commit/30b038d8cac0b8cd5dd4761c87f3f298760dd33a))

## [0.5.5](https://github.com/googleapis/google-cloud-go/compare/longrunning/v0.5.4...longrunning/v0.5.5) (2024-01-30)

### Bug Fixes

* **longrunning:** Enable universe domain resolution options ([fd1d569](https://github.com/googleapis/google-cloud-go/commit/fd1d56930fa8a747be35a224611f4797b8aeb698))

## [0.5.4](https://github.com/googleapis/google-cloud-go/compare/longrunning/v0.5.3...longrunning/v0.5.4) (2023-11-01)

### Bug Fixes

* **longrunning:** Bump google.golang.org/api to v0.149.0 ([8d2ab9f](https://github.com/googleapis/google-cloud-go/commit/8d2ab9f320a86c1c0fab90513fc05861561d0880))

## [0.5.3](https://github.com/googleapis/google-cloud-go/compare/longrunning/v0.5.2...longrunning/v0.5.3) (2023-10-26)

### Bug Fixes

* **longrunning:** Update grpc-go to v1.59.0 ([81a97b0](https://github.com/googleapis/google-cloud-go/commit/81a97b06cb28b25432e4ece595c55a9857e960b7))

## [0.5.2](https://github.com/googleapis/google-cloud-go/compare/longrunning/v0.5.1...longrunning/v0.5.2) (2023-10-12)

### Bug Fixes

* **longrunning:** Update golang.org/x/net to v0.17.0 ([174da47](https://github.com/googleapis/google-cloud-go/commit/174da47254fefb12921bbfc65b7829a453af6f5d))

## [0.5.1](https://github.com/googleapis/google-cloud-go/compare/longrunning/v0.5.0...longrunning/v0.5.1) (2023-06-20)

### Bug Fixes

* **longrunning:** REST query UpdateMask bug ([df52820](https://github.com/googleapis/google-cloud-go/commit/df52820b0e7721954809a8aa8700b93c5662dc9b))

## [0.5.0](https://github.com/googleapis/google-cloud-go/compare/longrunning/v0.4.2...longrunning/v0.5.0) (2023-05-30)

### Features

* **longrunning:** Update all direct dependencies ([b340d03](https://github.com/googleapis/google-cloud-go/commit/b340d030f2b52a4ce48846ce63984b28583abde6))

## [0.4.2](https://github.com/googleapis/google-cloud-go/compare/longrunning/v0.4.1...longrunning/v0.4.2) (2023-05-08)

### Bug Fixes

* **longrunning:** Update grpc to v1.55.0 ([1147ce0](https://github.com/googleapis/google-cloud-go/commit/1147ce02a990276ca4f8ab7a1ab65c14da4450ef))

## [0.4.1](https://github.com/googleapis/google-cloud-go/compare/longrunning/v0.4.0...longrunning/v0.4.1) (2023-02-14)

### Bug Fixes

* **longrunning:** Properly parse errors with apierror ([#7392](https://github.com/googleapis/google-cloud-go/issues/7392)) ([e768e48](https://github.com/googleapis/google-cloud-go/commit/e768e487e10b197ba42a2339014136d066190610))

## [0.4.0](https://github.com/googleapis/google-cloud-go/compare/longrunning/v0.3.0...longrunning/v0.4.0) (2023-01-04)

### Features

* **longrunning:** Add REST client ([06a54a1](https://github.com/googleapis/google-cloud-go/commit/06a54a16a5866cce966547c51e203b9e09a25bc0))

## [0.3.0](https://github.com/googleapis/google-cloud-go/compare/longrunning/v0.2.1...longrunning/v0.3.0) (2022-11-03)

### Features

* **longrunning:** rewrite signatures in terms of new location ([3c4b2b3](https://github.com/googleapis/google-cloud-go/commit/3c4b2b34565795537aac1661e6af2442437e34ad))

## v0.1.0

Initial release.


# longrunning

[![Go Reference](https://pkg.go.dev/badge/cloud.google.com/go/longrunning.svg)](https://pkg.go.dev/cloud.google.com/go/longrunning)

A helper library for working with long running operations.

## Install

```bash
go get cloud.google.com/go/longrunning
```

## Go Version Support

See the [Go Versions Supported](https://github.com/googleapis/google-cloud-go#go-versions-supported)
section in the root directory's README.

## Contributing

Contributions are welcome. Please, see the [CONTRIBUTING](https://github.com/GoogleCloudPlatform/google-cloud-go/blob/main/CONTRIBUTING.md)
document for details.

Please note that this project is released with a Contributor Code of Conduct.
By participating in this project you agree to abide by its terms. See
[Contributor Code of Conduct](https://github.com/GoogleCloudPlatform/google-cloud-go/blob/main/CONTRIBUTING.md#contributor-code-of-conduct)
for more information.