---
abstract: |
    We investigate in the paper general (not necessarily definite) canonical systems of differential equation in the framework of extension theory of symmetric linear relations. For this aim we first introduce the new notion of a boundary relation $\G:\gH^2\to\HH$ for $A^*$, where $\gH$ is a Hilbert space, $A$ is a symmetric linear relation in $\gH, \cH_0$ is a boundary Hilbert space and $\cH_1$ is a subspace in $\cH_0$. Unlike known concept of a boundary relation (boundary triplet) for $A^*$ our definition of $\G$ is applicable to relations $A$ with possibly unequal deficiency indices $n_\pm(A)$. Next we develop the known results on minimal and maximal relations induced by the general canonical system $ J y'(t)-B(t)y(t)=\D (t)f(t)$ on an interval $\cI=(a,b),\; -\infty\leq a<b\leq\infty $ and then by using a special (so called decomposing) boundary relation for $\Tma$ we describe in terms of boundary conditions proper extensions of $\Tmi$ in the case of the regular endpoint $a$ and arbitrary (possibly unequal) deficiency indices $n_\pm
    (\Tmi)$. If the system is definite, then decomposing boundary relation $\G$ turns into the decomposing boundary triplet $\Pi=\bt$ for $\Tma$. Using such a triplet we show that self-adjoint decomposing boundary conditions exist only for Hamiltonian systems; moreover, we describe all such conditions in the compact form. These results are generalizations of the known results by Rofe-Beketov on regular differential operators. We characterize also all maximal dissipative and accumulative separated boundary conditions, which exist for arbitrary (not necessarily Hamiltonian) definite canonical systems.
address: |
    Department of Calculus\
    Lugans’k National University\
    2 Oboronna, Lugans’k, 91011\
    Ukraine
author:
- Vadim Mogilevskii
title: 'Boundary relations and boundary conditions for general (not necessarily definite) canonical systems with possibly unequal deficiency indices'
---

Introduction
============

Assume that $\gH$ is a Hilbert space, $A$ is a closed symmetric linear relation in $\gH$ and $A^*$ is the adjoint linear relation of $A$. Moreover, denote by $[\gH_1, \gH_2]$ the set of all bounded operators between $\gH_1$ and $\gH_2$ and let $[\gH]=[\gH, \gH]$.

Recall [@GorGor; @Mal92] that a triplet $\Pi=\bt $, where $\cH$ is an auxiliary Hilbert space and $\G_0,\G_1: A^*\to \cH$ are (boundary) linear maps, is called a boundary triplet for $A^*$ if the map $\G:=(\G_0\;\;\G_1)^\top:A^*\to \cH\oplus\cH$ is surjective and the following ”abstract Green’s identity” holds $$\label{1.1}
(f',g)-(f,g')=(\G_1  \hat f, \G_0 \hat g)-(\G_0\hat f,\G_1\hat g), \quad \hat
f=\{f,f'\},\;\hat g=\{g,g'\}\in  A^*.$$

In [@DM91; @Mal92] an abstract Weyl function $M_\Pi(\l)$ was associated with a boundary triplet $\Pi$. This function is defined for all $\l\in\CR$ by the equality $$\begin{gathered}
\label{1.2}
\G_1\{f_\l,\l f_\l\}=M_\Pi(\l)\G_0\{f_\l,\l f_\l\}, \quad f_\l\in\ker (A^*-\l).\end{gathered}$$ It turns out that $M(\l)$ is a Nevanlinna $[\cH]$-valued function, i.e., $M(\l)$ is holomorphic on $\CR$, $M^*(\l)=M(\ov\l)$ and $\im\l\cd \im M(\l)\geq
0, \; \l\in\CR $. Moreover, the Nevanlinna function $M(\l)$ is uniformly strict, that is $0\in\rho (\im M(\l)), \; \l\in\CR$.

By choosing a suitable boundary triplet for a concrete problem one can parametrize various classes of extensions $\wt A\supset A$ in the most convenient form. Moreover, the Weyl function enables to characterize spectra of extensions $\wt A$ in the similar way as classical $m$-functions in the spectral theory of Sturm-Liouville operators and Jacobi matrices. These and other reasons made a boundary triplet and the corresponding Weyl function the convenient tools in the extension theory of symmetric operators (linear relations) and its applications (see [@GorGor; @DM91; @Mal92] and references therein). At the same time the theory of boundary triplets and their Weyl functions was developed in [@GorGor; @DM91; @Mal92] only for symmetric relations $A$ with equal deficiency indices $n_+(A)=n_-(A)$.

To cover the case $n_+(A)\neq n_-(A)$ we generalized in [@Mog06.2] definition of a boundary triplet as follows. Assume that $\cH_0$ is a Hilbert space, $\cH_1$ is a subspace in $\cH_0$ and $\G_j: A^*\to \cH_j, \;
j\in\{0,1\}$ are linear maps. Then a collection $\Pi=\bta$ is a boundary triplet (a $D$-triplet in terminology of [@Mog06.2]) for $A^*$ if the map $\G:=(\G_0\;\;\G_1)^\top:A^*\to \cH_0\oplus\cH_1$ is surjective and the identity $$\label{1.3}
(f',g)-(f,g')=(\G_1  \hat f, \G_0 \hat g)-(\G_0\hat f,\G_1\hat g)+i(P_2\G_0\hat
f,P_2\G_0\hat g), \quad \hat f=\{f,f'\},\;\hat g=\{g,g'\}\in  A^*$$ holds in place of (here $P_2$ is the orthoprojector in $\cH_0$ onto $\cH_2:=\cH_0\ominus\cH_1$). Associated with such a triplet $\Pi$ is the Weyl function $M_{\Pi +} (\l)$ defined for all $\l\in\bC_+$ by $$\begin{gathered}
\label{1.4}
\G_1\{f_\l,\l f_\l\}=M_{\Pi +}(\l)\G_0\{f_\l,\l f_\l\}, \quad f_\l\in\ker
(A^*-\l)\end{gathered}$$ The function $M_{\Pi +}(\l)$ is holomorphic on $\bC_+$, takes on values in $[\cH_0,\cH_1]$ and possesses a number of properties similar to those of the Weyl function . In particular, the function $M_\Pi(\l)=M_{\Pi
+}(\l)\up \cH_1$ is a uniformly strict Nevanlinna function with values in $[\cH_1]$.

A boundary triplet $\bta$ for $A^*$ enables to parametrize efficiently all proper extensions of $A$. Namely, if $\cK$ is a Hilbert space and $\{(C_0,C_1);\cK\}$ is a pair of operators $C_j\in [\cH_j,\cK]$, then the equality (the abstract boundary condition) $$\label{1.5}
 \wt A= \{ \hat f\in A^*: C_0\G_0  \hat f + C_1 \G_1 \hat f =0\}$$ defines the proper extension $A\subset \wt A\subset A^*$ and conversely each such an extension $\wt A$ admits a unique representation . Moreover, the extension $\wt A$ is maximal dissipative, maximal accumulative or self-adjoint if and only if the operator pair $\{(C_0,C_1);\cK\}$ belongs to one of the special classes introduced in [@Mog06.1].

It turns out that each boundary triplet $\Pi=\bta$ satisfies the relation $$\begin{gathered}
\label{1.5a}
\dim\cH_1=n_-(A)\leq n_+(A)=\dim\cH_0\end{gathered}$$ and, therefore, it is applicable to symmetric relations $A$ with unequal deficiency indices. Clearly, in the case $\cH_0=\cH_1=:\cH$ such a triplet $\Pi
$ and the corresponding Weyl function $M_\Pi(\l)$ turn into the similar objects in the sense of [@GorGor; @Mal92].

In [@DM95] the notion of a boundary triplet $\Pi=\bt$ for $A^*$ has been extended to the case where the corresponding Weyl function $M_\Pi(\l)$ is a (not necessarily uniformly strict) Nevanlinna function such that $0\notin\sigma_p(\im M(i))$. Next, the concepts of a boundary relation and its Weyl family which generalize the above notions of a boundary triplet and its Weyl function were introduced in [@DM06]. According to [@DM06] a boundary relation for $A^*$ is a (possibly multivalued) linear map $\G:=(\G_0\;\;\G_1)^\top:\gH^2\to \cH\oplus\cH$ such that $\dom\G$ is dense in $A^*$, the Green’s identity holds and a certain maximality condition is satisfied. The Weyl function of the boundary relation $\G$ is defined by $$\begin{gathered}
M(\l)=\{\{\G_0\{f_\l,\l f_\l\}, \G_1\{f_\l,\l f_\l\}\}: f_\l\in\ker (A^*-\l)
\}, \quad\l\in\CR\end{gathered}$$ and now it belongs to the class of Nevanlinna families; moreover, if the map $\G_0$ is surjective, then $M(\l)$ is a Nevanlinna operator function. In the paper [@DM09] the Weyl function was used for description of various classes of the exit space extensions $\wt A(=\wt A^*)\supset A$.

In the present paper the new concept of a boundary relation for $A^*$ with possibly unequal boundary spaces $\cH_0$ and $\cH_1$ is introduced. Roughly speaking this relation is a (possibly multivalued) linear map $\G:=(\G_0\;\;\G_1)^\top:\gH^2\to \cH_0\oplus\cH_1$ such that $\ov{\dom\G}=A^*$, the Green’s identity holds and a certain maximality condition is satisfied (here as before $\cH_0$ is a Hilbert space and $\cH_1$ is a subspace in $\cH_0$). Moreover, by means of the equality $$\begin{gathered}
\label{1.6}
M_+(\l)=\{\{\G_0\{f_\l,\l f_\l\}, \G_1\{f_\l,\l f_\l\}\}: f_\l\in\ker (A^*-\l)
\}, \quad\l\in\bC_+\end{gathered}$$ we associate with a boundary relation $\G$ the Weyl family $M_+(\l)$.

In the paper we study substantially the boundary relations $\BR$ with $\dim\cH_0 <\infty$. We show that in this case $\dom\G=A^*$ and there is a boundary triplet $\Pi_\G=\{\cK_0\oplus\cK_1,G_0, G_1\}$ for $A^*$ with $\cK_j\in\cH_j, \; j\in \{0,1\}$ such that $\G$ can be represented roughly speaking as a direct sum of (the graph of) the operator $G=(G_0\;\;G_1)^\top$ and $ \mul\G$.

The multivalued part $\mul \G$ which is a linear relation from $\cH_0$ to $\cH_1$ is of importance in our considerations. If $\mul \G$ is the operator, then the corresponding Weyl family $M_+(\l)$ is the holomorphic operator function with values in $[\cH_0, \cH_1]$, which admits the block representation by means of the Weyl function $M_{\Pi_\G +} (\l)$ of the boundary triplet $\Pi_\G$ and $\mul \G $. In the case $\cH_0=\cH_1=:\cH$ one has also $\cK_0=\cK_1=:\cK$ and the mentioned representation of $M(\l)$ is $$\begin{gathered}
\label{1.7}
M(\l)=\begin{pmatrix} M_{\Pi_\G}(\l) & F \cr F^* & F'\end{pmatrix}:
\cK\oplus\cK^\perp\to \cK\oplus\cK^\perp,\end{gathered}$$ where $F$ and $F'$ are the operators defined in terms of $\mul\G$. The equality shows that $M(\l)$ is a Nevanlinna function and $M_{\Pi_\G}(\l)$ is the uniformly strict part of $M(\l)$.

Note that for the boundary relation $\BR$ with $\dim\cH_0 <\infty$ the equalities $$\begin{gathered}
\dim\cH_0=n_+(A)+\dim (\mul\G), \qquad \dim\cH_1=n_-(A)+\dim (\mul\G)\end{gathered}$$ are valid (c.f. ), so that $n_-(A)\leq n_+(A)<\infty$. At the same time in the case of unequal deficiency indices $n_+(A)\neq n_-(A)$ each boundary relation $\G:\gH^2\to \cH^2$ for $A^*$ in the sense of [@DM06] satisfies the equality $\dim \cH=\infty$ (see [@DM09 Proposition 3.2]). This assertion shows that in the case $n_+(A)\neq n_-(A)$ our definition of a boundary relation is more natural and convenient for applications. Observe also that other generalizations of boundary triplets can be found e.g. in [@BehLan07].

Next by using the concept of a boundary relation we investigate in the paper linear relations induced by a general (not necessarily definite) canonical system of differential equations with possibly unequal deficiency indices. Such a system is of the form $$\begin{gathered}
\label{1.8}
J y'(t)-B(t)y(t)=\D (t)f(t), \quad t\in\cI,\end{gathered}$$ where $J$ is an operator in the finite-dimensional Hilbert space $\bH$ such that $J^*=J^{-1}=-J$ and $B(t)$ and $\D(t)$ are locally integrable $[\bH]$-valued functions defined on an interval $\cI=(a,b),\; -\infty\leq a< b
\leq\infty,$ and such that $B(t)=B^*(t)$ and $ \D(t)\geq 0$ a.e. on $\cI$. Without loss of generality we assume that $$\begin{gathered}
\label{1.9}
\bH=H\oplus\hat H\oplus H\end{gathered}$$ with the Hilbert spaces $H$ and $\hat H$ and the operator $J$ is $$\begin{gathered}
 \label{1.10}
J=\begin{pmatrix} 0 & 0&-I_H \cr 0& i I_{\hat H}&0\cr I_H&
0&0\end{pmatrix}:H\oplus\hat H\oplus H \to H\oplus\hat H\oplus H.\end{gathered}$$ The canonical system is called Hamiltonian if $\hat H=\{0\}$, in which case the operator $J$ takes the form $$\begin{gathered}
J=\begin{pmatrix} 0 & -I_H \cr  I_H& 0\end{pmatrix}:H\oplus H \to H\oplus H.\end{gathered}$$ Clearly, the Hamiltonian system is a particular case of the system .

Denote by $\lI$ the semi-Hilbert space of $\bH$-valued Borel functions $f(t)$ on $\cI$ with $\int_\cI (\D(t)f(t),f(t))\, dt < \infty $ and let $(f,g)_\D$ be the semi-definite inner product in $\lI$. Assume also that $\LI$ is the corresponding Hilbert space of equivalence classes and $\pi$ is the quotient map from $\lI$ onto $\LI$, so that the inner product in $\LI$ is $(\wt f, \wt g
)(=(\pi f,\pi g))=(f,g)_\D, \;\; \wt f,\wt g\in \LI$.

The null manifold $\cN$ of the system plays an essential role in our considerations. Recall [@KogRof75] that $\cN$ is defined as the set of all solutions of the equation $J y'(t)-B(t)y(t)=0$ such that $\D(t)y(t)=0$ a.e. on $\cI$. The system is said to be definite if $\cN=\{0\}$ and indefinite in the opposite case.

As is known the extension theory of symmetric relations is the natural framework for boundary value problems involving canonical systems of differential equations (see [@Orc; @LT82; @DLS88; @DLS93; @HSW00; @BHSW10; @LesMal03] and references therein). This framework is based on the concept of minimal and maximal relations which are defined as follows. Let $\tma$ be the set of all pairs $\{y,f\}\in\lI\times \lI $ satisfying the system and let $\cT_0$ be the set of all $\{y,f\}\in\tma$ such that $y$ has compact support. Then $\tma$ and $\cT_0$ are linear relations in $\lI$ and the Lagrange’s identity $$(f,z)_\D-(y,g)_\D=[y,z]_b -[y,z]_a,\quad \{y,f\}, \; \{z,g\} \in\tma.$$ holds with $$\label{1.11}
[y,z]_a:=\lim_{t \downarrow a}(J y(t),z(t)),\qquad [y,z]_b:=\lim_{t \uparrow
b}(J y(t),z(t)), \quad y,z \in\dom\tma.$$ By using introduce also the linear relation $\tmi$ in $\lI$ by $$\label{1.12}
\tmi=\{\{y,f\}\in\tma:\,[y,z]_a= [y,z]_b=0, \; z\in\dom\tma\}.$$ Moreover, in the case of the regular endpoint $a$ (that is, if $a\neq -\infty$ and $B(t)$ and $\D(t)$ are integrable on $(a,\b), \;\b\in\cI$) let $$\label{1.13}
\cT_a=\{\{y,f\}\in\tma:\, y(a)=0 \;\;\text{and}\;\; [y,z]_b=0, \;
z\in\dom\tma\}.$$ All the above relations in $\lI$ naturally generate by means of the equalities $$\begin{gathered}
\Tmi=\{\{\pi y, \pi f\}:\{y,f\}\in\tmi\}, \;\; \;\;\;T_a=\{\{\pi y, \pi
f\}:\{y,f\}\in\cT_a\},\label{1.14} \\
 T_0=\{\{\pi y, \pi f\}:\{y,f\}\in\cT_0\},
\;\;\;\;\;\Tma=\{\{\pi y, \pi f\}:\{y,f\}\in\tma\} \nonumber\end{gathered}$$ linear relations $\Tmi, \; T_a, \; T_0$ and $\Tma$ in the Hilbert space $\LI$.

As was shown in [@Orc] (see also [@LesMal03; @BHSW10]) in the case of the *definite system* $T_0$ is a symmetric linear relation in $\LI$, $\Tmi$ is closure of $T_0$ and $\Tma=\Tmi^*(=T_0^*)$; moreover, if the endpoint $a$ is regular then $\Tmi=T_a$. In view of this assertion $\Tmi$ and $\Tma$ are called minimal and maximal relations respectively, which is in full accord with similar definition of minimal and maximal operator for an ordinary differential expression [@Nai]. At once certain difficulties arise in the case of an *indefinite system* , which can be explained as follows. In the definite case the quotient mapping $\pi$ isomorphically maps $\dom\tma$ onto $\dom\Tma$, which enables one to identify in fact the relations $\tma$ and $\Tma$. If the system is indefinite, then the mapping $\pi\up\dom\tma$ has as nontrivial kernel the null manifold $\cN$, so that the immediate identifying of $\tma$ and $\Tma$ becomes impossible.

The above difficulties were partially overcome in the papers by Kac [@Kac83; @Kac84] (the case $\dim\bH=2$) and Lesch and Malamud [@LesMal03] (the case $\dim\bH=n<\infty$), where general (not necessarily definite) systems were studied. In these papers first the equality $T_0^*=\Tma$ is proved and then the minimal relation is defined as closure of $T_0$.

In the present paper we show that for the general system the minimal relation in $\LI$ can be also defined by the first equality in with $\tmi$ in the form . Moreover in the case of the regular endpoint $a$ the minimal relation coincides with the relation $T_a$ defined by and the second equality in . Observe also that $\cT_a\subset \tmi$ and an interesting in our opinion fact is that generally speaking $\cT_a\neq\tmi$ (for more details see Proposition \[pr4.11a\] and Example \[ex4.11b\]).

Next assume that $a$ is a regular endpoint for the canonical system , $$\begin{gathered}
\nu_+:=\dim\ker (i J-I)(=\dim H), \qquad \nu_-:=\dim\ker (i J+I)(=\dim (\hat
H\oplus H))\end{gathered}$$ and let $\nu_{b+} $ and $\nu_{b-} $ be indices of inertia of the skew-Hermitian form $[y,z]_b$ (for simplicity assume that $\nu_{b+}\geq \nu_{b-}$). The equality $\Tmi=T_a$ enables us to describe all proper extensions of $\Tmi$ in terms of boundary conditions. For this aim we use a special boundary relation for $\Tma$ which we call decomposing. This boundary relation is defined as follows.

Let $\cH_b$ and $\hat\cH_b$ be finite-dimensional Hilbert spaces and let $$\begin{gathered}
\label{1.15}
\G_b=(\G_{0b}:\,  \hat\G_b:\,  \G_{1b})^\top:\dom\tma\to
\cH_b\oplus\hat\cH_b\oplus \cH_b\end{gathered}$$ be a surjective linear map such that $$\begin{gathered}
\label{1.16}
[y,z]_b=i (\hat\G_b y, \hat\G_b z)-(\G_{1b}y,\G_{0b}z)+(\G_{0b}y,\G_{1b}z),
\quad y,z \in \dom\tma.\end{gathered}$$ (it is not difficult to prove the existence of such a map $\G_b$). Moreover, for each function $y\in\dom\tma$ let $$\begin{gathered}
\label{1.17}
y(t)=\{y_0(t),\, \hat y(t), \, y_1(t)  \}\end{gathered}$$ be the representation of $y(t)$ in accordance with the decomposition . Then the decomposing boundary relation $\G:(\LI)^2\to\HH$ for $\Tma$ is of the form $$\begin{gathered}
\label{1.18}
\G=\lb \lb {\pi y\choose \pi f}, {\G_0' y\choose \G_1'y } \rb
:\{y,f\}\in\tma\rb,\end{gathered}$$ where $\cH_0$ and $\cH_1$ are Hilbert spaces defined by means of $\bH, \cH_b$ and $\hat \cH_b$ and $\G_j': \dom\tma\to \cH_j, \; j\in\{0,1\}$ are linear maps constructed with the aid of $y(a)$ and the operators from . If $\Tmi$ has equal deficiency indices $n_+(\Tmi)=n_-(\Tmi)$, then $\cH_0=\cH_1=H\oplus\hat H\oplus \cH_b$ and the decomposing boundary relation can be written as $$\begin{gathered}
\label{1.19}
\G=\lb \lb {\pi y\choose \pi f}, {\{ y_0(a),\, \tfrac i {\sqrt 2}\ (\hat
y(a)-\hat\G_b y),\,\G_{0b}y \}\choose \{ y_1(a),\, \tfrac 1 {\sqrt 2} (\hat
y(a)+\hat\G_b y),\,-\G_{1b}y \} } \rb :\{y,f\}\in\tma\rb.\end{gathered}$$ In the case of the regular system one can put in $\G_{0b}y=y_0(b),
\; \G_{1b}y=y_1(b)$ and $\hat\G_b y=\hat y(b)$. If $b$ is not regular, then $\G_b y $ can be represented by means of certain limits at the point $b$ associated with the function $y\in\dom\tma$ (for more details see Remark \[rem5.2\] ). Therefore the decomposing boundary relation $\G$ is given by in terms of boundary values of the function $y\in\tma$ at the endpoint $a$ (regular value) and $b$ (singular value), which is of importance in our considerations of canonical systems.

Recall [@KogRof75] that the formal deficiency indices of the system are defined via $$\begin{gathered}
N_\pm=\dim \{y\in\lI:\, J y'(t)-B(t)y(t)=\l\D(t)y(t) \;\;\text{a.e. on}\;\;
\cI\}, \quad \l\in\bC_\pm.\end{gathered}$$ As was shown in [@LesMal03] the relations $$\begin{gathered}
N_+=n_+(\Tmi)+k_\cN, \qquad N_-=n_-(\Tmi)+k_\cN\end{gathered}$$ hold with $k_\cN=\dim\cN$. In the present paper by using just a fact of existence of a decomposing boundary relation we prove the equalities $$\begin{gathered}
\label{1.20}
N_+=\nu_+ + \nu_{b+}, \qquad N_-=\nu_- + \nu_{b-}.\end{gathered}$$ Formula yields the known estimates $\nu_\pm\leq N_\pm\leq \dim\bH$ obtained by analytic methods in[@Atk; @KogRof75]. Observe also that in a somewhat different way the equalities were proved for definite systems in [@BHSW10 Lemma 4.15].

Existence of the nontrivial multivalued part $\mul\G$ of the decomposing boundary relation is caused by the nontrivial null manifold $\cN$, which can be seen from the equalities $$\begin{gathered}
\label{1.21a}
\mul\G=\{\{\G_0'y,\G_1'y\}:\, y\in\cN\}, \qquad \dim (\mul\G)=k_\cN(=\dim\cN).\end{gathered}$$ Formula implies that for the definite canonical system the decomposing boundary relation turns into the decomposing boundary triplet $\Pi=\bta$ for $\Tma$. In the case $n_+(\Tmi)=n_-(\Tmi)$ this triplet is of the form $\Pi=\bt$ with the boundary Hilbert space $\cH=H\oplus\hat H \oplus \cH_b$ and the operators $\G_j$ given by $$\begin{gathered}
\G_0 \{\wt y, \wt f\} =\{ y_0(a),\, \tfrac i {\sqrt 2} (\hat y(a)-\hat\G_b
y),\,\G_{0b}y \} (\in H\oplus\hat H \oplus \cH_b),\qquad\qquad\qquad\quad
\label{1.22}\\
\G_1 \{\wt y, \wt f\} = \{ y_1(a),\, \tfrac 1 {\sqrt 2} (\hat y(a)+\hat\G_b
y),\,-\G_{1b}y \}(\in H\oplus\hat H \oplus \cH_b), \quad \{\wt y, \wt f\} \in
\Tma \label{1.23}\end{gathered}$$ (here $\G_{0b}, \;\G_{1b}$ and $\hat\G_b$ are taken from ). In the case of the regular system one can put $\cH= H\oplus\hat H \oplus H$ and $$\begin{gathered}
\G_0 \{\wt y, \wt f\}=\{ y_0(a),\, \tfrac i {\sqrt 2} (\hat y(a)-\hat
y(b)),\, y_0(b) \}(\in H\oplus\hat H \oplus H),\qquad\qquad\qquad\quad\label{1.24} \\
\G_1 \{\wt y, \wt f\}= \{ y_1(a),\, \tfrac 1 {\sqrt 2}(\hat y(a)+\hat
y(b)),\,-y_1(b) \}(\in H\oplus\hat H \oplus H),\quad \{\wt y, \wt f\} \in \Tma.
\label{1.25}\end{gathered}$$ The boundary triplet $\bt$ defined via and is similar to that introduced, in fact, by Rofe-Beketov [@Rof69] for regular differential operators of a higher order. Observe also that other constructions of a boundary triplet for $\Tma$ in the case of the definite system can be found in [@BHSW10].

The decomposing boundary triplet , enables us to describe maximal dissipative, maximal accumulative and self-adjoint boundary conditions, which define in terms of boundary values the extensions $\wt A\supset \Tmi$ of the corresponding class. As a consequence we obtain the known description of self-adjoint boundary conditions, given in [@Atk; @GK; @Orc] for regular definite systems and in [@Kra89] for definite Hamiltonian systems with the regular endpoint $a$.

Finally by using the concept of a decomposing boundary triplet we examine separated boundary conditions of various classes. Recall that self-adjoint separated boundary conditions for definite Hamiltonian systems were studied with the aid of analytic methods by many authors (see [@HinSch93; @Kra89] and references therein). In the present paper we show that self-adjoint separated boundary conditions for the definite canonical system exist if and only if this system is Hamiltonian. Moreover, for the Hamiltonian system the decomposing boundary triplet $\bt$ for $\Tma$ takes the form $$\begin{gathered}
\G_0 \{\wt y, \wt f\} =\{ y_0(a),\, \G_{0b}y \} (\in H\oplus \cH_b),\qquad \G_1
\{\wt y, \wt f\} = \{ y_1(a),\, -\G_{1b}y \}(\in H \oplus \cH_b), \quad \{\wt
y, \wt f\} \in \Tma\end{gathered}$$ and the general form of self-adjoint separated boundary conditions is $$\begin{gathered}
\label{1.26}
\wt A=\{\{\wt y, \wt f\}:\, N_{0a}y_0(a)+N_{1a}y_1(a)=0, \;\;
N_{0b}\G_{0b}y+N_{1b}\G_{1b}y=0 \},\end{gathered}$$ where the operators $N_{0a},\; N_{1a}$ and $N_{0b},\; N_{1b}$ are entries of the self-adjoint operator pairs $\{(N_{0a}, N_{1a})\}$ and $\{(N_{0b},
N_{1b})\}$. These results are generalizations of those obtained by Rofe-Beketov in [@Rof69] for regular differential operators. Moreover, formula includes as a particular case the results on self-adjoint separated boundary conditions in [@HinSch93; @Kra89].

An interesting in our opinion fact is the existence of maximal dissipative an accumulative separated boundary conditions for the not necessarily Hamiltonian system (in the paper we describe all these conditions). An important subclass of maximal dissipative (accumulative) separated conditions are those defined by a self-adjoint condition at the regular endpoint $a$ and the maximal dissipative (accumulative) boundary condition at the singular endpoint $b$. This subclass of boundary conditions may be useful in the theory of not orthogonal spectral functions associated with the system (we are going to touch upon this subject elsewhere).

Preliminaries
=============

Linear relations
----------------

The following notations will be used throughout the paper: $\gH$, $\cH$ denote Hilbert spaces; $[\cH_1,\cH_2]$ is the set of all bounded linear operators defined on $\cH_1$ with values in $\cH_2$; $[\cH]:=[\cH,\cH]$; $A\up \cL$ is the restriction of an operator $A$ onto the linear manifold $\cL$; $P_\cL$ is the orthogonal projector in $\gH$ onto the subspace $\cL\subset\gH$; $\bC_+\,(\bC_-)$ is the upper (lower) half-plain of the complex plain.

Recall that a linear relation $T$ from a linear space $L_0 $ to a linear space $L_1$ is a linear manifold in the Cartesian product $L_0\times L_1$. It is convenient to write $T: L_0\to L_1$ and interpret $T$ as a multi-valued linear mapping from $L_0$ into $L_1$. If $L_0=L_1=:L$ one speaks of a linear relation $T$ in $L$. For a linear relation $T:L_0\to L_1$ we denote by $\dom T,\,\ran T,
\,\ker T$ and $\mul T$ the domain, range, kernel and the multivalued part of $T$ respectively. The inverse $T^{-1}$ is a linear relation from $L_1$ to $L_0$ defined by $T^{-1}=\{\{f', f\}:\{f,f'\}\in T\}$.

Assume now that $\cH_0$ and $\cH_1$ are Hilbert spaces. Then the linear space $\cH_0\times\cH_1$ with the inner product $(\{f,f'\}, \{g,g'\})_{\cH_0
\oplus\cH_1} =(f,g)_{\cH_0}+(f',g')_{\cH_1}$ is a Hilbert space $\cH_0\oplus\cH_1$. The set of all closed linear relations from $\cH_0$ to $\cH_1$ (in $\cH$) will be denoted by $\C (\cH_0,\cH_1)$ ($\C(\cH)$). A closed linear operator $T$ from $\cH_0$ to $\cH_1$ is identified with its graph $\text {gr}\, T\in\CA$. For a linear relation $T:\cH_0\to \cH_1$ we denote by $T^*(\in\CB)$ the adjoint relation

In the case $T\in\CA$ we write $0\in \rho (T)$ if $\ker T=\{0\}$ and $\ran
T=\cH_1$, or equivalently if $T^{-1}\in [\cH_1,\cH_0]$; $0\in \hat\rho (T)$ if   $\ker T=\{0\}$ and  $\ran (T)$ is a closed subspace in $\cH_1$. For a linear relation $T\in \C(\cH)$ we denote by $\rho (T):=\{\l \in \bC:\ 0\in \rho
(T-\l)\}$ and $\hat\rho (T)=\{\l \in \bC:\ 0\in \hat\rho (T-\l)\}$ the resolvent set and the set of regular type points of $T$ respectively.

For a linear relation $T\in \C(\cH)$ and for any $\l \in \bC$ we let $$\gN_\l (T):=\ker (T^*-\l)\,(=\cH\ominus \ran (T-\ov\l)), \quad \hat
\gN_{\l}(T):=\{\{f,\l f\}:f\in \gN_{\l }(T)\}\subset T^*.$$ If $\ov\l\in\hat\rho (T)$, then $\gN_\l (T)$ is a defect subspace of $T$. Recall also the following definition.

\[def2.0\] A holomorphic operator function $\Phi (\cd):\bC\setminus\bR\to [\cH]$ is called a Nevanlinna function if $\im\, z\cd \im \Phi (z)\geq 0 $ and $\Phi ^*(z)=
\Phi (\ov z), \; z\in\bC\setminus\bR$.

Operator pairs 
---------------

Let $\cK,\cH_0,\cH_1$ be Hilbert spaces. A pair of operators $C_j\in
[\cH_j,\cK],  \;j\in\{0,1\}$ is called admissible if the range of the operator $$\label{2.1}
C=(C_0:\,C_1):\cH_0\oplus\cH_1\to\cK$$ coincides with $\cK$. In the sequel all pairs are admissible unless otherwise stated.

Two pairs $(C_0^{(j)}:\,C_1^{(j)}):
\cH_0\oplus\cH_1\to\cK_j, \; j\in\{1,2\}$ will be called equivalent if $C_0^{(2)}=XC_0^{(1)}$ and $C_1^{(2)}=XC_1^{(1)}$ with some isomorphism $X\in
[\cK_1,\cK_2]$.

It is clear that the set of all operator pairs falls into nonintersecting classes of equivalent pairs. Moreover, the equality $$\label{2.2}
\t=\{(C_0,C_1);\cK\}:=\{\{h_0, h_1\}\in\cH_0\oplus\cH_1:\, C_0h_0+C_1h_1=0\}$$ establishes a bijective correspondence between all linear relations $\t\in\CA$ and all equivalence classes of operator pairs . Therefore in the sequel we identify (by means of ) a linear relation $\t\in\CA$ and the corresponding class of equivalent operator pairs $C_j\in [\cH_j,\cK], \;
j\in \{0,1\}$.

Next recall some results and definitions from our paper [@Mog06.1].

Assume that $\cH_0$ is a Hilbert space, $\cH_1$ is a subspace in $\cH_0$, $\cH_2:=\cH_0\ominus\cH_1$ and $P_j$ is the orthoprojector in $\cH_0$ onto $\cH_j,\; j\in\{1,2\}$. For an operator pair (linear relation) $\t=\{(C_0,C_1);\cK\} (\in\CA)$ we let $$\begin{gathered}
\wt S_\t:=2 \im (C_1C_{01}^*)-C_{02}C_{02}^*, \quad \wt S_\t\in [\cK],\end{gathered}$$ where $C_{01}$ and $C_{02}$ are entries of the block representation $C_0=(C_{02}:\, C_{01}):\cH_2\oplus\cH_1\to\cK$.

\[def2.1\]$\,$ [@Mog06.1] An operator pair (linear relation) $\t=\{(C_0,C_1);\cK\}(\in\CA)$ belongs to the class:

1\) $Dis (\cH_0,\cH_1)$, if $\wt S_\t\geq 0$ and $0\in\rho (C_{01}-iC_1) $;

2\) $Ac (\cH_0,\cH_1))$, if $\wt S_\t \leq 0$ and $0\in\rho (C_0+iC_1P_1) $;

3\) $Sym (\cH_0,\cH_1))$, if $\wt S_\t = 0$ and $0\in\rho (C_{01}-iC_1) \cup
(C_0+iC_1P_1) $;

4))$Self (\cH_0,\cH_1)$, if $\t\in Dis (\cH_0,\cH_1)\cap Ac (\cH_0,\cH_1))$.

Note that the inclusion $0\in\rho (C_{01}-iC_1)\cup \rho (C_0+iC_1P_1)$ implies that $\ran (C_0:\, C_1)=\cK$. Therefore each of the above definitions 1) – 4) gives an admissible operator pair $(C_0:\, C_1)$.

If $\t\in Dis (\cH_0,\cH_1)$ ($\t\in Ac (\cH_0,\cH_1)$), then $\dim\cK=\dim\cH_1$ (resp. $\dim\cK=\dim\cH_0$). Therefore for an operator pair $\t\in Dis (\cH_0,\cH_1)\;(\t\in Ac (\cH_0,\cH_1))$ one can put $\cK=\cH_1$ (resp. $\cK=\cH_0$).

In the case $\cH_0=\cH_1:=\cH$ we let $Dis (\cH)=Dis (\cH,\cH)$ and similarly the classes $ Ac (\cH),\; Sym (\cH)$ and $ Self (\cH)$ are defined. In view of Definition \[def2.1\] for each operator pair (linear relation) $\t=\{(C_0,C_1);\cK\}(\in\C (\cH))$ the following equivalences hold: $$\begin{gathered}
\t\in Dis (\cH) \iff (\im (C_1C_0^*)\geq 0 \;\;\text{and}\;\; 0\in\rho
(C_0-iC_1) )\label{2.2.1}\\
\t\in Ac (\cH) \iff (\im (C_1C_0^*)\leq 0 \;\;\text{and}\;\; 0\in\rho
(C_0+iC_1) )\label{2.2.2}\\
\t\in Sym (\cH) \iff (\im (C_1C_0^*)= 0 \;\;\text{and}\;\; 0\in\rho
(C_0-iC_1)\cup\rho (C_0+iC_1) )\label{2.2.3}\\
\t\in Self (\cH) \iff (\im (C_1C_0^*)= 0 \;\;\text{and}\;\; 0\in\rho
(C_0-iC_1)\cap\rho (C_0+iC_1) )\label{2.2.4}\end{gathered}$$ Moreover, the classes $Dis (\cH), \;Ac (\cH),\; Sym (\cH)$ and $ Self (\cH)$ coincide with the known classes of all maximal dissipative, maximal accumulative, maximal symmetric and self-adjoint linear relations in $\cH$ respectively.

The following proposition is immediate from Definition \[def2.1\].

\[pr2.2\] 1) In the case $\dim \cH_0<\infty$ the class $Self (\cH_0,\cH_1)$ is not empty if and only if $\cH_0=\cH_1=:\cH$.

2\) Let $\dim\cH<\infty $ and let $\t=\{(C_0,C_1);\cK\}(\in \C (\cH))$ be an admissible operator pair such that $\dim\cK=\dim\cH$. Then the following equivalences hold $$\begin{gathered}
\t\in Dis (\cH) \Leftrightarrow (\im (C_1C_0^*)\geq 0, \quad \t\in Ac (\cH)
\Leftrightarrow (\im (C_1C_0^*)\leq 0, \quad \t\in Self (\cH) \Leftrightarrow
(\im (C_1C_0^*)= 0.\end{gathered}$$

Boundary triplets and Weyl functions
------------------------------------

Let $A\in\C (\gH)$ be a closed symmetric linear relation in the Hilbert space $\gH$ and let $n_\pm (A):=\dim \gN_\l(A), \; \l\in\bC_\pm$ be deficiency indices of $A$. Denote by $\exa$ the set of all proper extensions of $A$, i.e., the set of all relations $\wt A\in \C (\gH)$ such that $A\subset\wt A\subset
A^*$.

Next assume that $\cH_0$ is a Hilbert space, $\cH_1$ is a subspace in $\cH_0$ and $\cH_2:=\cH_0\ominus\cH_1$, so that $\cH_0=\cH_1\oplus\cH_2$. Denote by $P_j$ the orthoprojector in $\cH_0$ onto $\cH_j,\; j\in\{1,2\} $.

\[def2.3\]$\,$[@Mog06.2] A collection $\Pi=\bta$, where $\G_j: A^*\to \cH_j, \; j\in\{0,1\}$ are linear mappings, is called a boundary triplet for $A^*$, if the mapping $\G :\hat f\to
\{\G_0 \hat f, \G_1 \hat f\}$ from $A^*$ into $\cH_0\oplus\cH_1$ is surjective and the following Green’s identity $$\label{2.3}
(f',g)-(f,g')=(\G_1  \hat f,\G_0 \hat g)_{\cH_0}- (\G_0 \hat f,\G_1 \hat
g)_{\cH_0}+i(P_2\G_0 \hat f,P_2\G_0 \hat g)_{\cH_2}$$ holds for all $\hat
f=\{f,f'\}, \; \hat g=\{g,g'\}\in A^*$.

In the following propositions some properties of boundary triplets are specified (see [@Mog06.2]).

\[pr2.4\] If $\Pi=\bta$ is a boundary triplet for $A^*$, then $$\label{2.4}
\dim \cH_1=n_-(A)\leq n_+(A)=\dim \cH_0.$$ Conversely for any symmetric linear relation $A\in \C (\gH)$ with $n_-(A)\leq
n_+(A)$ there exists a boundary triplet for $A^*$.

\[pr2.5\] Let $\Pi=\bta$ be a boundary triplet for $A^*$. Then:

1\) $\ker \G_0\cap\ker\G_1=A$ and $\G_j$ is a bounded operator from $A^*$ into $\cH_j, \;  j\in\{0,1\}$;

2\) the set of all proper extensions $\wt A\in\exa$ is parameterized by linear relations (operator pairs) $\t=\{(C_0,C_1);\cK\}$. More precisely, the mapping $$\label{2.5}
\t\to  A_\t :=\{ \hat f\in A^*:\{\G_0  \hat f,\G_1 \hat f \}\in \t\}$$ establishes a bijective correspondence between the linear relations $\t\in\CA$ and the extensions $ \wt A= A_\t\in\exa$. If $\t$ is given as an operator pair $\t=\{(C_0,C_1);\cK \}$, then $A_\t$ can be represented in the form of an abstract boundary condition: $$\label{2.5a}
A_\t=\{\hat f\in A^*: C_0\G_0 \hat f+C_1\G_1\hat  f=0 \}$$

3\) the extension $ A_\t$ is maximal dissipative, maximal accumulative, maximal symmetric or self-adjoint if and only if $\t$ belongs to the class $Dis,\; Ac,\; Sym$ or $Self (\cH_0,\cH_1)$ respectively;

4\) The equality $$\label{2.6}
A_0:=\ker \G_0=\{\hat f\in A^*:\G_0 \hat f=0\}$$ defines the maximal symmetric extension $A_0\in\exa$ such that $n_-(A_0)=0$.

It turns out that for every $\l\in\bC_+\;(z\in\bC_-)$ the map $\G_0\up \hat
\gN_\l (A)$ (resp $P_1\G_0\up \hat \gN_z (A)$) is an isomorphism. This makes it possible to introduce the $\g$-fields $\g_{\Pi +}(\cdot):\Bbb
C_+\to[\cH_0,\gH], \; \; \g_{\Pi -}(\cdot):\Bbb C_-\to[\cH_1,\gH]$ and the Weyl functions $M_{\Pi +}(\cdot):\bC_+\to [\cH_0,\cH_1], \;\; M_{\Pi
-}(\cdot):\bC_-\to [\cH_1,\cH_0]$ by $$\begin{gathered}
\g_{\Pi +} (\l)=\pi_1(\G_0\up\hat \gN_\l (A))^{-1}, \;\;\l\in\Bbb C_+;\quad
\g_{\Pi -} (z)=\pi_1(P_1\G_0\up\hat\gN_z (A))^{-1}, \;\; z\in\Bbb C_-,
\label{2.7}\\
\G_1 \up \hat\gN_\l (A)=M_{\Pi +}(\l)\G_0 \up \hat\gN_\l (A),\quad \l\in\bC_+,
\label{2.8}\\
(\G_1+iP_2\G_0)\up \hat\gN_z (A)=M_{\Pi -}(z)P_1\G_0 \up \hat\gN_z (A),\quad
z\in \bC_-.\label{2.9}\end{gathered}$$ (here $\pi_1$ is the orthoprojection in $\gH\oplus\gH$ onto $\gH\oplus \{0\}$). According to [@Mog06.2] all functions $\g_{\Pi\pm}$ and $M_{\Pi\pm} $ are holomorphic on their domains and $(M_{\Pi +}(\l))^*=M_{\Pi-}(\ov \l),
\;\l\in\bC_+$.

\[rem2.6\] In the case $\cH_0=\cH_1:=\cH$ Definition \[def2.3\] coincides with that of the boundary triplet (boundary value space) $\Pi=\{\cH,\G_0,\G_1\}$ for $A^*$ given in [@GorGor]. For such a triplet $n_+(A)=n_-(A)=\dim \cH$, $\,A_0(=\ker \G_0 )$ is a self-adjoint extension of $A$ and the relations $$\label{2.10}
\g_\Pi(\l)=\pi_1(\G_0\up\hat\gN_\l(A))^{-1}, \qquad \G_1\up\hat
\gN_\l(A)=M_\Pi(\l)\G_0 \up\hat\gN_\l(A), \qquad \l\in\rho (A_0)$$ define the $\g$-field $\g_\Pi(\cd):\rho (A_0)\to [\cH,\gH]$ and the Weyl function $M_\Pi(\cd):\rho (A_0)\to [\cH]$ [@DM91] associated with operator functions – via $\g_\Pi(\l)=\g_{\Pi\pm}
(\l)$ and $M_\Pi(\l)=M_{\Pi\pm}(\l), \;\l\in\bC_\pm $.

Boundary relations and their Weyl families
==========================================

Let $\gH $ and $\cH_0$ be Hilbert spaces, let $\cH_1$ be a subspace in $\cH_0$, let $\cH_0=\cH_1\oplus\cH_2$ be the corresponding orthogonal decomposition of $\cH_0$ with $\cH_2:=\cH_0\ominus\cH_1$ and let $P_j$ be the orthoprojector in $\cH_0$ onto $\cH_j, \; j\in \{0,1\}$. In the sequel we deal with linear relations from $\gH^2$ into $\cH_0\oplus\cH_1$. If $\G$ is such a relation, then an element $\hat\f\in\G$ will be denoted by $\hat\f =\{\hat f,\hat h\}$, where $\hat f=\{f,f'\}\in\gH^2\; (f,\,f'\in \gH)$ and $\hat h=\{h_0,h_1\}\in
\cH_0\oplus\cH_1\; (h_0\in\cH_0,\; h_1\in\cH_1)$. In this case it will be convenient to write $$\label{3.1a}
\hat\f=\{\hat f,\hat h\}=\lb\hat f, {h_0 \choose h_1} \rb=\lb
\begin{pmatrix} f\cr f' \end{pmatrix},
\begin{pmatrix} h_0\cr h_1 \end{pmatrix}\rb.$$ If in addition $\cH_j$ is decomposed as $\cH_j=\cH_{j1}\oplus\cH_{j2}\oplus
\dots  \cH_{j,n_j}, \; j\in\{0,1\}$, then the equality will be also written as $$\hat\f=\lb\hat f, {\{h_{01}, h_{02}, \dots, h_{0,n_0}\}\choose \{h_{11},
h_{12}, \dots, h_{1,n_1}\}} \rb,$$ where $h_{0k}=P_{\cH_{0k}}h_0$ and $h_{1k}=P_{\cH_{1k}}h_1$ are components of $h_0$ and $h_1$ respectively.

For a linear relation $\BR$ its multivalued part is a linear relation from $\cH_0$ into $\cH_1$ given by $$\mul\G =\lb \{h_0,h_1\}\in\cH_0\oplus\cH_1: \lb 0,{h_0\choose h_1}\rb \in\G \rb.$$ Using $\mul\G$ we define linear manifolds $\cK_\G'$ and $\cK_G''$ in $\cH_1$ via $$\begin{gathered}
\cK_\G'=P_1\,\dom (\mul\G)=\lb k'\in\cH_1: \lb 0,{k'+h_2\choose h_1} \rb \in\G
\;\;\text{for some}\;\; h_2\in\cH_2 \;\; \text{and}\; \; h_1\in\cH_1 \rb,
\label{3.3}\\
\cK_\G''=\mul (\mul\G)=\lb k''\in\cH_1:\lb 0,{0\choose
k''}\rb\in\G\rb.\label{3.4}\end{gathered}$$ Next introduce the signature operators $$\label{3.5}
J_\gH=\begin{pmatrix} 0 & -i I_\gH \cr iI_\gH & 0\end{pmatrix}:\gH\oplus\gH\to
\gH\oplus\gH, \;\;\;\; J_{01}=\begin{pmatrix} P_2 & -i I_{\cH_1} \cr iP_1 &
0\end{pmatrix}:\cH_0\oplus\cH_1\to \cH_0\oplus\cH_1$$ and denote by $(\gH^2,J_\gH)$ and $(\HH,J_{01})$ the corresponding Krein spaces. Recall [@Shm76] that a linear relation $\BR$ is called an isometric relation from $(\gH^2,J_\gH)$ into $(\HH,J_{01})$ if $$\label{3.6}
(J_\gH \hat f,\hat g)_{\gH^2}=(J_{01} \hat h, \hat x)_{\cH_0\oplus \cH_1},
\quad \{\hat f, \hat h\}, \;\{\hat g, \hat x\}\in\G$$ or, equivalently, if the identity $$\label{3.7}
(f',g)_\gH -  (f,g')_\gH= (h_1,x_0)_{\cH_0} - (h_0,x_1)_{\cH_0}+i (P_2h_0,P_2
x_0)_{\cH_2}$$ holds for every $\{{f \choose f'}, {h_0 \choose h_1}\}, \;\{{g \choose g'},
{x_0 \choose x_1}\} \in \G $.

\[def3.2\] Let $A$ be a closed symmetric linear relation in $\gH$, let $\cH_0$ be a Hilbert space and let $\cH_1$ be a subspace in $\cH_0$. A linear relation $\BR$ is called a boundary relation for $A^*$ if:

1\) $\dom \G$ is dense in $A^*$ and $\G$ is an isometric relation from $(\gH^2,J_\gH)$ into $(\HH,J_{01})$, i.e., the abstract Green’s identity holds;

2\) if $\hat\f=\{{g\choose g'}, {x_0\choose x_1}\}\in \gH^2\oplus
(\cH_0\oplus\cH_1)$ satisfies for every $\{{f\choose f'},
{h_0\choose h_1}\}\in\G$, then $\hat\f\in\G$.

The conditions 1) and 2) of Definition \[def3.2\] imply that the boundary relation $\G$ is a unitary relation from $(\gH^2,J_\gH)$ to $(\HH, J_{01})$ [@Shm76]. Therefore $\G$ is closed and $\ker\G=A$.

\[def3.2a\] The families of linear relations $M_+(\l):\cH_0\to\cH_1, \; \l\in\bC_+$ and $M_-(z):\cH_1\to\cH_0, \; z\in\bC_-$ given by $$\begin{gathered}
M_+(\l)=\lb \{h_0, h_1\}\in \HH: \lb {f\choose \l f}, {h_0\choose h_1}\rb
\in\G \;\;\text{for some} \;\; f\in\gH \rb, \;\;\l\in\bC_+; \label{3.7.1}\\
M_-(z)=\lb \{P_1h_0, h_1+iP_2 h_0\}: \lb {f\choose z f}, {h_0\choose h_1}\rb
\in\G \;\;\text{for some} \;\; f\in\gH \rb, \;\; z\in\bC_- \label{3.7.2}\end{gathered}$$ are called the Weyl families corresponding to the boundary relation $\BR$ for $A^*$.

If $M_+(\cd)$ (resp. $M_-(\cd)$) is operator-valued, it is called the Weyl function corresponding to the boundary relation $\G$.

In the sequel we deal with boundary relations of the special form introduced in the following proposition.

\[pr3.3\] Assume that $\Pi=\{\cK_0\oplus\cK_1,G_0,G_1\}$ is a boundary triplet for $A^*$ (see Definition \[def2.3\]), $\cK_2:=\cK_0\ominus\cK_1$, $\cK'$ and $\cK''$ are Hilbert spaces and $$\label{3.7a}
\cH_1:=\cK_1\oplus \cK'\oplus\cK'', \qquad \cH_0:=\cK_0\oplus \cK'\oplus\cK''
(=\cK_2\oplus\cH_1).$$ Moreover, let $F_0\in [\cK',\cK_0], \; F'\in [\cK']$, let $$\label{3.8}
F_0=(F_2\;\; F_1)^\top : \cK'\to \cK_2\oplus \cK_1$$ be the block representation of $F_0$ and let $$\label{3.9}
F'-(F')^*+i F_2^*F_2=0.$$ Then the equality $$\label{3.10}
\G=\lb \lb \hat f, {\{G_0\hat f-iF_2k',\, k',\, 0\}\}\choose \{G_1\hat f +F_1
k',\, F_0^* G_0\hat f +F'k',\, k''\}} \rb :\hat f\in A^*,\; k'\in\cK', \;
k''\in\cK'' \rb$$ defines the boundary relation $\BR$ for $A^*$ such that $\cK_\G'=\cK'$ and $\cK_\G''=\cK''$.

It is easily seen that the following assertion (a) is valid:

\(a) an element $\hat\f=\{\hat g, {x_0\choose x_1}\}\in \gH^2\oplus
(\cH_0\oplus\cH_1)$ with $$\label{3.11}
\hat g=\{g,g'\}\in\gH^2, \quad x_0=\{m_0,\,x_0', \, x_0''\}\in\cK_0\oplus
\cK'\oplus\cK'', \quad x_1=\{m_1,\, x_1',\, x_1''\}\in\cK_1\oplus
\cK'\oplus\cK''$$ satisfies the identity for every $\{{f\choose f'}, {h_0\choose
h_1}\}\in\G$ if and only if $x_0''=0$ and the following equalities hold $$\begin{gathered}
(f',g)-(f,g')=\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad
\qquad\qquad\qquad\qquad\qquad
\qquad\qquad\label{3.12} \\
=(G_1 \hat f, m_0)_{\cK_0}+(F_0^* G_0\hat f, x_0')_{\cK'}-(G_0 \hat f,
m_1)_{\cK_0}+i(P_{\cK_2} G_0\hat f,P_{\cK_2} m_0)_{\cK_2}, \;\;\; \hat f =
\{f,f'\} \in A^*; \nonumber\\
(F_0k', m_0)_{\cK_0}+(F'k',x_0')_{\cK'}-(k',x_1')_{\cK'}=0, \quad
k'\in\cK'.\label{3.13}\end{gathered}$$ Let $\{\hat g, {x_0\choose x_1}\}\in\G$, so that $\hat g\in A^*$ and $x_0, x_1$ are given by with $$m_0=G_0 \hat g-i F_2 x_0', \quad x_0''=0, \quad m_1=G_1\hat g +F_1x_0', \quad
x_1'=F_0^* G_0\hat g+F' x_0'.$$ Then substitution of such $m_0, \; m_1$ and $x_1'$ into , and the immediate calculation with taking and into account show that the equalities and are satisfied. Therefore by assertion (a) the identity holds for every $\{{f \choose f'}, {h_0 \choose h_1}\}, \;\{{g \choose g'}, {x_0 \choose
x_1}\} \in \G $.

Assume now that an element $\{\hat g, {x_0\choose x_1}\}\in \gH^2\oplus
(\cH_0\oplus\cH_1)$ given by satisfies the identity for every $\{{f\choose f'}, {h_0\choose h_1}\}\in\G$. Then by assertion (a) $x_0''=0$ and the equalities , are fulfilled.

If $\hat f=\{f,f'\}\in A$, then $G_0\hat f=G_1\hat f=0$ and by $(f',g)-(f,g')=0$. This implies that $\hat g\in A^*$. Next, in view of $F_0^* G_0\hat f =F_2^* P_{\cK_2}G_0\hat f+F_1^* P_{\cK_1}G_0\hat f
$ and the equality can be written as $$\begin{gathered}
(f',g)-(f,g')=\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad
\qquad\qquad\qquad\qquad\qquad
\qquad\qquad\label{3.14}\\
=(G_1 \hat f, P_{\cK_1}m_0)-(P_{\cK_1} G_0\hat f, m_1-F_1 x_0')+i(P_{\cK_2}
G_0\hat f,P_{\cK_2} m_0+iF_2x_0'), \;\;\;\; \hat f = \{f,f'\} \in A^*.\nonumber\end{gathered}$$ Since the map $G=(G_0\;\;G_1)^\top$ is surjective, it follows from and that $$P_{\cK_1}m_0=P_{\cK_1}G_0\hat g, \quad m_1-F_1 x_0'=G_1\hat g,\quad P_{\cK_2}
m_0+iF_2x_0'=P_{\cK_2}G_0\hat g$$ and, consequently, $$m_0=G_0\hat g-iF_2x_0', \quad m_1=G_1\hat g+F_1 x_0'.$$ Moreover, by using first and then , one obtains $$x_1'=F_0^* m_0+(F')^* x_0'=F_0^* G_0\hat g+((F')^*-i F_0^* F_2)x_0'=F_0^*
G_0\hat g+ F'x_0'.$$ Thus $\{\hat g, {x_0\choose x_1}\}\in \G$ and the linear relation satisfies both conditions of Definition \[def3.2\].

Finally the equalities $\cK_\G'=\cK'$ and $\cK_\G''=\cK''$ are immediate from .

\[pr3.3a\] Let under the assumptions of Proposition \[pr3.3\] $\cK''=\{0\}$, so that $$\label{3.14.1}
\cH_1:=\cK_1\oplus \cK', \qquad \cH_0:=\cK_0\oplus \cK' (=\cK_2\oplus\cH_1).$$ and the boundary relation $\BR$ for $A^*$ is (c.f. )

$$\label{3.14.2}
\G=\lb \lb \hat f, {\{G_0\hat f-iF_2k',\, k'\}\}\choose \{G_1\hat f +F_1 k',\,
F_0^* G_0\hat f +F'k'\}} \rb :\hat f\in A^*,\; k'\in\cK' \rb.$$

Assume also that $\g_{\Pi\pm}(\cd)$ and $M_{\Pi\pm}(\cd)$ are the $\g$-fields and Weyl functions corresponding to the boundary triplet $\Pi$ (see -) and $\G_0:\gH^2\to \cH_0$ is the linear relation given by $$\G_0=P_{\cH_0\oplus \{0\}}\G=\lb\{\hat f, h_0\}\in \gH^2\oplus\cH_0: \lb \hat
f, {h_0\choose h_1} \rb \in \G \;\; \text{for some}\;\; h_1\in\cH_1 \rb.$$ Then: 1) $\ker\G_0=\ker G_0$, so that the equality $$\label{3.14.3}
A_0:=\ker \G_0=\{\hat f\in A^*: \{\hat f, 0\}\in\G_0\}$$ gives the maximal symmetric extension $A_0\in\exa$ with $n_-(A_0)=0 \,(\iff
\bC_+\subset \rho (A_0))$;

2\) the equalities $$\begin{gathered}
\hat\g_{+} (\l)=(\G_0\up\hat \gN_\l (A))^{-1},\qquad \g_{+}
(\l)=\pi_1\hat\g_{+}
(\l),\;\; \;\;\l\in\Bbb C_+;\label{3.14.4}\\
 \hat\g_{-} (z)=(P_1\G_0\up\hat\gN_z
(A))^{-1}, \qquad  \g_-(z)=\pi_1 \hat\g_{-} (z),\;\;\;\; z\in\Bbb
C_-,\label{3.14.5}\end{gathered}$$ define the holomorphic operator functions ($\g$-fields) $\g_{+}(\cdot):\Bbb
C_+\to[\cH_0,\gH]$ and $\g_{-}(\cdot):\Bbb C_-\to[\cH_1,\gH]$. Moreover, $$\label{3.14.6}
\g_+(\l)=\g_{\Pi +}(\l)\,(I_{\cK_0}\;\;iF_2), \;\;\l\in\bC_+; \qquad
\g_-(z)=\g_{\Pi -}(z)P_{\cK_1}, \;\;z\in\bC_-$$ and the following identities hold $$\begin{gathered}
\g_+ (\mu)=\g_+ (\l )+ (\mu-\l )(A_0-\mu)^{-1} \g _{+} (\l), \quad \l, \mu\in
\bC_+  \label{3.14.7}\\
 \gamma _{-} (\omega)=\gamma _{-} (z)+
(\omega- z)(A_0^*-\omega)^{-1} \gamma _{-} (z),  \quad z, \omega\in
\bC_-  \label{3.14.8}\\
\gamma _{-} (z)P_1=\gamma _+ (\l)+ (z-\l)(A_0^*-z)^{-1} \gamma _{+} (\l), \quad
\l\in\bC_+,\; z\in \bC_-. \label{3.14.9}\end{gathered}$$

3\) the corresponding Weyl families are the holomorphic operator functions $M_{+}(\cdot):\bC_+\to [\cH_0,\cH_1]$ and $ M_{-}(\cdot):\bC_-\to
[\cH_1,\cH_0]$ associated with $M_{\Pi\pm}(\cd)$ by $$\label{3.14.10}
M_+(\l)=\begin{pmatrix} M_{\Pi +}(\l) & F_1+i\,M_{\Pi +}(\l)F_2 \cr F_0^* &
(F')^*\end{pmatrix}: \cK_0\oplus\cK'\to \cK_1\oplus\cK', \;\;\;\l\in\bC_+$$ and $M_-(z)=M_+^*(\ov z), \; z\in\bC_-$. Moreover, the block representations $$\begin{gathered}
M_+(\l)=(N_+(\l)\;\; M(\l)):\cK_2\oplus\cH_1 \to \cH_1, \quad
\l\in\bC_+\label{3.14.11}\\
M_-(z)=(N_-(z)\;\; M(z))^\top: \cH_1\to\cK_2\oplus\cH_1  , \quad
z\in\bC_-\label{3.14.12}\\
\cM(\l)=\begin{pmatrix} \tfrac i 2 I_{\cK_2} & 0 \cr  N_+(\l) &
M(\l)\end{pmatrix}:\cK_2\oplus\cH_1 \to \cK_2\oplus\cH_1, \quad
\l\in\bC_+\label{3.14.13}\\
\cM(z)=\begin{pmatrix} -\tfrac i 2 I_{\cK_2} & N_-(z) \cr 0 & M(z)
\end{pmatrix} :\cK_2\oplus\cH_1 \to \cK_2\oplus\cH_1, \quad
z\in\bC_-\label{3.14.14}\end{gathered}$$ induce the Nevanlinna operator functions $M(\l)\,(\in [\cH_1])$ and $\cM(\l)\,
(\in [\cH_0])$;

4\) the following identity holds $$\label{3.14.15}
\cM(\mu)-\cM^*(\l)=(\mu-\ov\l) \g_+^*(\l)\g_+(\mu), \quad \mu,\l\in\bC_+.$$

The statement 1) is immediate from and Proposition \[pr2.5\], 3).

2\) It follows from that $$\begin{gathered}
\G_0\up \hat\gN_\l(A)=\{\{\hat f_\l, \{G_0\hat f_\l-i \,F_2k',\, k'\}\}:\hat
f_\l\in \hat\gN_\l(A), \; k'\in\cK'\}, \;\;\;\;\l\in\bC_+\\
P_1\G_0\up \hat\gN_z (A)=\{\{\hat f_z, \{P_1 G_0\hat f_z, \, k'\}\}:\hat f_z\in
\hat\gN_z(A), \; k'\in\cK'\}, \;\;\;\; z\in\bC_-.\end{gathered}$$ This and imply that the equalities and correctly define the operator functions $\g_+(\cd)$ and $\g_-(\cd)$ such that holds.

Next, one can prove the identities - in the same way as in [@Mog06.2 Proposition 3.15]. These identities show that the functions $\g_\pm(\cd)$ are holomorphic on their domains.

3)-4). Combining with one obtains $$M_+(\l)=\lb {\{G_0\hat f_\l-iF_2k',\, k'\}\choose \{M_{\Pi+}(\l)G_0 \hat f_\l
+F_1 k',\, F_0^* G_0\hat f_\l +F'k'\}}:\hat f_\l\in \hat\gN_\l (A),\; k'\in\cK'
\rb .$$ Letting here $h_0=G_0\hat f_\l-iF_2k'$ and taking into account we get $$M_+(\l)=\lb {\{h_0,\, k'\}\choose \{M_{\Pi+}(\l)h_0 +(F_1+i M_{\Pi+}(\l)F_2
)k',\, F_0^* h_0 +(F')^* k'\}}:h_0\in\cH_0,\; k'\in\cK' \rb .$$ This equality is equivalent to . The identity is proved in the same way as similar one in [@Mog06.2 Proposition 3.17]. Finally, implies that $\cM(\cd) $ and $M(\cd)$ are Nevanlinna operator functions.

In the next proposition we show that under the condition $\dim \cH_0<\infty $ formula gives the general form of a boundary relation $\BR$ for $A^*$.

\[pr3.4\] Let $A$ be a closed symmetric linear relation in $\gH$ and let $\BR$ be a boundary relation for $A^*$ such that $\dim \cH_0<\infty $. Then:

1\) $n_\pm(A)<\infty$ and $\dom\G=A^*$;

2\) the subspaces $\cK_\G'$ and $\cK_\G''$ (see and ) are mutually orthogonal and, consequently, the decompositions $$\label{3.14.16}
\cH_1:=\cK_1\oplus \cK_\G'\oplus\cK_\G'', \qquad \cH_0:=\cK_0\oplus
\cK_\G'\oplus\cK_\G''$$ hold with $\cK_1:=\cH_1\ominus (\cK_\G'\oplus\cK_\G'')$ and $\cK_0:=\cH_2\oplus \cK_1$;

3\) for any $\{\hat f, {h_0\choose h_1}\}\in \G$ the inclusion $h_0\in
\cK_0\oplus \cK_\G'$ is valid;

4\) for any $k'\in\cK_\G'$ there exists a unique triple of elements $ h_2\in
\cH_2, \; k_1\in\cK_1$ and $m'\in\cK_\G'$ such that $$\label{3.15}
\lb 0, {-i h_2+k' \choose k_1+m'} \rb\in\G.$$ Hence the relation defines the operators $F_2 \in [\cK_\G',\cH_2],
\;F_1 \in [\cK_\G',\cK_1] $ and $F'\in [\cK_\G']$ via $$\label{3.16}
F_2 k'= h_2, \qquad F_1 k'=k_1, \qquad F'k'=m'\quad (k'\in\cK_\G').$$ Moreover, these operators satisfy ;

5\) there exist linear maps $G_j:A^*\to \cK_j, \; j\in\{0,1\}$ such that $\{\cK_0\oplus\cK_1, G_0, G_1\}$ is a boundary triplet for $A^*$ and $\G$ admits the representation with $\cK'=\cK_\G', \;\cK''=\cK_\G''$ and the operator $F_0$ given by .

1\) Since $A=\ker \G$ and $\dim (\HH)<\infty$, it follows that $\dim (\dom\G /
A)<\infty$. Hence $\dom \G=\ov{\dom \G}=A^*$ and, consequently, $n_+(A)+n_-(A)=\dim (A^*/ A) <\infty$.

2\) - 3). Let $\{\hat f, {h_0\choose h_1}\}\in \G$ and $k''\in\cK_\G''$, so that $\{0, {0\choose k''}\}\in \G$. Then by $(h_0, k'')=0$, which implies that $h_0\in \cH_0\ominus \cK_\G''$.

Assume now that $k'\in\cK_\G'$, so that $\{0, {k'+h_2\choose h_ 1}\}\in \G$ with some $h_2\in \cH_2 (\,\subset \cH_0 \ominus \cK_\G'' )$ and $h_1\in\cH_1$. Then the above statement gives $k'+h_2\in \cH_0\ominus \cK_\G''$ and , consequently, $k'\in \cH_0\ominus \cK_\G''$. Therefore $\cK_\G'\perp \cK_\G''$.

4\) Let $k'\in\cK_\G'$. Then according to and the first equality in $\{0, {k'-i h_2\choose k_1+m'+k''}\}\in \G$ with some $ h_2\in
\cH_2, \; k_1\in\cK_1,\; m'\in \cK_\G'$ and $k''\in\cK_\G''$, which in view of implies that $\left\{0, {k'-i h_2\choose k_1+m'}\right\}\in \G$. Let us show that such $ h_2, \; k_1$ and $m'$ are unique for a given $k'$. If $\left\{0, {k'-i\wt h_2\choose \wt k_1+ \wt m'}\right\}\in \G$ with some $\wt
h_2\in \cH_2, \;\wt k_1\in\cK_1$ and $\wt m'\in \cK_\G'$, then $\left\{0,
{i(h_2-\wt h_2)\choose (\wt k_1-k_1)+ (\wt m'-m')}\right\}\in \G$ and the identity yields $0=i|| h_2-\wt h_2||^2$. Therefore $ h_2-\wt h_2=0$ and, consequently, $(\wt k_1-k_1)+ (\wt m'-m')\in \cK_\G''$. Now the decomposition yields $\wt k_1-k_1=0, \; \wt m'-m'=0$, so that $\wt
h_2=  h_2, \;\wt k_1=k_1$ and $\wt m'=m'$.

The equality for operators $F_2$ and $F'$ is immediate from identity applied to $\left\{0, {-i F_2k'+k' \choose
F_1k'+F'k'}\right\}$ $\;\; (k'\in\cK_\G')$.

5\) Combining and with and one obtains $$\label{3.17}
\G_\infty:=\{0\}\oplus \mul\G=\lb \left\{0, {\{-i F_2k',\,k', \, 0 \} \choose
\{F_1k', \, F'k', \, k''\}}\right\}: k'\in\cK_\G', \; k''\in\cK_\G''  \rb.$$ Let $T\subset \G$ be a linear relation from $\gH^2$ to $\HH$ given by $$\label{3.18}
T=\lb \lb \hat f, {h_0 \choose h_1}\rb\in\G: h_0\in \cK_0, \; h_1\in
\cK_0\oplus \cK_\G' \rb.$$ Then in view of and statement 3) $T\cap \G_\infty=\{0\}$ and the decomposition $$\label{3.19}
\G=T\dotplus\G_\infty$$ is valid. This and the equality $\dom \G=A^*$ imply that $T$ is the operator with the domain $\dom T=A^*$. Moreover, applying the Green’s identity to elements $\left\{0, {\{-i F_2k',\,k', 0 \} \choose \{F_1k', \,
F'k', \, 0\}}\right\}\in \G_\infty $ and $\left\{\hat f, {\{k_0,\,0,\, 0 \}
\choose \{k_1, \, m', \, 0\}}\right\}\in T $ one obtains $$\begin{gathered}
0=(m',k')_{\cK'}-(k_0,F_1 k')_{\cK_0}-(k_0,F_2 k')_{\cK_0}=(
m',k')_{\cK'}-(k_0,F_0 k')_{\cK_0}, \;\;\;k'\in\cK_\G'.\end{gathered}$$ Hence $m'=F_0^* k_0$ and formula can be written as $$\label{3.20}
T=\lb \lb \hat f, {\{G_0\hat f,\, 0, \, 0\} \choose \{G_1\hat f,\, F_0^*G_0\hat
f, \, 0 \}}\rb: \hat f \in A^* \rb,$$ where $G_0:=P_{\cK_0\oplus \{0\}}T$ and $G_1:=P_{\{0\}\oplus \cK_1}T$ are linear maps from $A^*$ to $\cK_0$ and $\cK_1$ respectively. Combining with and we arrive at the representation of $\G$.

Now it remains to show that the operators $G_0$ and $G_1$ form the boundary triplet $\Pi=\{\cK_0\oplus\cK_1, G_0, G_1\}$ for $A^*$. Applying the identity to elements of the linear relation $T$ (see ) one obtains the Green’s identity for $G_0$ and $G_1$.

To prove surjectivity of the map $G=(G_0\;\;G_1)^\top$ assume that $\{m_0,\,m_1\}\in \cK_0\oplus\cK_1$ and $(G_0\hat f, m_0)+(G_1\hat f, m_1)=0$ for all $\hat f\in A^*$. Then the element $$\label{3.21}
\hat\f= \lb 0, {\{m_1+i\,P_2m_0,\,0, \, 0 \}\choose \{-P_1m_0, \;
F_0^*(m_1+i\,P_2m_0), \; 0\}}  \rb \in \gH^2\oplus (\cH_0\oplus \cH_1)$$ satisfies and and by assertion (a) in the proof of Proposition \[pr3.3\] $\hat\f$ satisfies the identity for all $\{{f\choose f'}, {h_0\choose h_1}\}\in\G$. Therefore by Definition \[def3.2\] and $\hat\f\in \G_\infty$, which in view of gives $m_0=0$ and $m_1=0$. This implies that $\ran
G=\cH_0\oplus\cH_1$.

The following two corollaries arise from Propositions \[pr3.3\] and \[pr3.4\].

\[cor3.5\] Let $\BR$ be a boundary relation for $A^*$ with $\dim\cH_0 < \infty$ and let $n_\G:=\dim (\mul\G)$. Then: 1) $n_-(A)\leq n_+(A)<\infty$ and $$\label{3.22}
\dim \cH_0=n_+(A)+n_\G, \qquad \dim \cH_1=n_-(A)+n_\G.$$

2\) in the case $\mul\G=\{0\}$ (and only in this case) the relation $\G$ turns into the boundary triplet for $A^*$. More precisely, if $\mul\G=\{0\}$, then $\G_0=P_{\cH_0\oplus \{0\}}\G$ and $\G_1=P_{\{0\}\oplus\cH_1 }\G$ are operators and $\Pi=\bta$ is a boundary triplet for $A^*$.

1\) Let $\{\cK_0\oplus\cK_1, G_0,G_1\}$ be a boundary triplet for $A^*$ defined in Proposition \[pr3.4\], 5). Then according to one has $\dim
\cK_0=n_+(A)$ and $\dim \cK_1=n_-(A)$. Moreover, implies that $\dim (\cK_\G'\oplus\cK_\G'')=n_\G$. This and decompositions of $\cH_0$ and $\cH_1$ yield .

2\) If $\mul\G=\{0\}$, then by $\cK_\G'=\cK_\G''=\{0\}$. Therefore in view of and the boundary triplet $\{\cK_0\oplus\cK_1, G_0,G_1\}$ satisfies the equalities $\cK_j=\cH_j$ and $G_j=\G_j, \; j\in\{0,1\}$.

\[cor3.6\] Assume that $A$ is a closed symmetric linear relation in $\gH$, $\cH_0$ is a Hilbert space with $\dim\cH_0 < \infty$, $\cH_1$ is a subspace in $\cH_0$ and $\BR$ is a linear relation such that $\dom\G=A^*$ and $\ker\G=A$. If the identity is satisfied for $\G$ and $$\label{3.23}
\dim (\HH)=n_+(A)+n_-(A)+2\,n_\G,$$ then $\G$ is a boundary relation for $A^*$.

Applying the same arguments as in the proof of Proposition \[pr3.4\] one obtains decompositions of $\cH_0$ and $\cH_1$ and the equality with $\cK'=\cK_\G', \; \cK''=\cK_\G''$ and operators $G_j:A^*\to
\cK_j, \; j\in\{0,1\},$ satisfying the Green’s identity . Moreover, by $\dim (\cK_\G' \oplus\cK_\G'')=n_\G$ and together with gives $\dim (\cK_0\oplus\cK_1)=n_+(A)+n_-(A)$. Observe also that in view of $\ker G=\ker \G= A $ (here $G=(G_0\;\;G_1)^\top$) and hence $$\dim (\dom G / \ker G)=\dim (A^* / A) =n_+(A)+n_-(A)=\dim (\cK_0\oplus\cK_1).$$ This implies that $\ran G=\cK_0\oplus\cK_1$ and, consequently, the operators $G_0$ and $G_1$ form the boundary triplet $\{\cK_0\oplus\cK_1, G_0,G_1\}$ for $A^*$. Therefor by Proposition \[pr3.3\] $\G$ is the boundary relation for $A^*$.

In the case $\cH_0=\cH_1=:\cH$ the above statements on boundary relations can be rather simplified. Namely, the following corollary is immediate from Propositions \[pr3.3\] - \[pr3.4\].

\[cor3.7\]

Assume that $\Pi=\{\cK,G_0,G_1\}$ is a boundary triplet for $A^*$ (in the sense of [@GorGor]), $\cK'$ and $\cK''$ are Hilbert spaces and $$\cH:=\cK\oplus \cK'\oplus\cK''.$$ Moreover, let $F\in [\cK',\cK]$ and $ F'=(F')^*\in [\cK']$ be linear operators. Then the equality $$\label{3.25}
\G=\lb \lb \hat f, {\{G_0\hat f,\, k',\, 0\}\}\choose \{G_1\hat f +F k',\, F^*
G_0\hat f +F'k',\, k''\}} \rb :\hat f\in A^*,\; k'\in\cK', \; k''\in\cK'' \rb$$ defines the boundary relation $\G:\gH^2\to\cH^2$ for $A^*$.

If in addition $\cK''=\{0\}$, then the following statements are valid:

1\) the equality defines the self-adjoint extension $A_0$ of $A$ and $A_0=\ker G_0$;

2\) the relations $$\begin{gathered}
\hat\g (\l)= (\G_0\up\hat \gN_\l (A))^{-1},\qquad \g (\l)=\pi_1\hat\g
(\l),\;\; \;\;\l\in\rho (A_0);\label{3.26}\\
 {\rm gr} M(\l)=\lb \{h,h'\}\in \cH^2: \lb {f\choose \l f}, {h\choose h'}\rb
\in\G \;\;\text{for some} \;\; f\in\gH \rb, \;\;\l\in\rho (A_0), \label{3.27}\end{gathered}$$ define the $\g$-field $\g(\cdot):\rho (A_0) \to[\cH,\gH]$ and the Weyl function $M(\cdot):\rho (A_0)\to [\cH]$ corresponding to $\G$. Moreover, $\g(\l)$ and $M(\l)$ are associated with the $\g$-field $\g_\Pi(\l)$ and the Weyl function $M_\Pi(\l)$ for the triplet $\Pi $ (see ) via $$\label{3.28}
\g(\l)=\g_{\Pi}(\l)P_{\cK}, \qquad M(\l)=\begin{pmatrix} M_{\Pi}(\l) & F \cr
F^* & F'\end{pmatrix}: \cK\oplus\cK'\to \cK\oplus\cK', \quad \l\in\rho (A_0)$$ and the following identities hold $$\begin{gathered}
\g(\mu)=\g(\l)+(\mu-\l)(A_0-\mu)^{-1} \g(\l), \;\;\; \mu,\l\in\rho
(A_0)\label{3.29}\\
M(\mu)-M^*(\l)=(\mu-\ov\l) \g^*(\l)\g(\mu), \quad \mu,\l\in\rho
(A_0).\label{3.30}\end{gathered}$$ The identity implies that $M(\cd)$ is a Nevanlinna operator function.

Conversely, let $\G:\gH^2\to\cH^2$ be a boundary relation for $A^*$ with $\dim\cH<\infty$ and let $\cK_\G'=\dom (\mul\G), \; \cK_\G''=\mul (\mul\G)$ (c.f. and ). Then $$n_+(A)=n_-(A)=\dim \cH-n_\G$$ (here $n_\G=\dim (\mul\G)$) and the following statements hold:

1\) $\cH=\cK\oplus\cK_\G'\oplus \cK_\G''$, where $\cK=\cH\ominus (\cK_\G'\oplus
\cK_\G'')$;

2\) $\G$ admits the representation with some boundary triplet $\Pi=\{\cK,G_0,G_1\}$ for $A^*$ and operators $F\in [\cK_\G',\cK]$ and $
F'=(F')^*\in [\cK_\G']$.

\[rem3.8\] 1) In the case $\cH_0=\cH_1=:\cH$ our Definitions \[def3.2\] and \[def3.2a\] of the boundary relation $\G:\gH^2\to\cH^2$ for $A^*$ and the corresponding Weyl family $M(\cd)$ coincide with that introduced in [@DM06].

2\) The identities and were proved in [@DM06] (see also [@DM09]).

Canonical systems
=================

Notations
---------

Let $\cI=\langle a,b\rangle\; (-\infty\leq a< b\leq\infty)$ be an interval of the real line (in the case $a> -\infty $ (resp. $b<\infty$) the endpoint $a$ (resp. $b$) may or may not belong to $\cI$) and let $\bH$ be a finite-dimensional Hilbert space. Denote by $\cL_{loc}^1(\cI)$ the set of all Borel operator functions $F(\cd)$ defined almost everywhere on $\cI$ with values in $[\bH]$ and such that $\int\limits_{[\a,\b]}||F(t)||\,dt<\infty$ for any finite segment $[\a,\b]\subset \cI$.

Next, denote by $\AC$ the set of all functions $f(\cd):\cI\to \bH$, which are absolutely continuous on any segment $[\a,\b]\subset \cI$. Moreover, let $\ACf$ be the set of all functions $f\in\AC$ with the following property: if $a\in
\cI$ (resp. $b\in\cI$), then $f(a)=0$ (resp. $f(b)=0$); otherwise $f(t)=0$ on some interval $(a,\a)\subset \cI$ (resp. $(\b,b)\subset \cI$). Clearly, in the case of a finite segment $\cI=[a,b]$ the set $AC_0(\cI)$ coincides with the set of all functions $f\in \AC$ such that $f(a)=f(b)=0$.

Let $\D(\cd)\in \cL_{loc}^1(\cI)$ be an operator function such that $\D(t)\geq
0 $ a.e. on $\cI$. Denote by $\lI$ the linear space of all Borel functions $f(t)$ defined almost everywhere on $\cI$ with values in $\bH$ and such that $\int\limits_{\cI}(\D (t)f(t),f(t))_\bH \,dt<\infty$. As is known [@Kac50; @DunSch] $\lI$ is a semi-Hilbert space with the semi-definite inner product $(\cd,\cd)_\D$ and semi-norm $||\cd||_\D$ given by $$(f,g)_\D=\int_{\cI}(\D (t)f(t),g(t))_\bH \,dt, \quad
||f||_\D=((f,f)_\D)^{\frac1 2 }, \qquad f,g\in \lI.$$ The semi-Hilbert space $\lI$ gives rise to the Hilbert space $\LI=\lI /
\{f\in\lI: ||f||_\D=0\}$, i.e., $\LI$ is the Hilbert space of all equivalence classes ($f$ equivalent to $g$ means $\D(t) (f(t)-g(t))=0$ a.e. on $\cI$) in $\lI$. The inner product and norm in $\LI$ are defined by $$(\wt f, \wt g)=(f,g)_\D, \quad ||\wt f||=(\wt f, \wt f)^{\frac 1 2}=||f||_\D,
\qquad \wt f, \wt g\in\LI,$$ where $f\in\wt f \; (g\in\wt g)$ is any representative of the class $\wt f$ (resp. $\wt g$).

In the sequel we systematically use the quotient map $\pi$ from $\lI$ onto $\LI$ given by $\pi f=\wt f(\ni f), \; f\in \lI$. Moreover, we let $\wt\pi:=\pi\oplus\pi: (\lI)^2 \to (\LI)^2$, so that $\wt \pi\{f,g\}=\{\wt f,
\wt g\}, \;\; f,g \in \lI$. It is clear that $\ker \pi=\{f\in\lI:\, \D(t)f(t)=0
\;\;\text{a.e. on}\;\;\cI\}$.

Minimal and maximal relations
-----------------------------

Let as above $\cI=\langle a,b\rangle \;(-\infty \leq a <b\leq\infty )$ be an interval and let $\bH$ be a Hilbert space with $n:=\dim \bH\leq\infty$. Moreover, let $B(\cd), \D (\cd)\in \cL_{loc}^1(\cI)$ be operator functions such that $B(t)=B^*(t)$ and $\D(t)\geq 0$ a.e. on $\cI$ and let $J\in [\bH]$ be a signature operator ( this means that $J^*=J^{-1}=-J$).

A canonical system (on an interval $\cI$) is a system of differential equations of the form $$\label{4.1}
J y'(t)-B(t)y(t)=\D(t) f(t), \quad t\in\cI,$$ where $f(\cd)\in \lI$. Together with we consider also the homogeneous canonical system $$\label{4.1.1}
J y'(t)-B(t)y(t)=\l \D(t) y(t), \quad t\in\cI, \quad \l\in\bC.$$ A function $y\in\AC$ is a solution of (resp. ) if the equality (resp. holds a.e. on $\cI$. In the sequel we denote by $\cN_\l$ the linear space of all solutions of the homogeneous system belonging to $\lI$: $$\label{4.1.2}
\cN_\l=\{y\in \AC\cap \lI: J y'(t)-B(t)y(t)=\l \D(t) y(t) \;\;\text{a.e. on}
\;\; \cI\}, \quad\l\in\bC.$$ It follows from that $\dim \cN_\l\leq \dim \bH <\infty$.

As was shown in [@KogRof75] the set of all solutions of such that $\D(t)y(t)=0$ (a.e. on $\cI$) does not depend on $\l$. This enables one to introduce the following definition.

\[def4.0\]$\,$[@KogRof75] The null manifold $\cN$ of the system is the subspace of $\cN_\l\;(\l\in\bC)$ given by $$\label{4.1.3}
\cN=\cN_\l\cap\ker\pi=\{y\in \AC: J y'(t)-B(t)y(t)=\l \D(t) y(t)\;\; \text{and}
\;\; \D(t)y(t)=0 \;\;\text{a.e. on} \;\; \cI\}.$$

For each $c\in\cI$ denote by $\bH_c$ the subspace $$\label{4.1.4}
\bH_c=\{y(c): y(\cd)\in \cN\}(\subset \bH)$$ and let $$\label{4.1.5}
k_\cN=\dim \cN=\dim \bH_c.$$ Clearly, $\cN\subset\cN_\l \; (\l\in\bC)$ and for any fixed $\l_0 \in \bC
\setminus \bR$ $$\label{4.1.6}
\cN=\cN_{\l_0}\cap \cN_{\ov \l_0}.$$ According to [@Orc; @Kac83; @Kac84; @LesMal03] the canonical system induces the *maximal relations* $\tma$ in $\lI$ and $\Tma$ in $\LI$, which are defined by $$\begin{gathered}
\tma=\{\{y,f\}\in\lI\times\lI:y\in\AC \;\;\text{and}\;\; J y'(t)-B(t)y(t)=\D(t)
f(t)\;\;\text{a.e. on}\;\; \cI \},\label{4.2}\\
\Tma=\wt\pi \tma=\{\{\wt y, \wt f\}\in\LI\oplus\LI: \wt y=\pi
y\;\;\text{and}\;\; \wt f=\pi f\;\;\text{for some}\;\;
\{y,f\}\in\tma\}.\label{4.3}\end{gathered}$$ For $\{y,f\}, \; \{z,g\}\in\tma$ and a segment $[\a,\b]\subset\cI$ integration by parts yields $$\int_{[\a,\b]}(\D(t)f(t),z(t))\,dt-\int_{[\a,\b]}(\D(t)y(t),g(t))\,dt=(J
y(\b),z(\b))-(J y(\a),z(\a)).$$ Hence there exist the limits $$\label{4.4}
[y,z]_a:=\lim_{\a \downarrow a}(J y(\a),z(\a)),\qquad [y,z]_b:=\lim_{\b
\uparrow b}(J y(\b),z(\b)), \quad y,z \in\dom\tma$$ and the following Lagrange’s identity holds $$\label{4.5}
(f,z)_\D-(y,g)_\D=[y,z]_b -[y,z]_a,\quad \{y,f\}, \; \{z,g\} \in\tma.$$ Formula defines the boundary bilinear forms $[\cd,\cd]_a $ and $[\cd,\cd]_b $ on $\dom \tma$, which play an essential role in what follows. By using this form we define the *minimal relations* $\tmi$ in $\lI$ and $\Tmi$ in $\LI$ via $$\begin{gathered}
\tmi=\{\{y,f\}\in\tma:[y,z]_a=0 \;\; \text{and}\;\; [y,z]_b=0\;\;\text{for
every}\;\; z\in \dom \tma \},\label{4.6}\\
\Tmi= \wt\pi \tmi=\{\{\wt y, \wt f\}\in\LI\oplus\LI: \,\wt y=\pi
y\;\;\text{and}\;\; \wt f=\pi f\;\;\text{for some}\;\;
\{y,f\}\in\tmi\}.\label{4.7}\end{gathered}$$ Moreover, introduce linear relations $\cT_0$ in $\lI$ and $T_0$ in $\LI$ by letting $$\begin{gathered}
\label{4.8}
\cT_0=\{\{y,f\}\in \tma:\,  y\in\ACf\}, \quad T_0=\wt\pi\cT_0.\end{gathered}$$ It is clear that $\cT_0\subset \tmi\subset\tma$ and $T_0\subset\Tmi\subset\Tma$.

Our next goal is to show that $\Tmi$ is a closed symmetric linear relation and $T_{min}^*=\Tma$. To do this we start with the following definition.

\[def4.1\] A finite endpoint $a$ (resp. $b$) of the interval $\cI=\langle a,b \rangle$ is said to be a regular endpoint of the canonical system if $a\in\cI$ (resp. $b\in\cI$). The canonical system is called regular if both endpoints $a$ and $b$ are regular; otherwise it is called singular.

Clearly, in the case of the regular endpoint $a$ (resp. b) integrals $\int\limits_{[a,c]}||B(t)||\, dt$ and $\int\limits_{[a,c]}||\D(t)||\, dt$ (resp. $\int\limits_{[c,b]}||B(t)||\, dt$ and $\int\limits_{[c,b]}||\D(t)||\,
dt$) are finite for any $c\in (a,b)$.

If the system is regular, then the identity can be written as $$\label{4.10}
(f,z)_\D-(y,g)_\D=(J y(b),z(b)) -(J y(a),z(a)),\quad \{y,f\}, \; \{z,g\}
\in\tma.$$ In the case of the regular system we associate with every subspace $\bK\subset\bH$ two pairs of linear relations $\cT_\bK, \;\cT_{\bK *} $ in $\lI$ and $T_\bK, \; T_ {\bK^*} $ in $\LI$ given by $$\begin{gathered}
\cT_\bK=\{\{y,f\}\in\tma: y(a)\in\bK \;\;\text{and} \;\; y(b)=0\},\label{4.11}\\
\cT_{\bK *}=\{\{y,f\}\in\tma: y(a)\in (J\bK)^\perp\}, \label{4.12}\\
T_\bK=\wt\pi \cT_\bK, \qquad T_{\bK *}=\wt\pi \cT_{\bK *}.\label{4.13}\end{gathered}$$

\[lem4.2\] If the system is regular, then for any subspace $\bK\in\bH$ and $\l\in\bC$ $$\label{4.14}
\ran (T_\bK-\l)=(\ker (T_{\bK*}-\ov\l))^\perp.$$

It follows from and that $\ran (T_\bK-\l)$ is the set of all $\wt f\in\LI$ with the following property: there are $f\in\wt f$ and $y\in\AC$ such that $$\label{4.15}
y(a)\in\bK, \quad y(b)=0 \;\;\text{and}\;\; \{y,f+\l y\}\in\tma.$$ On the other hand, and imply that $$\label{4.16}
\ker (T_{\bK*}-\ov\l)=\{\wt z\in\LI:\, \{z, \ov\l z\}\in\tma \;\;\text{and}\;\;
z(a)\in (J\bK)^\perp \;\;\text{for some}\;\; z\in\wt z \}.$$ Let $\wt f\in \ran (T_\bK-\l), \; \wt z\in \ker (T_{\bK*}-\ov\l)$ and let $\{y,f+\l y\}, \; \{z, \ov\l z\}$ be the corresponding elements of $\tma$ from , . Applying to these elements the Lagrange’s identity one obtains $$(f,z)_\D=-(Jy(a), z(a))=0.$$ Hence $(\wt f,\wt z)=0$ and, consequently, $\ran (T_\bK-\l)\subset (\ker
(T_{\bK*} -\ov\l))^\perp$.

To prove the inverse inclusion assume that $\wt f\in (\ker (T_{\bK*} -\ov\l))
^\perp$ and let $f\in\wt f, \; f\in\lI$. Moreover, let $y\in\AC$ be the solution of the equation $$Jy'-B(t)y=\D(t) (f(t)+\l y)$$ with the initial data $y(b)=0$, so that $\{y, f+\l y\}\in\tma$. Next, for every $h\in (J\bK)^\perp$ let $z_h\in \AC$ be the solution of the equation $$Jz'-B(t)z= \ov\l \D(t)z$$ with the initial data $z_h(a)=h$ and let $\wt z_h=\pi z_h$. Since $\{z_h, \ov\l
z_h \}\in \tma$ and $z_h(a)\in (J\bK)^\perp $, it follows from that $\wt z_h\in \ker (T_{\bK*}-\ov\l)$ and, therefore, $(\wt f, \wt z_h)=0$. Now application of the identity to $\{y, f+\l y\}$ and $\{z_h,
\ov\l z_h\}$ gives $$(Jy(a), h)=(Jy(a),z_h(a))-(Jy(b),z_h(b))=-(f,z_h)_\D=-(\wt f, \wt z_h)=0,
\;\;\;\; h\in (J\bK)^\perp,$$ which implies that $y(a)\in\bK$. Thus for an arbitarry $\wt f\in (\ker
(T_{\bK*} -\ov\l)) ^\perp$ we have constructed $f\in\wt f$ and $y\in\AC$ satisfying . This gives the requiered inclusion $(\ker (T_{\bK*}
-\ov\l)) ^\perp \subset \ran (T_\bK-\l)$.

\[lem4.3\] If the system is regular, then for any subspace $\bK\subset\bH$ $$\label{4.17}
(T_\bK)^*=T_{\bK *}.$$ In the particular case $\bK=\{0\}$ formula gives $T_0^*=\Tma$.

Applying to $\{y,f\}\in \cT_\bK$ and $\{z,g\}\in \cT_{\bK *}$ we obtain $ (f,z)_\D-(y,g)_\D=0$. Therefore by one has $T_{\bK
*}\subset (T_\bK)^*$.

Let us prove the inverse inclusion. First observe that in view of (with $\l=0$) $\dim \ker T_{\bK *}\leq \dim \cN_0 <\infty$. Hence $\ker T_{\bK
*}$ is a closed subspace in $\LI$ and gives $$\label{4.18}
\ker T_{\bK *}=(\ran T_\bK)^\perp.$$

Let $\{\wt z, \wt f\}\in (T_\bK)^*$. Choose $f\in\wt f, \; f\in\lI$ and let $y\in \AC$ by the solution of with $y(a)=0$. Then $\{y,f\}\in
\cT_{\bK *}$ and by $\{\wt y, \wt f\}\in T_{\bK *}(\subset
(T_\bK)^*)$ (here $\wt y=\pi y$). Thus $\{\wt z-\wt y,0\}\in (T_\bK)^*$, which implies that $\wt z-\wt y \in (\ran T_\bK)^\perp$. Therefore by $\wt z -\wt y\in \ker T_{\bK *}$, so that $\{\wt z
-\wt y, 0\}\in T_{\bK *}$. Now representing $\{\wt z, \wt f\}$ as $$\{\wt z, \wt f\}=\{\wt z -\wt y, 0\}+\{\wt y, \wt f\}$$ and taking into account that both terms in the right hand part belong to $T_{\bK *}$ one obtains $\{\wt z, \wt f\}\in T_{\bK *}$. This proves the desired inclusion $(T_\bK)^*\subset T_{\bK *}$.

\[lem4.4\] Let the singular canonical system be defined on an interval $\cI=\langle a,b \rangle$. For every finite segment $\cI'=[a',b']\subset\cI$ denote by $\cT_{max}^{\cI'}$ and $T_{max}^{\cI'}$ maximal relations in $\cL_\D^2(\cI')$ and $L_\D^2(\cI')$ respectively induced by the restriction of the system onto $\cI'$. Then there exist a finite segment $\cI_0'\subset \cI $, a point $c\in\cI_0'$ and a subspace $\bH_0\subset \bH$ with the following property: for any segment $\cI'\supset\cI_0'$ and for any $\{\wt y, \wt f\}\in T_{max}^{\cI'}$ there exists a unique function $\overset\circ y \in AC (\cI') $ such that $\overset\circ y\in \wt y,\;\;
\overset\circ y(c)\in \bH_0 $ and $\{\overset\circ y, f\}\in \cT_{max}^{\cI'}$ for any $f\in\wt f$.

Fix a point $c\in\cI $ and for any segment $\cI'\ni c$ put $$\begin{gathered}
\cN^{\cI'}=\{y\in AC(\cI'): \, J y'(t)-B(t)y(t)=0 \;\;\text {and}\;\;
\D(t)y(t)=0\;\;\text{a.e. on}\;\;\cI'\},\label{4.19}\\
\bH^{\cI'}=\{y(c):\, y(\cd)\in \cN^{\cI'}\}.\nonumber\end{gathered}$$ Clearly, the inclusion $\cI_1'\subset\cI_2'$ yields $\bH^{\cI_2'}\subset
\bH^{\cI_1'}$. Since $\dim \bH<\infty$, this implies that there exists a finite segment $\cI_0'\subset\cI$ such that $\bH^{\cI'}=\bH^{\cI_0'}$ for all $\cI'\supset \cI_0'$. Put $\bH_0:=(\bH^{\cI_0'})^\perp$ and show that such $\cI_0'$ and $\bH_0$ have the desired property.

If $\cI'\supset \cI_0'$ and $\{\wt y, \wt f\}\in T_{max}^{\cI'}$, then according to definition there is a function $y\in AC (\cI')$ such that $y\in \wt y $ and for any $f\in\wt f$ the equality holds a.e. on $\cI'$. Let $\overset\circ y \in AC(\cI')$ be the solution of the equation on $\cI'$ with the initial data $\overset\circ
y(c)=P_{\bH_0}y(c)(\in\bH_0)$ and let $\f= y- \overset\circ y$. Then $J
\f'(t)-B(t)\f(t)=0$ a.e. on $\cI'$ and $\f(c)=y(c)-\overset\circ
y(c)\in\bH_0^\perp$. Since $\bH_0^\perp=\bH^{\cI_0'}=\bH^{\cI'}$, it follows that $\f(c)\in \bH^{\cI'}$. Hence $\f\in\cN^{\cI'}$ and, consequently, $\D(t)(y(t)-\overset\circ y(t))=\D(t)\f(t)=0$ a.e. on $\cI'$. This means that $\overset\circ y\in \wt y, \; \overset\circ y(c)\in \bH_0$ and $\{\overset\circ
y, f\}\in \cT_{max}^{\cI'}$.

To prove uniqueness of such $\overset\circ y$ assume that $z\in AC (\cI')$ has the same properties, i.e., $z\in \wt y,\;\; z(c)\in \bH_0$ and $\{z, f\}\in
\cT_{max}^{\cI'}$ for any $f\in\wt f$. Then the function $\psi:= \overset \circ
y -z$ satisfies the equalities $J\psi'(t)-B(t)\psi (t)=0 $ and $\D(t)\psi(t)=0$ a.e. on $\cI'$. Hence $\psi\in\cN^{\cI'}$ and, consequently, $\psi
(c)\in\bH^{\cI'}(=\bH_0^\perp)$. On the other hand, $\psi(c)\in\bH_0$, so that $\psi(c)=0$. Therefore $\psi=0$ and hence $\overset \circ y=z$.

\[pr4.5\] Let $T_0$ be the linear relation in $\LI$ given by . Then $$\begin{gathered}
T_0^*=\Tma\label{4.22}\end{gathered}$$

In the case of the regular system the equality was proved in Lemma \[lem4.3\].

Assume now that the system is singular. Then applying the Lagrange’s identity to $\{y,f\}\in\tma$ and $\{z,g\}\in\cT_0 $ we obtain $$\label{4.23}
(f,z)_\D-(y,g)_\D=0.$$ Therefore by and one has $\Tma\subset T_0^*$.

Let us prove the inverse inclusion $T_0^*\subset \Tma$. Assume that $\{\wt y,
\wt f\} \in T_0^*$ and choose $y,f\in \lI$ such that $\pi y=\wt y$ and $\pi
f=\wt f$. Next, for every finite segment $\cI'=[a',b']\subset\cI$ denote by $y_{\cI'}$ and $f_{\cI'}$ the restrictions of the functions $y(\cd) $ and $f(\cd)$ onto $\cI'$ and let $\wt y_{\cI'}=\pi_{\cI'}y_{\cI'}, \; \wt
f_{\cI'}=\pi_{\cI'}f_{\cI'}$, where $\pi_{\cI'}$ is the quotient map from $\cL_\D^2(\cI')$ onto $L_\D^2(\cI')$. Assume also that $\cT_0^{\cI'}$ and $T_0^{\cI'}$ are linear relations in $\cL_\D^2(\cI')$ and $L_\D^2(\cI')$ respectively induced by the restriction of the system onto $\cI'$.

Let $\{z_{\cI'}, g_{\cI'}\}\in \cT_0^{\cI'}$ and let $z(t)$ and $g(t)\;
(t\in\cI)$ be functions obtained from $z_{\cI'}$ and $g_{\cI'}$ by means of their zero continuation onto $\cI$. Then $\{z,g\}\in \cT_0$ and, consequently, holds. Therefore $$\int_{\cI'}(\D(t)f_{\cI'}(t), z_{\cI'}(t))\, dt -\int_{\cI'}(\D(t)y_{\cI'}(t),
g_{\cI'}(t))\, dt=0, \quad \{z_{\cI'},g_{\cI'}\}\in\cT_0^{\cI'},$$ which implies that $\{\wt y_{\cI'}, \wt f_{\cI'}\}\in (T_0^{\cI'})^*$. Moreover, $(T_0^{\cI'})^*=T_{max}^{\cI'}$, because the restriction of onto $\cI'$ is a regular system. Thus, $\{\wt y_{\cI'}, \wt
f_{\cI'}\} \in T_{max}^{\cI'}$ for every finite segment $\cI'\subset\cI$.

Next, by Lemma \[lem4.4\] there exist a finite segment $\cI_0'\subset \cI $, a point $c\in\cI_0'$ and a subspace $\bH_0\subset \bH$ with the following property: for any finite segment $\cI'\supset\cI_0'$ there exists a unique function $\overset\circ y_{\cI'} \in AC (\cI') $ such that $\pi_{\cI'}\overset\circ y_{\cI'}= \wt y_{\cI'},\;\; \overset\circ
y_{\cI'}(c)\in \bH_0 $ and $\{\overset\circ y_{\cI'}, f_{\cI'}\}\in
\cT_{max}^{\cI'}$. Moreover, by using uniqueness of the function $\overset\circ
y_{\cI'}$ (for a given $\cI'$) one can easily verify that for any pair of segments $\cI_1', \; \cI_2'$ such that $\cI_0'\subset\cI_1'\subset
\cI_2'\subset\cI $ the restriction $\overset\circ y_{\cI_2'}\up \cI_1' $ coincides with $\overset\circ y_{\cI_1'}$. This allows us to introduce the function $\overset\circ y\in \AC$ by setting $\overset\circ y(t)=\overset\circ
y_{\cI'}(t), \; t\in\cI$, where $\cI'$ is an arbitrary segment such that $\cI_0'\subset\cI'\subset \cI $ and $t\in\cI'$. It is clear that $\pi
\overset\circ y=\wt y$ and $\{\overset\circ y, f\}\in \tma$, which gives the inclusion $\{\wt y, \wt f\}\in \Tma$. Therefore $T_0^*\subset \Tma$ and the equality is valid.

\[lem4.6\] Let the canonical system be given on an interval $\cI=[a,b\rangle$ with the regular endpoint $a$. Assume also that $\cT_1, \;\cT_2$ and $T_1,\;
T_2$ are linear relations in $\lI$ and $\LI$ respectively defined by $$\begin{gathered}
\cT_1=\{\{y,f\}\in\tma:\, [y,z]_b=0 \;\;\text{for every}\;\; z\in\dom\tma\},
\qquad T_1=\wt\pi\cT_1,\label{4.24}\\
\cT_2=\{\{y,f\}\in\tma:\, y(a)=0\}, \qquad T_2=\wt\pi\cT_2.\label{4.25}.\end{gathered}$$ Then $$\begin{gathered}
\label{4.26}
T_1^*=T_2.\end{gathered}$$

The inclusion $T_2\subset T_1^*$ follows from the identity applied to $\{y,f\}\in \cT_1$ and $\{z,g\}\in\cT_2$.

To prove the inverse inclusion assume that $\{\wt y, \wt f\}\in T_1^*$ and let $y,f\in \lI, \; \pi y=\wt y, \; \pi f =\wt f$. Moreover, for every $\b\in\cI$ let $\cI_\b:=[a,\b]$, let $y_\b$ and $f_\b$ be the restrictions of the functions $y(\cd)$ and $f(\cd)$ onto $\cI_\b$ and let $\wt y_\b=\pi_\b y_\b, \;
\wt f_\b=\pi_\b f_\b $, where $\pi_\b$ is the quotient map from $\cL_\D^2(\cI_\b)$ onto $L_\D^2 (\cI_\b)$. Consider also linear relations $\cT^\b, \; \cT^\b_1$ in $\cL_\D^2 (\cI_\b)$ and $T^\b, \; T^\b_1$ in $L_\D^2
(\cI_\b)$ given by $$\begin{gathered}
\cT^\b=\{\{y,f\}\in\cT_{max}^\b:\, y(\b)=0\}, \qquad
T^\b=\wt\pi_\b\cT^\b\nonumber\\
\cT_2^\b=\{\{y,f\}\in\cT_{max}^\b:\, y(a)=0\}, \qquad
T_2^\b=\wt\pi_\b\cT_2^\b.\label{4.27}\end{gathered}$$ It follows from (with $\bK=\bH$) that $(T^\b)^*=T_2^\b$ and the same arguments as in the proof of Proposition \[pr4.5\] give the inclusion $\{\wt y_\b, \wt f_\b\}\in T_2^\b, \;\b\in\cI$.

Next, according to definition of $T_2^\b$ there is a function $\ov
y_\b\in AC (\cI_\b)$ such that $\pi_\b \ov y_\b=\wt y_\b, \; \ov y_\b(a)=0$ and $J \ov y_\b'(t)-B(t)\ov y_\b(t)=\D(t)f_\b(t) $ a.e. on $\cI_\b$. Moreover, it is easily seen that for a given $\b\in\cI$ such a function is unique, so that $\ov y_{\b_1}=\ov y_{\b_2}\up \cI_{\b_1}$ for any $ \b_1<\b_2$. Therefore the equality $\ov y(t)=\ov y_\b(t), \; t\in\cI, \; \b>t$ correctly defines the function $\ov y\in \AC$ such that $\pi \ov y=\wt y$ and $\{\ov y, f\}\in\cT_2$. This implies that $\{\wt y, \wt f\}\in T_2$ and hence $T_1^*\subset T_2$.

\[pr4.7\] Let $a$ be a regular endpoint of the canonical system and let $\cT_a$ and $T_a$ be linear relations in $\lI$ and $\LI$ respectively given by $$\begin{gathered}
\label{4.28}
\cT_a=\{\{y,f\}\in\tma: y(a)=0 \;\;\text{and}\;\;\, [y,z]_b=0 \;\;\text{for
every}\;\; z\in\dom\tma\}, \qquad T_a=\wt\pi\cT_a.\end{gathered}$$ Moreover, let $T_0$ be the relation and let $\ov T_0$ be the closure of $T_0$. Then $T_a$ is a closed symmetric relation and $$\begin{gathered}
T_a=\ov T_0\label{4.30}\\
T_a^*=\Tma\label{4.31}\end{gathered}$$ Similarly if $b$ is a regular endpoint of the system and $$\begin{gathered}
\label{4.32}
\cT_b=\{\{y,f\}\in\tma: y(b)=0 \;\;\text{and}\;\;\, [y,z]_a=0 \;\;\text{for
every}\;\; z\in\dom\tma\}, \qquad T_b=\wt\pi\cT_b,\end{gathered}$$ then $T_b$ is a closed symmetric relation in $\LI$ and $$\begin{gathered}
\label{4.34}
T_b=\ov T_0, \qquad T_b^*=\Tma.\end{gathered}$$

Applying the Lagrange’s identity to $\{y,f\}\in \cT_a$ and $\{z,
g\}\in \tma$ one obtains the equality . Therefore $$\begin{gathered}
\label{4.35}
\Tma\subset T_a^* \;\;\;\text{and}\;\;\; T_a\subset \Tma^*.\end{gathered}$$ Moreover, by $T_a\subset \Tma$, which together with the first inclusion in shows that $T_a$ is symmetric.

Next, assume that $T_1$ and $T_2$ are the linear relations and . Since $T_1\subset\Tma$, it follows that $\Tma^*\subset T_1^*$ and by $\Tma^*\subset T_2$. Therefore for any $\{\wt y, \wt
f\}\in\Tma^*$ there is $\{y,f\}\in \tma$ such that $y(a)=0$ and $\wt \pi
\{y,f\}=\{\wt y, \wt f\}$. This and the identity give $$\begin{gathered}
[y,z]_b=(f,z)_\D-(y,g)_\D=0, \qquad z\in\dom \tma,\end{gathered}$$ which implies that $\{\wt y, \wt f\}\in T_a$. Hence $\Tma^*\subset T_a$ and by the second inclusion in $$\begin{gathered}
\label{4.36}
T_a=\Tma^*.\end{gathered}$$ Therefore $T_a$ is closed. Moreover, by $\Tma$ is also closed, which together with gives . Finally, combining with we arrive at .

Similarly one proves the relations .

\[cor4.8\] Under the assumptions of Lemma \[lem4.6\] the equality $T_2^*=T_1$ is valid.

It follows from that for each $\l\in\bC\setminus \bR$ the defect subspace of the close symmetric relation $T_a$ is $$\begin{gathered}
\gN_\l(T_a)=\ker (\Tma-\l)=\pi \cN_\l.\end{gathered}$$ Therefore $T_a$ has finite deficiency indices and gives $T_a\subset T_1\subset \Tma$. Consequently, $T_1$ is closed and the required equality $T_2^*=T_1$ follows from .

\[lem4.9\] Let the canonical system be defined on an interval $\cI=\langle a,
b \rangle $. For each subinterval $\cI_\b:=[\b, b\rangle\;(\b\in\cI) $ denote by $\cT_{max}^\b$ and $T_{max}^\b$ maximal relations induced by the restriction of the system onto $\cI_\b$ and let $\cT_1^\b$ and $T_1^\b$ be linear relations in $\cL_\D^2(\cI_\b)$ and $L_\D^2(\cI_\b)$ respectively given by $$\begin{gathered}
\label{4.37}
\cT_1^\b=\{\{y,f\}\in\cT_{max}^\b:\, [y,z]_b=0 \;\;\text{for every}\;\;
z\in\dom\cT_{max}^\b\}, \qquad T_1^\b=\wt\pi_\b\cT_1^\b.\end{gathered}$$ Then there exists a subinterval $\cI_{\b_0}\subset \cI$, a point $c\subset
\cI_{\b_0}$ and a subspace $\hat\bH\subset\bH$ with the following property: for any interval $\cI_\b\supset \cI_{\b_0}$ and for any $\{\wt y,\wt f\}\in T_1^\b$ there exists a unique function $\hat y\in AC(\cI_\b)$ such that $\pi_\b\hat
y=\wt y, \; \hat y(c)\in\hat\bH$ and $\{\hat y, f\}\in \cT_1^\b$ for any $f\in\wt f$.

Fix a point $c\in\cI$ and for any interval $\cI_\b(=[\b,b\rangle)\ni c$ let $$\begin{gathered}
\cN^\b=\ker \cT_1^\b\cap \ker \pi_\b=\{y\in AC(\cI_\b): \, \{y,0\}\in\cT_1^\b
\;\;\text{and}\;\; \D(t)y(t)=0 \;\;\text{a.e. on} \;\;
\cI_\b\},\\
\bH^\b = \{y(c):y\in \cN^\b\}.\end{gathered}$$ Let us prove the following assertion:

\(a) if $\cI_{\b_1}\subset \cI_{\b_2}(\iff \b_2 \leq \b_1)$ and $y_2\in \cN^{\b_2}$, then $y_1:=y_2\up \cI_{\b_1}\in\cN^{\b_1} $.

Indeed, the inclusion $y\in\cN^\b $ is equivalent to the relations $$\begin{gathered}
J y'(t)-B(t)y(t)=0 \;\;\text{and}\;\; \D(t)y(t)=0\;\; \text{a.e on}\;\; \cI_\b,
\label{4.40}\\
\lim\limits_{t\uparrow b}(J y(t), z(t))=0, \quad \{z,g\}\in \cT_{max}^\b.
\label{4.41}\end{gathered}$$ Since $y_1$ is a restriction of $y_2$ and holds for $y_2$ on $\cI^{\b_2}$, it follows that is valid for $y_1$ on $\cI^{\b_1}$. Next, assume that $\{z_1, g_1\}\in \cT_{max}^{\b_1} $ and let $z(t)$ be the solution of the equation $$\begin{gathered}
J z'(t)-B(t)z(t)=0, \quad t\in [\b_2,\b_1]\end{gathered}$$ such that $z(\b_1)=z_1(\b_1)$. Then the pair $\{z_2,g_2\}$ with $$\begin{gathered}
z_2(t)=\begin{cases} z_1(t), \; t\in \cI_{\b_1}\cr z(t), \; t\in [\b_2,\b_1)
\end{cases}, \qquad g_2(t)=\begin{cases} g_1(t), \; t\in \cI_{\b_1}\cr 0,\;
t\in [\b_2,\b_1)
\end{cases}\end{gathered}$$ belongs to $\cT_{max}^{\b_2}$ and, consequently, $\lim\limits_{t\uparrow b}(J
y_2(t), z_2(t))=0$. At the same time $z_1=z_2\up \cI_{\b_1}$, so that $\lim\limits_{t\uparrow b}(J y_1(t), z_1(t))=0$. Hence holds for $y_1$, which completes the proof of the assertion (a).

It follows from (a) that $\cI_{\b_1}\subset \cI_{\b_2}$ yields $\bH^{\b_2}
\subset \bH^{\b_1}$. Since $\dim \bH<\infty$, this implies that there exists a subinterval $\cI_{\b_0}=[\b_0, b\rangle$ such that $\bH^\b=\bH^{\b_0}$ for all $\cI_\b\supset\cI_{\b_0}$. Next by using the same arguments as in the proof of Lemma \[lem4.4\] one shows that the statement of the lemma holds for the constructed above interval $\cI_{\b_0}$ and the subspace $\hat \bH=
(\bH^{\b_0})^\perp$.

\[lem4.10\] Let the canonical system be given on an interval $\cI=\langle a,b
\rangle$. Assume also that $\cT_1, \;\cT_3$ and $T_1,\;T_3$ are linear relations in $\lI$ and $\LI$ respectively given by and $$\begin{gathered}
\label{4.42}
\cT_3=\{\{y,f\}\in\tma: \, [y,z]_a=0 \;\;\text{for every}\;\; z\in\dom\tma\},
\qquad T_3=\wt\pi\cT_3.\end{gathered}$$ Then $$\begin{gathered}
\label{4.43}
T_3^*=T_1.\end{gathered}$$

We give only the sketch of the proof, because it is similar to that of Proposition \[pr4.5\] and Lemma \[lem4.6\].

The inclusion $T_1\subset T_3^*$ follows from the Lagrange’s identity . To prove the inverse inclusion assume that $\{\wt y, \wt f\}\in
T_3^*$. For every interval $\cI_\b=[\b,b\rangle$ construct the restrictions $\wt y_\b,\; \wt f_\b \in L_\D^2 (\cI_\b)$ onto $\cI_\b$ in the same way as in the proof of Lemma \[lem4.6\]. Moreover, let $\cT_1^\b$ and $T_1^\b$ be the relations and let $\cT_2^\b=\{\{y,f\}\in\cT_{max}^\b: \,
y(\b)=0\}, \; T_2^\b=\wt\pi_\b \cT_2^\b$. Then by using the Lagrange’s identity one proves the inclusion $\{\wt y_\b,\; \wt f_\b\} \in (T_2^\b)^*$. At the same time by Corollary \[cor4.8\] $(T_2^\b)^*=T_1^\b$, so that $\{\wt y_\b,\; \wt
f_\b\} \in T_1^\b$ for every interval $\cI_\b$. Now by using Lemma \[lem4.9\] one obtains the inclusion $\{\wt y,\; \wt f\} \in T_1$. Hence $T_3^*\subset T_1$, which yields .

Now we are ready to prove the main theorem of the subsection.

\[th4.11\] Let $\Tma$ and $\Tmi$ be maximal and minimal relations and induced by the canonical system on the interval $\cI=\langle a, b\rangle$ and let $T_0$ be the relation . Then $\Tmi$ is a closed symmetric linear relation in $\LI$ and $$\begin{gathered}
\label{4.44}
\ov T_0=\Tmi, \qquad \Tmi^*=\Tma.\end{gathered}$$

If in addition the endpoint $a$ (resp. $b$) is regular and $T_a$ (resp. $T_b$) is the relation (resp. ), than $\Tmi=T_a$ (resp. $\Tmi=T_b$).

If the system is regular, then $\Tmi=T_0$ and every $\l\in\bC$ is a regular type point of $\Tmi$, that is $\hat\rho (\Tmi)=\bC$.

It follows from the Lagrange’s identity that $\Tma\subset\Tmi^*$ and $\Tmi\subset\Tma^*$. This and the obvious inclusion $\Tmi\subset\Tma$ show that $\Tmi$ is symmetric.

Next assume that $T_1$ and $T_3$ are the linear relations and . Since $T_3\subset \Tma$, it follows that $\Tma^*\subset T_3^*$ and by $\Tma^*\subset T_1$. Now the arguments similar to that in the proof of Proposition \[pr4.7\] give the equality $\Tma^*=\Tmi$, which together with leads to . Moreover, combining with and we arrive at the required statement for systems with the regular endpoint $a$ or $b$.

Assume now that the system is regular and show that in this case $$\begin{gathered}
\label{4.45}
\ker (T_0-\l)=\{0\}, \qquad \ov{\ran (T_0-\l)}=\ran (T_0-\l), \qquad \l\in\bC.\end{gathered}$$ If $\wt y\in \ker (T_0-\l)$, then $\{\wt y, \l\wt y\}\in T_0$ and, consequently, there is $y\in\AC$ such that $\pi y=\wt y, \; y(a)=y(b)=0$ and $y$ is a solution of . Hence $y=0$, which gives the first equality in . Moreover, formula (with $\bK=\{0\}$) implies the second equality in .

Since $T_0$ is symmetric, it follows from that $T_0$ is closed. Therefore by $\Tmi=T_0$ and yields the equality $\hat\rho (\Tmi)=\bC$.

Let $\cN$ be the null manifold of the canonical system . Then $\{y,0\}\in\tma$ for every $y\in\cN$ and the Lagrange’s identity gives $$\begin{gathered}
\label{4.45.1}
[y,z]_a=[y,z]_b, \qquad y\in\cN, \;z\in \dom \tma.\end{gathered}$$ This enables us to introduce the subspace $\cN'\subset\cN$ via $$\begin{gathered}
\cN'=\{y\in\cN:\,[y,z]_a =0, \;\; %\;\;\text{for every}\;\;
z\in\dom\tma\}= \{y\in\cN:\, [y,z]_b=0,\;\;%\;\;\text{for every}\;\;
 z\in\dom\tma\}.\label{4.45.2}\end{gathered}$$ Next, the relations $\{y,f\}\in\tma$ and $\wt\pi \{y,f\}=0$ mean that $y\in\AC,
\; f\in\lI$ and $$\begin{gathered}
J y'(t)-B(t)y(t)=\D(t)f(t), \quad \D(t)y(t)=0, \quad \D(t)f(t)=0
\;\;\;\;\text{a.e. on  }\;\; \cI.\end{gathered}$$ Therefore $$\begin{gathered}
\ker (\wt\pi\up\tma)=\{\{y,f\}\in\lI\times \lI:\, y\in \cN \:\:\text{and}
 \:\;\D(t)f(t)=0\;\;\text{a.e. on}\;\;\cI\},\label{4.45.6}\\
\ker (\wt\pi\up\tmi)=\{\{y,f\}\in\lI\times \lI:\, y\in \cN' \:\:\text{and}
 \:\;\D(t)f(t)=0\;\;\text{a.e. on}\;\;\cI\}.\label{4.45.7}\end{gathered}$$

\[pr4.11a\] Let $a$ be a regular endpoint of the system , let $\cT_a$ be the linear relation and let $\hat\cN'=\{\{y,0\}:\,y\in\cN'\}$. Then $$\begin{gathered}
\label{4.45.4}
\tmi=\cT_a\dotplus \hat\cN',\end{gathered}$$ which implies that the equality $\tmi=\cT_a$ holds if and only if $\cN'=\{0\}$.

Since $\cT_a\subset\tmi$ and by Theorem \[th4.11\] $\wt\pi \tmi=\wt\pi\cT_a
(=\Tmi)$, it follows that $$\begin{gathered}
\label{4.45.5}
\tmi=\cT_a+\ker (\wt\pi\up\tmi).\end{gathered}$$ Clearly, the inclusion $\{0,f\}\in\cT_a$ holds for any $f\in\lI$ with $\D(t)f(t)=0\;\;\text{a.e. on}\;\;\cI$. Combining this assertion with and one obtains $\tmi=\cT_a+\hat\cN'$. Moreover, for each $y\in\cN\cap\dom \cT_a$ one has $y(0)=0$ , so that $y=0$. Hence $\cT_a\cap\hat\cN'=\{0\}$, which gives the direct decomposition .

\[ex4.11b\] Consider the canonical system with $\bH=\bC^2$ and operator coefficients $J, \; B(t)$ and $\D(t)$ given in the standard basis of $\bC^2$ by the matrices $$\begin{gathered}
J=\begin{pmatrix} 0 & -1 \cr 1& 0 \end{pmatrix}, \quad B(t)=0, \quad
\D(t)=\begin{pmatrix} 1 & 0 \cr 0 & 0 \end{pmatrix}, \quad t\in [0,\infty).\end{gathered}$$ One immediately checks that for this system $\cN_\l=\cN=\{y(t)\equiv \{0,
C\}:\, C\in\bC\}$ and each function $z\in\dom \tma$ is of the form $z(t)=\{0,
\;z_2(t) \}(\in\bC^2)$. Hence $(J y(t), z(t))\equiv 0\;\;(y\in\cN, \;
z\in\dom\tma)$, so that $\cN'=\cN\neq \{0\}$. This example shows that there exist canonical systems with the regular endpoint $a$ such that $\tmi\neq
\cT_a$.

Deficiency indices and Neumann formulas
---------------------------------------

Let $\tma$ be the maximal relation in $\lI$ induced by the canonical system and let $\cN_\l$ be the subspace . It follows from that $$\begin{gathered}
\label{4.46}
\cN_\l=\ker (\tma-\l)=\{y\in\lI:\; \{y,\l y\}\in\tma\}, \quad\l\in\bC.\end{gathered}$$ Assume also that $\hat\cN_\l$ is a subspace in $\tma$ given by $\hat\cN_\l=\{\{y,\l y\}: y\in\cN_\l\}, \;\l\in\bC$.

\[def4.12\] $\,$[@KogRof75] The numbers $N_+=\dim \cN_i$ and $N_-=\dim\cN_{-i} $ are called the formal deficiency indices of the system .

It is clear that $N_\pm\leq n$. Moreover, if the system is regular, then $N_+=N_-=n$.

Next assume that $$\begin{gathered}
\gN_\l:=\gN_\l(\Tmi)=\ker (\Tma-\l), \quad \l\in\bC\end{gathered}$$ is the defect subspace and $$\begin{gathered}
n_\pm:=n_\pm (\Tmi )=\dim \gN_\l, \quad \l\in\bC_\pm\end{gathered}$$ are deficiency indices of the symmetric relation $\Tmi$ in $\LI$. It is easily seen that $\pi\cN_\l=\gN_\l$ and $\ker(\pi\up\cN_\l)=\cN$ for each $\l\in\bC$. This implies the following proposition.

\[pr4.13\]$\,$ [@KogRof75; @LesMal03] Given a canonical system . Then $N_\pm=\dim \cN_\l, \; \l\in\bC_\pm$ (i.e., $\dim \cN_\l$ does not depend on $\l$ in either $\bC_+$ or $\bC_-$) and $$\begin{gathered}
\label{4.49}
N_+=n_+ + k_\cN, \quad N_-=n_- + k_\cN.\end{gathered}$$

As is known (see for instance [@Ben72]), for any closed symmetric relation $A$ in $\gH$ the Neumann formula is valid. In the case of the minimal relation $\Tmi$ in $\LI$ this formula is $$\begin{gathered}
\label{4.50}
\Tma=\Tmi \dotplus \hat\gN_\l(\Tmi) \dotplus \hat\gN_{\ov\l}(\Tmi),
\quad\l\in\bC\setminus\bR.\end{gathered}$$ In the following proposition we show that similar formulas hold for $\tmi$ and $\tma$.

\[pr4.14\] Let $\tmi$ and $\tma$ be minimal and maximal relations in $\lI$ induced by the system . Assume also that $\cN$ is the null manifold , $\cN'\subset \cN$ is the subspace and let $k_{\cN'}=\dim \cN'$. Then: 1) for each $\l\in\bC\setminus\bR$ the following Neumann formulas hold $$\begin{gathered}
\label{4.51}
\tma=\tmi + (\hat\cN_\l \dotplus \hat\cN_{\ov\l}), \qquad \tmi\cap (\hat\cN_\l
\dotplus \hat\cN_{\ov\l})=\cN'\oplus \cN = \{\{y,f\}:\,y\in\cN', \; f\in\cN\}.\end{gathered}$$

2\) the following equality is valid $$\begin{gathered}
\label{4.52}
\dim (\tma/\tmi)=\dim (\dom\tma/\dom\tmi)= N_+ +N_- -k_\cN-k_{\cN'}.\end{gathered}$$

Since $\wt\pi\tma=\Tma, \; \wt\pi\tmi=\Tmi$ and $\wt\pi \hat\cN_\l=\hat\gN_
\l(\Tmi)$, it follows from that $$\begin{gathered}
\label{4.53}
\tma=\tmi + (\hat\cN_\l \dotplus \hat\cN_{\ov\l})+\ker (\wt\pi\up \tma),\quad
\l\in \bC\setminus\bR\end{gathered}$$ (clearly $\hat\cN_\l\cap \hat\cN_{\ov\l}=\{0\}$, so that the sum $\hat\cN_\l
\dotplus \hat\cN_{\ov\l}$ in is direct). Next, the inclusion $y\in\cN$ implies that $y\in \cN_\l\cap\cN_{\ov\l}$. Moreover, $$\begin{gathered}
\{y,0\}=\left\{
 -\tfrac {\ov\l}{\l-\ov\l} y, -\l\tfrac { \ov\l }{\l-\ov\l} y
\right\} + \left\{ \tfrac {\l}{\l-\ov\l} y, \ov\l\tfrac { \l }{\l-\ov\l} y
\right\}.\end{gathered}$$ and consequently $$\begin{gathered}
\label{4.55}
\{y,0\}\in\hat\cN_\l \dotplus \hat\cN_{\ov\l} \;\;\;\;\; (y\in\cN,\;\;
\l\in\CR).\end{gathered}$$ Furthermore for each $f\in\lI$ such that $\D(t)f(t)=0$ a.e on $\cI$ one has $\{0,f\}\in\tmi$. This and give the inclusion $\ker
(\wt\pi\up\tma)\subset \tmi + (\hat\cN_\l \dotplus \hat\cN_{\ov\l})$, which together with yields the first equality in .

Let us prove the second relation in . If $\{y,f\}\in \tmi\cap
(\hat\cN_\l \dotplus \hat\cN_{\ov\l})$, then $\wt\pi \{y,f\}\in \Tmi\cap
(\hat\gN_\l  \dotplus \hat\gN_{\ov\l})$ and hence $\wt\pi \{y,f\}=0$. Therefore by $y\in \cN'$ and in view of $\{y,0\}\in\hat\cN_\l
\dotplus \hat\cN_{\ov\l} $. Moreover, since $\{y,f\}\in\hat\cN_\l \dotplus
\hat\cN_{\ov\l}$ as well, one has $\{0,f\}\in\hat\cN_\l \dotplus
\hat\cN_{\ov\l} $. This implies that there exist $y\in\cN_\l$ and $z\in\cN_{\ov\l}$ such that $y+z=0$ and $\l y+\ov\l z =f$. Hence $f=(\ov\l-\l)z=(\l-\ov\l)y$, so that $f\in \cN_\l \cap\cN_{\ov\l}$ and by $f\in\cN$. Thus $\{y,f\}\in \cN'\oplus \cN$.

Conversely, let $\{y,f\}\in \cN'\oplus \cN$ with $y\in\cN'$ and $f\in\cN$. Then according to $\{y,0\}\in\tmi$ and gives $\{y,0\}\in\hat\cN_\l \dotplus \hat\cN_{\ov\l}$. Therefore the inclusion $\{y,0\}\in \tmi\cap (\hat\cN_\l \dotplus \hat\cN_{\ov\l})$ is valid. Next, $\{0,f\}\in \tmi$ and the representation $$\begin{gathered}
\{0,f\}=\tfrac 1 {\l-\ov\l}(\{f,\l f\}-\{f,\ov\l f\})\end{gathered}$$ together with shows that $\{0,f\}\in \hat\cN_\l \dotplus
\hat\cN_{\ov\l}$. Thus $\{0,f\}\in \tmi\cap ( \hat\cN_\l \dotplus
\hat\cN_{\ov\l})$ and therefore $\{y,f\}\in \tmi\cap ( \hat\cN_\l \dotplus
\hat\cN_{\ov\l})$ as well. This proves the second relation in .

To prove we first note that the equality $r:=\dim (\tma /
\tmi)=N_+ +N_- - k_\cN-k_{\cN'}$ is immediate from . Next assume that $\{\{y_j,f_j\}\}_1^r$ is a basis of $\tma$ modulo $\tmi$. Then the immediate checking shows that $\{y_j\}_1^r$ is the basis of $\dom \tma$ modulo $\dom \tmi $. Therefore $\dim (\dom \tma / \dom \tmi)=r(=\dim (\tma / \tmi))$ which completely proves .

In the following proposition we give a somewhat different form of the Neumann formulas, which hold in the case of the regular endpoint.

\[pr4.15\] Let $\cT_a$ be the linear relation induced by the system with the regular endpoint $a$. Then $$\begin{gathered}
\label{4.57}
\tma=\cT_a + (\hat\cN_\l \dotplus \hat\cN_{\ov\l}), \qquad \cT_a\cap
(\hat\cN_\l \dotplus \hat\cN_{\ov\l})=\{0\}\oplus \cN = \{\{0,f\}:
f\in\cN\},\quad \l\in\CR,\end{gathered}$$ and the following equality holds $$\begin{gathered}
\label{4.58}
\dim (\tma/\cT_a)=\dim (\dom\tma/\dom\cT_a)= N_+ +N_- -k_\cN.\end{gathered}$$

Let $\cN'\subset \cN$ be the subspace . Then by $\hat\cN'\subset \hat\cN_\l \dotplus \hat\cN_{\ov\l}$ and the first equality in together with gives the first equality in .

Next assume that $\{y,f\}\in \cT_a\cap (\hat\cN_\l \dotplus \hat\cN_{\ov\l})$. Since $\cT_a\subset\tmi$, it follows from that $y\in\cN'$ and $f\in\cN$. Moreover, by $y(a)=0$ and therefore $y=0$. Hence $\{y,f\}\in \{0\}\oplus \cN$. Conversely, in view of the second equality in each pair $\{0,f\}$ with $f\in\cN$ belongs to $\hat \cN_\l
\dotplus \hat\cN_{\ov\l}$ and obviously $\{0,f\}\in\cT_a$. Hence $\{0,f\}\in\cT_a\cap (\hat \cN_\l \dotplus \hat\cN_{\ov\l})$, which yields the second equality in . Finally, one proves formula in the same way as .

\[pr4.16\] Assume that the canonical system has the regular endpoint $a$. Moreover, let $\cT_1$ be the linear relation and let $\bH_1=\{y(a):\, y\in\dom\cT_1\}$. Then $\tmi\subset \cT_1\subset\tma$ and $$\begin{gathered}
 \label{4.60}
\bH_1=(J\bH_a)^\perp,\qquad \dim (\dom \cT_1 / \dom\cT_a)=n-k_\cN,\end{gathered}$$ where $\bH_a$ and $k_\cN$ are defined by and respectively.

It follows from that $(Jy(a), z(a))=0$ for any $y\in\dom\cT_1$ and $z\in\cN$. Therefore $\bH_1\subset (J\bH_a)^\perp$ and to prove the first equality in it remains to show that $(J\bH_a)^\perp\subset \bH_1$.

First assume that the system is regular and let $$\begin{gathered}
\label{4.61a}
\bH_1'=\{y(a):\, y\in\dom\tma \;\;\text{and}\;\; y(b)=0\}.\end{gathered}$$ Moreover, let $\cN_0'$ be the subspace in $\cN_0(=\ker \tma)$ given by $\cN_0'=\{y\in\cN_0:\, y(a)\in \bH_a^\perp\}$. Then $\cN_0'\cap \cN=\{0\}$ and, therefore, the equality $(y,y)_\D=0\; (y\in\cN_0')$ implies $y=0$. Hence $\cN_0'$ is a finite-dimensional Hilbert space with the inner product $(y,z)_\D$.

Let $h\in (J\bH_a)^\perp$, so that $Jh\in \bH_a^\perp$. Then $\f(z)=-(Jh,z(a)),\;\; z\in\cN_0' $ is an antilinear functional on $\cN_0'$ and hence there exists $f_h\in\cN_0'$ such that $$\begin{gathered}
 \label{4.62}
(f_h,z)_\D=-(J h,z(a)), \quad z\in\cN_0'.\end{gathered}$$ Next assume that $y\in\AC$ is the solution of the equation $Jy'-B(t)y=\D(t)
f_h(t)$ such that $y(b)=0$. Then $\{y,f_h\}\in\tma$ and, consequently, $y(a)\in
\bH_1' $. Therefore $y(a)\in (J\bH_a)^\perp$, which gives the inclusion $J
y(a)\in \bH_a^\perp$. Applying now the Lagrange’s identity to $\{y,f_h\} $ and $\{z,0\}\; (z\in\cN_0')$ and taking into account one obtains $$\begin{gathered}
-(J h,z(a))=(f_h,z)_\D=-(J y(a), z(a)), \quad z\in\cN_0'.\end{gathered}$$ In this equality $Jh\in\bH_a^\perp, \; J y(a)\in\bH_a^\perp$ and $z(a)$ takes on any values from $\bH_a^\perp$, when $z$ run through $\cN_0'$. Therefore $y(a)=h$ and, consequently, $h\in\bH_1'$. Thus $(J\bH_a)^\perp\subset \bH_1'$, which together with the obvious inclusion $\bH_1'\subset\bH_1(\subset
(J\bH_a)^\perp)$ gives $$\begin{gathered}
 \label{4.63}
\bH_1=\bH_1'=(J\bH_a)^\perp.\end{gathered}$$

Assume now that the system is singular. For each finite segment $\cI'=[a,\b]\subset\cI$ denote by $\cN^{\cI'}$ the linear space and let $\bH_a^{\cI'}=\{y(a): y\in\cN^{\cI'}\}$. It is easily seen that $\bH_a=\bigcap_{\cI'\subset\cI}\bH_a^{\cI'}$. Moreover, it was shown in the proof of Lemma \[lem4.4\] that there is a segment $\cI_0'=[a,\b_0]$ such that $\bH_a^{\cI_0'}\subset \bH_a^{\cI'}$ for all $\cI'\subset \cI_0'$ and $\bH_a^{\cI_0'} = \bH_a^{\cI'}$ for all $\cI'\supset \cI_0'$. This implies that $\bH_a^{\cI_0'}= \bH_a$.

Next assume that $\tma^{\cI_0'}$ is the maximal relation in $\cL_\D^2(\cI_0')$ induced by the restriction of the system onto $\cI_0'$. Since this restriction is regular, it follows from and that for any $h\in (J\bH_a)^\perp (= (J\bH_a^{\cI_0'})^\perp)$ there exists $\{y,f\}\in
\tma^{\cI_0'}$ such that $y(0)=h$ and $y(\b_0)=0$. Continuing the functions $y$ and $f$ by $0$ onto $\cI$ we obtain the pair $\{y,f\}\in\cT_1$ with $y(0)=h$. This yields the required inclusion $(J\bH_a)^\perp\subset \bH_1$.

Let us prove the second equality in . It follows from the first equality in that $r_1:=\dim\bH_1=n-k_\cN$. Let $\{y_j\}_1^{r_1}$ be a system of functions $y_j\in \dom \cT_1$ such that $\{y_j(0)\}_1^{r_1}$ is a basis in $\bH_1$. Then the immediate checking shows that this system forms a basis of $\dom\cT_1$ modulo $\dom\cT_a$, which yields the desired equality.

The canonical system is called definite if the corresponding null manifold $\cN=\{0\}$.

The following corollaries are implied by the above results on arbitrary (not necessarily definite) canonical systems.

\[cor4.18\] If the system is definite, then $N_\pm=n_\pm$ and the following Neumann formula holds $$\begin{gathered}
 \label{4.64}
\tma=\tmi\dotplus \hat \cN_\l\dotplus\hat\cN_{\ov\l}, \qquad \l\in\CR.\end{gathered}$$

The desired statements are immediate from Propositions \[pr4.13\] and \[pr4.14\].

\[cor4.19\] Let the canonical system with the regular endpoint $a$ be definite. Then $\cT_a=\tmi$ and for every $h\in\bH$ there exists $\{y,f\}\in\tma$ such that $y(a)=h$. If in addition the system is regular (that is, $\cI=[a,b]$), then for any $h_1, \; h_2\in\bH $ there is $\{y,f\}\in\tma$ such that $y(a)=h_1$ and $y(b)=h_2$.

The first statement follows from Propositions \[pr4.11a\] and \[pr4.16\].

Next assume that the system is definite and regular and let $h_1,h_2\in\bH$. Then by and there is $\{y_1,f_1\}\in \tma$ with $y_1(a)=h_1$ and $y_1(b)=0$. Moreover, by symmetry there is $\{y_2,f_2\}\in
\tma$ with $y_2(a)=0$ and $y_2(b)=h_2$. Clearly, the sum $\{y,f\}=\{y_1,f_1\}
+\{y_2,f_2\}$ has the required properties.

\[rem4.20\] For the definite system Theorem \[th4.11\] and Corollary \[cor4.19\] were proved in [@Orc]; the Neumann formula was obtained in [@LesMal03].

The general (not necessarily definite) canonical system of an arbitrary order $n$ was considered in [@LesMal03], where the minimal relation $\Tmi$ was defined as closure of $T_0$ (see ) and then the equality $\Tmi^*(=T_0^*)=\Tma$ was proved . Note in this connection that our definition , of $\Tmi$ seems to be more natural and convenient for applications; in particular cases of differential operators and definite canonical systems such a representation of $\Tmi$ can be found, e.g., in [@Nai; @BHSW10]. Observe also that our Proposition \[pr4.16\] improves similar result in [@LesMal03 Proposition 2.12].

Boundary relations for canonical systems and boundary conditions
================================================================

Boundary bilinear forms
-----------------------

In this section we suppose that the canonical systems is defined on the interval $\cI=[a, b\rangle $ with the regular endpoint $a$.

As is known the signature operator in is unitary equivalent to $$\begin{gathered}
 \label{5.1}
J=\begin{pmatrix} 0 & 0&-I_H \cr 0& i\d I_{\hat H}&0\cr I_H&
0&0\end{pmatrix}:H\oplus\hat H\oplus H \to H\oplus\hat H\oplus H,\end{gathered}$$ where $\d\in\{-1,1\}$ and $H,\;\hat H$ are finite-dimensional Hilbert spaces. The numbers $\d, \dim H$ and $\dim\hat H$ are unitary invariants of $J$, which are defined by the following relations: if we let $$\begin{gathered}
\label{5.3}
\nu_+=\dim\ker (i J-I) \;\;\; \text{and} \;\;\; \nu_-=\dim\ker (i J+I),\end{gathered}$$ then $$\begin{gathered}
\label{5.3a}
\d=\sign (\nu_- -\nu_+), \quad \dim H=\text{min} \{\nu_+,\nu_-\}, \quad  \dim
\hat H=|\nu_- -\nu_+|.\end{gathered}$$ Using this fact we assume without loss of generality that $$\begin{gathered}
\label{5.4}
\bH=H\oplus\hat H \oplus H\end{gathered}$$ and the signature operator $J$ in is given by .

Next consider the boundary bilinear form $[\cd,\cd]_b$ on $\dom \tma $ defined by . Clearly, this form is skew-Hermitian (that is $[y,z]_b=-\ov{[z.y]}_b$) and its kernel coincides with $\dom\cT_1$, where $\cT_1$ is the linear relation . Moreover, since $\dom \tmi\subset
\dom\cT_1\subset\dom \tma $ and by $\dim (\dom\tma /
\dom\tmi)<\infty$, there exists a (not unique) direct decomposition $$\begin{gathered}
\label{5.7}
\dom\tma=\dom \cT_1\dotplus \cD_{b+}\dotplus \cD_{b-}\end{gathered}$$ such that $$\begin{gathered}
\label{5.9}
\nu_{b+}:=\dim \cD_{b+}<\infty, \quad \nu_{b-}:=\dim \cD_{b-}<\infty\end{gathered}$$ and the following relations are valid $$\begin{gathered}
\label{5.7a}
\im [y,y]_b>0, \;\; 0\neq y\in\cD_{b+}; \quad \im [z,z]_b<0, \;\; 0\neq
z\in\cD_{b-}; \quad [y,z]_b=0, \;\; y\in\cD_{b+}, \; z\in\cD_{b-}.\end{gathered}$$ As is known the numbers $\nu_{b+} $ and $\nu_{b-}$ are called indices if inertia of the form $[\cd,\cd]_b$.These numbers are uniquely defined by the form and do not depend on the choice of the decomposition .

\[lem5.1\] Let $[\cd,\cd]_b$ be the bilinear form with the indices of inertia and let $\d_b:=\sign (\nu_{b+}-\nu_{b-})$. Then: 1) there exists Hilbert spaces $\cH_b$ and $\hat\cH_b$ and a surjective linear map $$\begin{gathered}
\label{5.13}
\G_b=(\G_{0b}:\,  \hat\G_b:\,  \G_{1b})^\top:\dom\tma\to
\cH_b\oplus\hat\cH_b\oplus \cH_b\end{gathered}$$ such that $$\begin{gathered}
\label{5.15}
[y,z]_b=i\d_b (\hat\G_b y, \hat\G_b z)-(\G_{1b}y,\G_{0b}z)+(\G_{0b}y,\G_{1b}z),
\quad y,z \in \dom\tma.\end{gathered}$$ Letting $\bH_b:=\cH_b\oplus\hat\cH_b\oplus \cH_b$ and introducing the signature operator $J_b\in [\bH_b]$ by $$\begin{gathered}
\label{5.16a}
J_b=\begin{pmatrix} 0 & 0&-I_{\cH_b} \cr 0& i\d_b I_{\hat \cH_b}&0\cr
I_{\cH_b}& 0&0\end{pmatrix}:\cH_b\oplus\hat \cH_b\oplus \cH_b \to
\cH_b\oplus\hat \cH_b\oplus \cH_b\end{gathered}$$ one can represent the identity as $$\begin{gathered}
\label{5.16b}
[y,z]_b=(J_b \G_b y, \G_b z)_{\bH_b},\quad y,z \in \dom\tma.\end{gathered}$$

2\) if a surjective linear map $\G_b$ of the form satisfies , then $\ker\G_b=\dom\cT_1$ and $$\begin{gathered}
\label{5.17}
\dim\cH_b=\text{min} \{\nu_{b+}, \nu_{b-}\}, \qquad \dim \hat\cH_b=|\nu_{b+}-
\nu_{b-}|.\end{gathered}$$

1\) Assume for definiteness that $\nu_{b+}\geq \nu_{b-}$, so that $\d_b=1$. It follows from that $\cD_{b+}$ and $\cD_{b-}$ are finite-dimensional Hilbert spaces with the inner products $(y_1,y_2)_+=-i[y_1,
y_2]_b, \; y_1, y_2\in \cD_{b+}$ and $(z_1,z_2)_-=i[z_1,z_2]_b, \; z_1,z_2\in
\cD_{b-}$ respectively. Moreover, by $$\begin{gathered}
\label{5.18}
\dom\tma=\dom \cT_1\dotplus (\hat\cH_b\oplus\cH_b )\dotplus \cD_{b-},\end{gathered}$$ where $\hat\cH_b$ and $\cH_b$ are subspaces in $\cD_{b+}$ such that $\dim\cH_b=\nu_{b-}(=\dim \cD_{b-})$ and $\cD_{b+}=\hat\cH_b\oplus\cH_b$.

Let $V$ be a unitary operator from $\cD_{b-}$ onto $\cH_b$ and let $$\begin{gathered}
\label{5.19}
\hat\G_b= \cP_{\hat\cH_b}, \qquad \G_{0b}=\tfrac1 {\sqrt 2}(\cP_{\cH_b}+V
\cP_{\cD_{b-}}),  \qquad  \G_{1b}=-\tfrac {i} {\sqrt 2}(\cP_{\cH_b}-V
\cP_{\cD_{b-}}),\end{gathered}$$ where $\cP_{\hat\cH_b}, \;\cP_{\cH_b}$ and $\cP_{\cD_{b-}}$ are the skew projections onto the subspaces $\hat\cH_b, \; \cH_b$ and $\cD_{b-}$ corresponding to the decomposition . The immediate checking shows that the map $\G_b$ given by and is surjective and satisfies .

Similarly one proves the statement 1) in the case $\nu_{b+}< \nu_{b-}$.

The statement 2) immediately follows from surjectivity of $\G_b$ and the identity .

\[rem5.2\] One can show that the map $\G_b$ in Lemma \[lem5.1\] can be represented in the more explicit form. Namely, it is not difficult to prove that there exist systems of functions $\{\psi_j\}_1^{\nu_b},\;\{\f_j\}_1^{\hat\nu_b} $ and $\{\t_j\}_1^{\nu_b}$ in $\dom\tma$ with $\nu_b=\text{min} \{\nu_{b+},
\nu_{b-}\}$ and $\hat\nu_b=|\nu_{b+}- \nu_{b-}|$ such that the operators $$\begin{gathered}
\G_{0b}y =\{[y,\psi_j ]_b\}_1^{\nu_b}, \quad   \hat\G_b y=\{[y,\f_j
]_b\}_1^{\hat\nu_b}, \quad \G_{1b}y =\{[y,\t_j ]_b\}_1^{\nu_b}, \quad
y\in\dom\tma\end{gathered}$$ form the surjective linear map $\G_b=(\G_{0b}:\,  \hat\G_b:\,
\G_{1b})^\top:\dom\tma\to \bC^{\nu_b}\oplus\bC^{\hat \nu_b}\oplus \bC^{\nu_b}$ satisfying the identity . This assertion shows that for each $y\in\dom\tma$ the elements $\G_{0b} y, \G_{1b}y$ and $\hat\G_b y$ are, in fact, boundary values of the function $y(\cd)$ at the endpoint $b$.

Decomposing boundary relations
------------------------------

Assume without loss of generality that the Hilbert space $\bH$ and the signature operator $J$ in are defined by and respectively. In this case each function $y(\cd)\in\dom\tma$ admits the representation $$\begin{gathered}
\label{5.21}
y(t)=\{y_0(t),\, \hat y(t),\, y_1(t)\}(\in \bH), \quad t\in\cI,\end{gathered}$$ where $y_0(t), \hat y(t)$ and $y_1(t)$ are components of $y(t)$ corresponding to the decomposition .

Let $\nu_+$ and $\nu_-$ by given by and let $\nu_{b+}$ and $\nu_{b-}$ be indices of inertia . Then according to Lemma \[lem5.1\] there exist Hilbert spaces $\cH_b$ and $\hat\cH_b$ satisfying and the surjective linear map $\G_b=(\G_{0b}:\,  \hat\G_b:\,
\G_{1b})^\top$ such that holds. Without loss of generality assume that $$\begin{gathered}
\label{5.22}
\nu_{b+}-\nu_{b-}\geq \nu_- -\nu_+\end{gathered}$$ and consider the following three alternative cases:

\(i) $\nu_--\nu_+\geq 0$

It follows from and that in this case $$\begin{gathered}
\label{5.23}
\dim H=\nu_+, \quad \dim\hat H=\nu_- -\nu_+, \quad \dim\cH_b=\nu_{b-}, \quad
\dim\hat\cH_b=\nu_{b+}-\nu_{b-}\end{gathered}$$ and the inequality gives $\dim \hat H \leq \dim\hat\cH_b$. Therefore without loss of generality we can assume that $\hat H$ is a subspace in $\hat\cH_b$. Letting $\cH_2=\hat\cH_b\ominus \hat H$, we obtain $\hat\cH_b=\cH_2\oplus \hat H$, so that the operator $\hat \G_b$ in admits the block representation $\hat \G_b=(\G_{2b}:\,
\hat\G_{1b})^\top: \dom\tma\to\cH_2\oplus \hat H$. Put $$\begin{gathered}
\cH_1=H\oplus \hat H\oplus \cH_b,\quad \cH_0=\cH_2\oplus\cH_1= \cH_2\oplus
H\oplus \hat H\oplus \cH_b\end{gathered}$$ and introduce the operators $$\begin{gathered}
\G_0'y=\{ \G_{2b}y,\, y_0(a),\, \tfrac i {\sqrt 2}(\hat
y(a)-\hat\G_{1b}y)\},\,\G_{0b}y\} (\in\cH_2\oplus H\oplus  \hat
H\oplus \cH_b),\label{5.25}\\
\G_1'y=\{y_1(a),\,  \tfrac 1 {\sqrt 2}(\hat y(a)+\hat\G_{1b}y),\, -\G_{1b}y\}
(\in H\oplus \hat H \oplus \cH_b),\quad y\in\dom\tma \label{5.26}.\end{gathered}$$

\(ii) $\nu_- -\nu_+ < 0$ and $\nu_{b+}-\nu_{b-}\geq 0 $, so that $$\begin{gathered}
\label{5.27}
\dim H=\nu_-, \quad \dim\hat H=\nu_+ -\nu_-, \quad \dim\cH_b=\nu_{b-}, \quad
\dim\hat\cH_b=\nu_{b+}-\nu_{b-}.\end{gathered}$$ In this case we put $$\begin{gathered}
 \cH_2=\hat H\oplus \hat\cH_b, \quad \cH_1=H \oplus \cH_b,\quad
\cH_0= \cH_2\oplus \cH_1=(\hat H\oplus \hat\cH_b)\oplus H \oplus \cH_b\nonumber\\
\G_0'y=\{\{\hat y(a),\, \hat\G_b y\},\, y_0(a),\, \G_{0b}y\} (\in (\hat
H\oplus\hat\cH_b)\oplus H\oplus  \cH_b),\label{5.28}\\
\G_1'y=\{y_1(a),  -\G_{1b}y\} (\in H \oplus \cH_b),\quad
y\in\dom\tma\label{5.29}.\end{gathered}$$

\(iii) $\nu_{b+}-\nu_{b-}< 0 $, so that $$\begin{gathered}
\label{5.30}
\dim H=\nu_-, \quad \dim\hat H=\nu_+ -\nu_-, \quad \dim\cH_b=\nu_{b+}, \quad
\dim\hat\cH_b=\nu_{b-}-\nu_{b+}.\end{gathered}$$ In view of one has $\dim \hat \cH_b \leq \dim \hat H$, which enables us to assume by analogy with the case (i) that $\hat\cH_b\subset \hat H $. Letting $\cH_2=\hat H\ominus \cH_b$ one obtains $\hat H=\cH_2\oplus
\hat\cH_b$, which implies the representation $\hat y(t)=\{\hat y_2(t),\, \hat
y_b(t)\}(\in \cH_2\oplus \hat\cH_b)$ of the functions $\hat y(t)$ from .

In the case (iii) we let $$\begin{gathered}
\cH_1=H\oplus \hat\cH_b \oplus \cH_b,\quad \cH_0=\cH_2\oplus\cH_1=\cH_2\oplus
H\oplus \hat\cH_b \oplus \cH_b\nonumber\\
\G_0'y=\{\hat y_2(a),\, y_0(a),\,-\tfrac i {\sqrt 2}(\hat y_b(a)-\hat\G_b
y)\},\,\G_{0b}y\}(\in \cH_2\oplus H\oplus \hat \cH_b \oplus \cH_b)\label{5.31}\\
 \G_1'y=\{y_1(a),  \tfrac 1 {\sqrt
2}(\hat y_b(a)+\hat\G_b y),-\G_{1b}y\} (\in H\oplus \hat\cH_b \oplus
\cH_b),\quad y\in\dom\tma\label{5.32}.\end{gathered}$$

Note that in each of the cases (i)–(iii) $\cH_1$ is a subspace in $\cH_0$, $\cH_2=\cH_0\ominus \cH_1$ and $\G_j'$ is a linear map from $\dom\tma$ to $\cH_j, \; j\in \{0.1\}$. Moreover, formulas , and imply that in all cases (i)–(iii)

$$\begin{gathered}
\label{5.32a}
\dim\cH_0=\nu_+ +\nu_{b+}, \qquad \dim\cH_1=\nu_- +\nu_{b-}.\end{gathered}$$

\[th5.3\] Assume that $a$ is a regular endpoint for the canonical system and the inequality is satisfied. Moreover, let $\cH_j$ be Hilbert spaces and $\G_j':\dom\tma\to \cH_j, \; j\in\{0,1\}$ be linear maps constructed for the alternative cases (i)–(iii) just before the theorem. Then the equality $$\begin{gathered}
\label{5.33}
\G=\lb \lb {\pi y\choose \pi f}, {\G_0' y\choose \G_1'y } \rb
:\{y,f\}\in\tma\rb\end{gathered}$$ defines the boundary relation $\G:(\LI)^2\to \HH$ for $\Tma(=\Tmi^*)$ with $$\begin{gathered}
\label{5.34}
\dim\cH_0=N_+ \quad\emph{and} \quad \dim\cH_1=N_-.\end{gathered}$$

Let us show that the linear relation satisfies the assumptions of Corollary \[cor3.6\] for $A:=\Tmi$.

Assume that $\G'=(\G_0':\, \G_1')^\top: \dom\tma\to \HH$. Then definitions – of $\G_0'$ and $\G_1'$ and the equality $\ker
\G_b=\dom\cT_1$ (see Lemma \[lem5.1\], 2)) imply that $\ker \G'=\cT_a$. Therefore by one has $\ker\G=\wt\pi\,\ker \G'=\wt\pi \cT_a=\Tmi$. Moreover, it follows from that $\dom\G=\wt\pi\tma =\Tma $.

Next, the immediate calculations with taking into account show that in each of the cases (i)–(iii) the operators $\G_0'$ and $\G_1'$ satisfy the relation $$\begin{gathered}
[y,z]_b- (J y(a), z(a))=(\G_1'y, \G_0'z)-(\G_0'y, \G_1'z)+ i (P_2\G_0'y,
P_2\G_0'z), \quad  y,z\in\dom\tma.\end{gathered}$$ This and the Lagrange’s identity give the identity for $\G$.

Now it remains to prove . It follows from and that $$\begin{gathered}
\nu_{b+}+\nu_{b-}=\dim (\dom\tma / \dom \cT_1)=\dim (\dom\tma / \dom
\cT_a)-\dim (\dom\cT_1 / \dom \cT_a).\end{gathered}$$ Combining this equality with and the second equality in one obtains $$\begin{gathered}
\nu_{b+}+\nu_{b-}=(N_+ +N_- -k_\cN) -(n-k_\cN)=N_++N_--n.\end{gathered}$$ This and give $$\begin{gathered}
\label{5.35}
\dim (\HH)=(\nu_++\nu_-)+(\nu_{b+}+\nu_{b-})=n+(N_++N_--n)=N_++N_-.\end{gathered}$$

Next, in view of one has $\mul\G=\{\G'y:\, \{y,f\}\in \ker
(\wt\pi\up \tma)\;\;\text{for some}\;\;f\in\lI\}$ and yields $$\begin{gathered}
\label{5.35.1}
\mul\G=\G'\cN=\{\{\G_0'y,\G_1' y\}:\, y\in\cN\}.\end{gathered}$$ Since obviously $\ker (\G'\up\cN)=\{0\}$, it follows from that $$\begin{gathered}
\label{5.36}
n_\G(=\dim (\mul\G))=\dim\cN=k_N.\end{gathered}$$ This and imply that $$\begin{gathered}
\label{5.37}
n_+ +n_- + 2n_\G=(N_+ -k_\cN)+(N_- -k_\cN)+2k_\cN=N_+ +N_-.\end{gathered}$$ Combining now and we arrive at the required equality $$\begin{gathered}
\dim (\HH)=n_+ +n_- + 2n_\G.\end{gathered}$$ Thus according to Corollary \[cor3.6\] formula defines the boundary relation $\G$ for $\Tma$. Moreover, combining with and we obtain the equalities .

\[def5.4a\] The boundary relation $\G:(\LI)^2\to \HH$ constructed in Theorem \[th5.3\] will be called a decomposing boundary relation for $\Tma$.

\[pr5.4\] The formal deficiency indices of the canonical system with the regular endpoint $a$ can be calculated via $$\begin{gathered}
\label{5.38}
N_+=\nu_+ +\nu_{b+}, \qquad N_-=\nu_- +\nu_{b-},\end{gathered}$$ where $\nu_\pm$ are the numbers and $\nu_{b+}, \; \nu_{b-}$ are indices of inertia of the bilinear form $[\cd,\cd]_b$. It follows from that in the case of the regular endpoint $a$ the following inequalities hold $$\begin{gathered}
\label{5.39}
\nu_+\leq N_+\leq n, \qquad \nu_-\leq N_-\leq n.\end{gathered}$$

If $\nu_\pm$ and $\nu_{b\pm}$ satisfy , then the equalities follow from and . In the opposite case $\nu_{b+}-\nu_{b-}< \nu_- -\nu_+$ the equalities can be obtained by passage to the system $$\begin{gathered}
-J y'(t)+B(t)y(t)=\D(t)f(t).\end{gathered}$$

In the case $N_+=N_-$ the construction of the decomposing boundary relation for $\Tma$ can be rather simplified. Namely, the following corollary is valid.

\[cor5.5\] Assume that $a$ is a regular endpoint for the canonical system . Then:

1\) this system has equal deficiency indices $N_+=N_-$ if and only if $$\begin{gathered}
\label{5.40}
\nu_{b+}-\nu_{b-}=\nu_- -\nu_+\end{gathered}$$ 2) if $N_+=N_-$, then there exist a Hilbert space $\cH_b$ with $\dim\cH_b=\text{min}\{\nu_{b+},\nu_{b-}\}$ and a surjective linear map $$\begin{gathered}
\label{5.41}
\G_b=(\G_{0b}:\,  \hat\G_b:\,  \G_{1b})^\top:\dom\tma\to \cH_b\oplus\hat
H\oplus \cH_b\end{gathered}$$ such that the identity holds with $\d_b=\d(=\sign (\nu_- -
 \nu_+))$. Moreover, for each such a map $\G_b$ the equality $$\begin{gathered}
\label{5.42}
\G=\lb \lb {\pi y\choose \pi f}, {\{ y_0(a),\, \tfrac i {\sqrt 2}\d (\hat
y(a)-\hat\G_b y),\,\G_{0b}y \}\choose \{ y_1(a),\, \tfrac 1 {\sqrt 2} (\hat
y(a)+\hat\G_b y),\,-\G_{1b}y \} } \rb :\{y,f\}\in\tma\rb\end{gathered}$$ defines the decomposing boundary relation $\G:(\LI)^2 \to (H\oplus\hat H\oplus
\cH_b )^2$ for $\Tma$.

In the case of the regular system one can put $\cH_b= H$ and $$\begin{gathered}
\label{5.43}
\G=\lb \lb {\pi y\choose \pi f}, {\{ y_0(a),\, \tfrac i {\sqrt 2}\d (\hat
y(a)-\hat y(b)),\, y_0(b) \}\choose \{ y_1(a),\, \tfrac 1 {\sqrt 2} (\hat
y(a)+\hat y(b)),\,-y_1(b) \} } \rb :\{y,f\}\in\tma\rb.\end{gathered}$$

The statement 1) follows from .

2\) Combining with and one obtains $\dim
\hat\cH_b=\dim \hat  H$. Therefore one can put in $\hat\cH_b= \hat
H$, in which case the map $\G_b$ takes the form and the equality for $\G$ can be represented as . In the case of the regular system one can put $\cH_b=H$ and $\G_b y=\{y_0(b),\, \hat y(b), \,
y_1(b)\}(\in\bH)$, so that the equality takes the form .

\[cor5.6\] Assume that the canonical system with the regular endpoint $a$ has minimal formal deficiency indices $N_+=\nu_+$ and $N_-=\nu_-$. If $N_+ \geq
N_-$, then the equality $$\begin{gathered}
\label{5.44}
\G=\lb \lb {\pi y\choose \pi f}, {\{ \hat y(a), \,y_0(a)\}\choose  y_1(a) } \rb
:\{y,f\}\in\tma\rb\end{gathered}$$ defines the decomposing boundary relation $\G:(\LI)^2\to (\hat H\oplus H)\oplus
H $ for $\Tma$.

Since $\nu_{b+}=\nu_{b-}=0$, it follows from that $\cH_b=\hat\cH_b=\{0\}$. Combining this equalities with , and we obtain the representation for $\G$.

Boundary conditions for definite systems
----------------------------------------

As is known (see for instance [@LesMal03])the maximal operator $\Tma$ induced by the definite system possesses the following property: for each $\{\wt y, \wt f\}\in \Tma$ there exists a unique function $y\in\AC$ such that $y\in\wt y$ and $\{y,f\}\in\tma$ for each $f\in\wt f$. Bellow, without any additional comments, we associate such a function $y\in\AC$ with each pair $\{\wt y, \wt f\}\in \Tma$.

\[th5.9\] Let under the conditions of Theorem \[th5.3\] the canonical system be definite. Then:

1\) The operators $\G_j:\Tma\to\cH_j, \; j\in \{0,1\}$ given by $$\begin{gathered}
\label{5.52}
\G_0\{\wt y, \wt f\}=\G_0'y, \qquad \G_1\{\wt y, \wt f\}=\G_1'y, \qquad \{\wt
y, \wt f\}\in\Tma\end{gathered}$$ form the boundary triplet $\Pi=\bta$ for $\Tma$.

In the case of minimal deficiency indices $n_+=\nu_+$ and $n_-=\nu_-$ one has $\cH_0=\hat H\oplus H, \; \cH_1=H$ and the equality takes the form $$\begin{gathered}
\label{5.52a}
\G_0\{\wt y, \wt f\}= \{\hat y(a), \, y_0(a)\}(\in\hat H\oplus H), \quad
\G_1\{\wt y, \wt f\}= y_1(a)(\in H), \quad \{\wt y, \wt f\} \in \Tma.\end{gathered}$$

2\) If $n_+=n_-$, then the statement 2) of Corollary \[cor5.5\] holds and the decomposing boundary relation turns into the boundary triplet $\Pi=\{\cH,\G_0, \G_1\}$ for $\Tma$ with $\cH=H\oplus\hat H \oplus \cH_b$ and the operators $\G_j: \Tma\to \cH$ given by $$\begin{gathered}
\G_0 \{\wt y, \wt f\} =\{ y_0(a),\, \tfrac i {\sqrt 2}\d (\hat y(a)-\hat\G_b
y),\,\G_{0b}y \} (\in H\oplus\hat H \oplus \cH_b),\qquad\qquad\qquad\quad\label{5.53}\\
\G_1 \{\wt y, \wt f\} = \{ y_1(a),\, \tfrac 1 {\sqrt 2} (\hat y(a)+\hat\G_b
y),\,-\G_{1b}y \}(\in H\oplus\hat H \oplus \cH_b), \quad \{\wt y, \wt f\} \in
\Tma, \label{5.54}\end{gathered}$$ with $\d=\sign (\nu_--\nu_+)$. In the case of the regular system one can put $\cH= H\oplus\hat H \oplus H$ and $$\begin{gathered}
\G_0 \{\wt y, \wt f\}=\{ y_0(a),\, \tfrac i {\sqrt 2}\d (\hat y(a)-\hat
y(b)),\, y_0(b) \}(\in H\oplus\hat H \oplus H),\qquad\qquad\qquad \label{5.55} \\
\G_1 \{\wt y, \wt f\}= \{ y_1(a),\, \tfrac 1 {\sqrt 2}(\hat y(a)+\hat
y(b)),\,-y_1(b) \}(\in H\oplus\hat H \oplus H), \quad \{\wt y, \wt f\} \in
\Tma.\label{5.56}.\end{gathered}$$

1\) Let $\G$ be the decomposing boundary relation for $\Tma$. Then by $\mul \G=\{0\}$ and Corollary \[cor3.5\], 2) implies that the operators form the boundary triplet $\Pi=\bta$ for $\Tma$. Moreover, in the case $n_\pm=\nu_\pm$ the equality gives .

The statement 2) of the theorem follows from Corollary \[cor5.5\], 2).

In the sequel the boundary triplet $\Pi=\bta$ defined in Theorem \[th5.9\], 1) will be called the decomposing boundary triplet for $\Tma$. In the case of equal deficiency indices $n_+=n_-$ such a triplet takes the form $\Pi=\{\cH,\G_0, \G_1\}$, where $\cH=H\oplus\hat H \oplus \cH_b$ and $\G_0, \;
\G_1$ are defined by and .

\[pr5.10\] Let the minimal relation $\Tmi$ induced by the definite system with the regular endpoint $a$ has equal deficiency indices $n_+=n_-$ and let $\Pi=\bt$ be the decomposing boundary triplet , for $\Tma$. Then for each operator pair (linear relation) $\t=\{(C_0,C_1);\cK\}$ given by the block representations $$\begin{gathered}
\label{5.57}
C_0=(C_{0a}:\hat C_a:C_{0b}):H\oplus\hat H \oplus \cH_b\to\cK, \qquad
C_1=(C_{1a}:\hat C_b:C_{1b}):H\oplus\hat H \oplus \cH_b\to\cK\end{gathered}$$ the equality (the boundary conditions) $$\begin{gathered}
\label{5.58}
\wt A=\{\{\wt y, \wt f\}\in\Tma:\,C_{0a}y_0(a)+\hat C_a \hat y(a)+
C_{1a}y_1(a)+C_{0b}\G_{0b}y+\hat C_b \hat \G_b y + C_{1b}\G_{1b}y=0 \}\end{gathered}$$ defines a proper extension $\wt A$ of $\Tmi$ and, conversely, for each such an extension there is a unique admissible operator pair (linear relation) $\t=\{(C_0,C_1);\cK\}$ given by and such that holds. Moreover, the extension is maximal dissipative, maximal accumulative or self-adjoint if and only if the operator pair (linear relation) $\wt\t=\{(\wt C_0,\wt C_1);\cK\}$ with $$\begin{gathered}
\label{5.59}
\wt C_0=(C_{0a}: -\tfrac {i\d}{\sqrt 2} (\hat C_a-\hat C_b):C_{0b}), \qquad \wt
C_1=(C_{1a}: \tfrac 1 {\sqrt 2} (\hat C_a+\hat C_b): -C_{1b})\end{gathered}$$ is maximal dissipative, maximal accumulative or self-adjoint respectively.

It follows from and that the boundary conditions can be written as $$\begin{gathered}
\label{5.59a}
\wt A=\{\{\wt y, \wt f\}\in\Tma:\, \wt C_0 \G_0 \{\wt y, \wt f\}+\wt C_1 \G_1
\{\wt y, \wt f\}=0 \}.\end{gathered}$$ This and Proposition \[pr2.5\] yield the desired statements.

In the following corollary we give a somewhat different description of proper extensions $\wt A\in Ext_{\Tma} $.

\[cor5.10a\] Assume that $a$ is a regular endpoint for the definite canonical system and $n_+=n_-=:m$. Let $\bH_b$ be a Hilbert space, let $J_b\in
[\bH_b] $ be a signature operator and let $\G_b:\dom\tma \to \bH_b$ be a surjective linear map such that holds (according to Lemma \[lem5.1\] such $\bH_b,\; J_b$ and $\G_b$ exist and $\dim \bH_b=\nu_{b+}+
\nu_{b+}$). Moreover, let $\cK$ be a Hilbert space with $\dim\cK=m$, let $C_a\in [\bH,\cK]$ and $C_b\in [\bH_b,\cK] $ be operators such that $\ran
(C_a:\,C_b)=\cK$ and let $\wt A\in Ext _{\Tmi}$ be an extension given by $$\begin{gathered}
\label{5.59b}
\wt A=\{\{\wt y, \wt f\}\in\Tma: \, C_a y(a)+C_b \G_b y=0\}.\end{gathered}$$ Then $\wt A$ is maximal dissipative, maximal accumulative or self-adjoint if and only if $$\begin{gathered}
\label{5.59c}
i(C_a J C_a^*-C_b J_b C_b^*)\leq 0, \quad i(C_a J C_a^*-C_b J_b C_b^*)\geq 0
\;\;\;\text{or} \;\;\; C_a J C_a^*=C_b J_b C_b^*\end{gathered}$$ respectively.

Assume without loss of generality that $\bH_b=\cH_b\oplus \hat H \oplus\cH_b $ and the operators $J_b$ and $\G_b$ are of the form and respectively (with $\hat H$ in place of $\hat\cH_b$). Then according to Theorem \[th5.9\] the Hilbert space $\cH=H\oplus \hat H\oplus
\cH_b$ and the operators , form the (decomposing) boundary triplet $\Pi=\bt$ for $\Tma$. Next assume that $$\begin{gathered}
C_a=(C_{0a}: \hat C_a :C_{1a}):H\oplus\hat H \oplus H\to\cK, \quad
C_b=(C_{0b}:\hat C_b: C_{1b}):\cH_b\oplus\hat H \oplus \cH_b\to\cK\end{gathered}$$ are the block representations of $C_a$ and $C_b$ and let $\wt C_0$ and $\wt
C_1$ be given by . Then can be written as and according to Proposition \[pr5.10\] $\wt A$ is maximal dissipative, maximal accumulative or self-adjoint if and only if the operator pair $\t==\{(\wt C_0,\wt C_1);\cK\}$ belongs to the same class. The immediate calculations show that $$\begin{gathered}
2\im (\wt C_1 \wt C_0^*)=i(C_b J_b C_b^*-C_a J C_a^*).\end{gathered}$$ Moreover, since $\ran (C_a:\,C_b)=\cK$, it follows that the operator pair $(\wt
C_0:\,\wt C_1 )$ is admissible. Applying now Proposition \[pr2.2\], 2) we arrive at the required statement.

\[def5.11\] The boundary conditions are said to be separated if there exists a decomposition $\cK=\cK_a\oplus\cK_b$ such that the operators are $$\begin{gathered}
C_0=\begin{pmatrix} N_{0a} & \hat N_a & 0 \cr 0 & 0 & N_{0b}
\end{pmatrix}:H\oplus\hat H \oplus \cH_b\to\cK_a\oplus\cK_b\label{5.60}\\
C_1=\begin{pmatrix} N_{1a} & 0 & 0 \cr 0 & \hat N_b & N_{1b}
\end{pmatrix}:H\oplus\hat H \oplus \cH_b\to\cK_a\oplus\cK_b\label{5.61}\end{gathered}$$ and, consequently, the equality takes the form $$\begin{gathered}
\label{5.62}
\wt A=\{\{\wt y, \wt f\}\in\Tma:\,N_{0a}y_0(a)+\hat N_a \hat y(a)+
N_{1a}y_1(a)=0, \;\; N_{0b}\G_{0b}y+\hat N_b \hat \G_b y + N_{1b}\G_{1b}y=0 \}.\end{gathered}$$

The separated boundary conditions will by called maximal dissipative, maximal accumulative or self-adjoint if they define the extension $\wt A $ of the corresponding class.

With the separated boundary conditions we associate the operators $$\begin{gathered}
S_a=\im (N_{1a}N_{0a}^*)+\tfrac 1 2 \hat N_a \hat N_a^*, \qquad  S_b=\im
(N_{1b}N_{0b}^*)+\tfrac 1 2 \hat N_b \hat N_b^*,\label{5.63}\\
\wt N_a=(N_{0a}-i N_{1a}\, :\, -i\sqrt{2} \hat N_a):H\oplus\hat H\to \cK_a,
\quad \wt N_b=(i\sqrt{2} \hat N_b\, :\,N_{0b}-i N_{1b} ): \hat H\oplus
\cH_b\to\cK_b.\label{5.64}\end{gathered}$$

\[th5.12\] Let for simplicity $\nu_-\geq \nu_+$ and let the assumptions of Proposition \[pr5.10\] be satisfied. Then: 1) the separated boundary conditions defined by – are maximal dissipative if and only if $$\begin{gathered}
\label{5.65}
S_a\geq 0, \quad S_b\leq 0 \;\;\; \emph {and} \;\;\; 0\in \rho (\wt N_a)\cap
\rho (N_{0b}+i N_{1b}),\end{gathered}$$ in which case the following equalities hold $$\begin{gathered}
\label{5.65a}
\dim\cK_a=\nu_-, \qquad \dim\cK_b=\nu_{b-}.\end{gathered}$$ The same boundary conditions are maximal accumulative if and only if $$\begin{gathered}
\label{5.66}
S_a\leq 0, \quad S_b\geq 0 \;\;\; \emph {and} \;\;\; 0\in \rho (N_{0a}+i
N_{1a})\cap\rho(\wt N_b),\end{gathered}$$ in which case $$\begin{gathered}
\label{5.66a}
\dim\cK_a=\nu_+, \qquad \dim\cK_b=\nu_{b+}\end{gathered}$$ (here $\cK_a$ and $\cK_b$ are Hilbert spaces from and ).

2\) self-adjoint separated boundary conditions exist if and only if $\nu_-=\nu_+$ or, equivalently, if and only if $\bH=H\oplus H $ and the operator $J$ in is $$\begin{gathered}
J=\begin{pmatrix} 0 & -I_H \cr I_H & 0 \end{pmatrix}:H\oplus H\to H\oplus H.\end{gathered}$$ If this condition is satisfied, then:

\(i) the decomposing boundary triplet , takes the form $\Pi=\bt$, where $\cH=H\oplus\cH_b$ and the operators $\G_j, \; j\in\{0,1\}$ are given by $$\begin{gathered}
\label{5.67}
\G_0 \{\wt y,\wt f\}=\{y_0(a), \G_{0b}y\}(\in H\oplus\cH_b), \quad \G_1 \{\wt
y,\wt f\}=\{y_1(a), -\G_{1b}y\}(\in H\oplus\cH_b), \quad\{\wt y,\wt f\}\in\Tma\end{gathered}$$

\(ii) the general form of self-adjoint separated boundary conditions is $$\begin{gathered}
\label{5.68}
\wt A=\{\{\wt y, \wt f\}\in\Tma:\,N_{0a}y_0(a)+ N_{1a}y_1(a)=0, \;\;
N_{0b}\G_{0b}y+ N_{1b}\G_{1b}y=0 \},\end{gathered}$$ where the operators $N_{ja}\in [H,\cK_a]$ and $N_{jb}\in [\cH_b,\cK_b], \;
j\in\{0,1\}$ are components of self-adjoint operator pairs $\t_a=\{(N_{0a},
N_{1a} ); \cK_a\}$ and $\t_b=\{(N_{0b}, N_{1b} ); \cK_b\}$.

1\) Let $\wt C_0$ and $\wt C_1$ be the operators corresponding to the separated boundary conditions . Then in view of and one has $$\begin{gathered}
\wt C_0=\begin{pmatrix} N_{0a} & -\frac i {\sqrt 2} \hat N_a & 0 \cr 0 &\frac i
{\sqrt 2} \hat N_b  & N_{0b}
\end{pmatrix}:H\oplus\hat H \oplus \cH_b\to\cK_a\oplus\cK_b\label{5.69}\\
\wt C_1=\begin{pmatrix} N_{1a} & \frac 1 {\sqrt 2} \hat N_a & 0 \cr 0 & \frac 1
{\sqrt 2}\hat N_b & -N_{1b}
\end{pmatrix}:H\oplus\hat H \oplus \cH_b\to\cK_a\oplus\cK_b\label{5.70}\end{gathered}$$ Combining now the last statement in Proposition \[pr5.10\] with formulas and we obtain the following assertion:

\(a) the boundary conditions are maximal dissipative (resp. maximal accumulative) if and only if $\im (\wt C_1 \wt C_0^*)\geq 0$ and $0\in\rho (\wt
C_0-i\wt C_1 )$ (resp. $\im (\wt C_1 \wt C_0^*)\leq 0$ and $0\in\rho (\wt
C_0+i\wt C_1 )$ ).

It follows from and that $$\begin{gathered}
\wt C_1 \wt C_0^*=\begin{pmatrix} N_{1a}N_{0a}^*+\tfrac i 2 \hat N_a \hat N_a^*
& -\tfrac i 2 \hat N_a \hat N_b^* \cr \tfrac i 2 \hat N_b \hat N_a^* &
-N_{1b}N_{0b}^*-\tfrac i 2 \hat N_b \hat N_b^*
\end{pmatrix}\end{gathered}$$ and, consequently, $ \im (\wt C_1 \wt C_0^*)=\text{diag}(S_a, -S_b)$. Hence the following equivalences are valid $$\begin{gathered}
\label{5.71}
\im (\wt C_1 \wt C_0^*)\geq 0 \iff S_a\geq 0 \;\;\text{and}\;\; S_b\leq 0;
\quad \im (\wt C_1 \wt C_0^*)\leq 0 \iff S_a\leq 0 \;\;\text{and}\;\; S_b\geq
0.\end{gathered}$$ Moreover, by and one has $$\begin{gathered}
\wt C_0-i\wt C_1 =\begin{pmatrix}\wt N_a & 0 \cr 0 & N_{0b}+i
N_{1b}\end{pmatrix}:(H\oplus\hat H)\oplus \cH_b\to \cK_a\oplus\cK_b,\\
\wt C_0+i\wt C_1 =\begin{pmatrix} N_{0a}+i N_{1a} & 0 \cr 0 & \wt N_b
\end{pmatrix}:H\oplus(\hat H\oplus \cH_b)\to
\cK_a\oplus\cK_b,\end{gathered}$$ which yields the equivalences $$\begin{gathered}
\label{5.72}
0\in\rho (\wt C_0-i\wt C_1 )\Leftrightarrow 0\in \rho (\wt N_a)\cap \rho
(N_{0b}+i N_{1b}),\quad 0\in\rho (\wt C_0+i\wt C_1 )\Leftrightarrow 0\in\rho
(N_{0a}+i N_{1a}) \cap \rho (\wt N_b).\end{gathered}$$ Now assertion (a) together with and gives the required description of all maximal dissipative and accumulative separated boundary conditions by means of and . Moreover, implies that $$\begin{gathered}
\dim\cK_a=\dim (H\oplus\hat H), \qquad \dim\cK_b=\dim\cH_b,\end{gathered}$$ which in view of and leads to . Similarly one proves the equalities .

2\) Since self-adjoint separated boundary conditions are simultaneously maximal dissipative and maximal accumulative, it follows from and that an existence of such conditions yields the equality $\nu_-=\nu_+$. Moreover, if this equality is satisfied, then the general form of self-adjoint separated boundary conditions follows from the statement 1) of the theorem.

\[rem5.13\] 1) Theorem \[th5.12\] enables one to introduce the important class of maximal accumulative (dissipative) separated boundary conditions, which consist of the self-adjoint condition at the regular endpoint $a$ and the maximal accumulative (dissipative) condition at the point $b$. If, for instance, $\nu_-\geq \nu_+$, then such separated conditions are defined by $$\begin{gathered}
\wt A=\{\{\wt y, \wt f\}\in\Tma:\,N_{0a}y_0(a)+ N_{1a}y_1(a)=0, \;\;
N_{0b}\G_{0b}y+\hat N_b \hat \G_b y + N_{1b}\G_{1b}y=0 \},\end{gathered}$$ where the operators $N_{0a}$ and $N_{0b}$ form the self-adjoint pair $\t_a=\{(N_{0a}, N_{1a} ); \cK_a\} $, while the operators $N_{0b}, \; N_{1b}$ and $\hat N_b$ form the maximal accumulative pair $\t_b=\{((\tfrac i {\sqrt
2}\hat N_b :N_{0b}),( \tfrac 1 {\sqrt 2}\hat N_b:-N_{1b} )); \cK_b\}$.

2\) For a regular definite system one can put in Corollary \[cor5.10a\] $\bH_b=\bH, \; J_b=J$ and $\G_b y=y(b), y\in\dom\tma$, in which case this corollary gives the following well known statement [@GK; @Orc]: the extension $\wt A= \{\{\wt y, \wt f\}\in\Tma :\, C_a y(a)+C_b y(b)=0\}$ is self-adjoint if and only if $C_aJC_a^*=C_bJC_b^*$. The case of the singular endpoint $b$ under the additional assumptions $\nu_+=\nu_-$ and $\mul
\Tma=\{0\}$ was considered in the paper [@Kra89], where the criterium for self-adjointness of the boundary condition in the form of the last equality in was obtained. Note in this connection that our approach based on the concept of a decomposing boundary triplet seems to be more convenient. In particular, such an approach made it possible to describe in Theorem \[th5.12\] various classes of separated boundary conditions.

[DHS]{}

F.V. Atkinson, *Discrete and continuous boundary problems*, Academic Press, New York, 1964.

J. Behrndt, S. Hassi, H. de Snoo, and R. Wiestma, *Square-integrable solutions and Weyl functions for singular canonical systems*, Math. Nachr. **284** (2011), no. 11-12, 1334–1384.

J. Behrndt and M. Langer, *Boundary value problems for elliptic partial differential operators on bounded domains*, J. Funct. Anal. **243** (2007), 536–565.

C. Bennevitz, *Symmetric relations on a Hilbert space*, Lecture notes in mathematics, vol. 280, Springer, Berlin, 1972.

V.A. Derkach, S. Hassi, M.M. Malamud, and H.S.V. de Snoo, *Boundary relations and their Weyl families*, Trans. Amer. Math. Soc. **358** (2006), no. 12, 5351–5400.

V.A. Derkach, S. Hassi, M.M. Malamud, and H.S.V. de Snoo, *Boundary relations and generalized resolvents of symmetric operators*, Russian J. Math. Ph. **16** (2009), no. 1, 17–60.

V.A. Derkach and M.M. Malamud, *Generalized resolvents and the boundary value problems for Hermitian operators with gaps*, J. Funct. Anal. **95** (1991),1–95.

V.A.  Derkach and M.M.  Malamud, *The extension theory of Hermitian operators and the moment problem*, J. Math. Sciences, 73, (1995), no 2, 141-242.

A. Dijksma, H. Langer, and H.S.V. de Snoo, *Hamiltonian systems with eigenvalue depending boundary conditions*, Oper. Theory Adv. Appl. **35** (1988), 37–83.

A. Dijksma, H. Langer, and H.S.V. de Snoo, *Eigenvalues and pole functions of Hamiltonian systems with eigenvalue depending boundary conditions*, Math. Nachr. **161** (1993), 107–154.

N. Dunford and J.T. Schwartz, *Linear operators. Part2. Spectral theory*, Interscience Publishers, New York-London, 1963.

I. Gohberg and M.G. Krein, *Theory and applications of Volterra operators in Hilbert space*, Transl. Math. Monographs, 24, Amer. Math. Soc., Providence, R.I., 1970

V.I. Gorbachuk and M.L. Gorbachuk, *Boundary problems for differential-operator equations*, Kluver Acad. Publ., Dordrecht-Boston-London, 1991. (Russian edition: Naukova Dumka, Kiev, 1984).

S. Hassi, H.S.V. de Snoo, and H. Winkler, *Boundary-value problems for two-dimensional canonical systems*, Integral Equations Operator Theory **36** (2000), 445–479.

D.B. Hinton and A. Schneider, *On the Titchmarsh-Weyl coe±cients for singular S-Hermitian systems I*, Math. Nachr. **163** (1993), 323–342.

I.S. Kats, *On Hilbert spaces generated by monotone Hermitian matrix-functions*, Khar’kov. Gos. Univ. Uchen. Zap. **34** (1950), 95-113=Zap.Mat.Otdel.Fiz.-Mat. Fak. i Khar’kov. Mat. Obshch. (4)**22** (1950), 95-113.

I.S. Kac, *Linear relations generated by canonical di®erential equations*, Funct. Anal. Appl., **17** (1983) 86-87 (Russian).

I.S. Kac, *Linear relations, generated by a canonical differential equation on an interval with regular endpoints, and the expansibility in eigenfunctions*, Deposited Paper, Odessa, 1984 (Russian).

V.I. Kogan and F.S. Rofe-Beketov, *On square-integrable solutions of symmetric systems of differential equations of arbitrary order*, Proc. Roy. Soc. Edinburgh Sect. A **74** (1974/75), 5–40.

A.M. Krall, *$M(\l)$-theory for singular Hamiltonian systems with one singular endpoint*, SIAM J. Math. Anal. **20** (1989), 664–700.

H. Langer and B. Textorius, *$L$-resolvent matrices of symmetric linear relations with equal defect numbers; appliccations to canonical differential relations*, Integral Equations Operator Theory **5** (1982), 208–243.

M. Lesch and M.M. Malamud, *On the deficiency indices and self-adjointness of symmetric Hamiltonian systems*, J. Differential Equations **189** (2003), 556–615.

M. M.  Malamud, *On the formula of generalized resolvents of a nondensely defined Hermitian operator*, Ukr. Math. Zh. **44**(1992), no. 12, 1658-1688.

V.I.Mogilevskii, *Nevanlinna type families of linear relations and the dilation theorem*, Methods Funct. Anal. Topology **12** (2006), no. 1, 38–56.

V.I.Mogilevskii, *Boundary triplets and Krein type resolvent formula for symmetric operators with unequal defect numbers*, Methods Funct. Anal. Topology **12** (2006), no. 3, 258–280.

M.A. Naimark, *Linear differential operators, vol. 1 and 2*, Harrap, London, 1967. (Russian edition: Nauka, Moscow, 1969).

B.C. Orcutt, *Canonical differential equations*, Dissertation, University of Virginia, 1969.

F.S. Rofe-Beketov, *Self-adjoint extensions of differential operators in the space of vector-valued functions*, Teor. Funkcii Funkcional. Anal. Prilozhen. **8** (1969), 3–24.

Yu.L. Shmul’an, *Theory of linear relations and spaces with indefinite metric*, Funktsional. Anal. i Prilozhen. **10** (1976), no. 1, 67–72.
