CHAPTER 17 OF THE LIMITS OF THE PENAL BRANCH OF JURUS PRUDENCE Section 1. Limits between private ethics and the art of legislation. So much for the division of offences in general. Now an offence is an act prohibited, or what comes to the same thing, an act of which the contrary is commanded by the law. And what is it that the law can be employed in doing, besides prohibiting and commanding? It should seem then, according to this view of the matter, that were we to have settled what may be proper to be done with relation to offences, we should thereby have settled everything that may be proper to be done in the way of law. Yet that branch which concerns the method of dealing with offences, and which is termed sometimes the criminal, sometimes the penal branch, is universally understood to be but one out of two branches, which compose the whole subject of the art of legislation. That which is termed the civil being the other. Between these two branches then, it is evident enough there cannot but be a very intimate connection. So intimate is it indeed that the limits between them are by no means easy to mark out. The case is the same in some degree between the whole business of legislation, civil and penal branches taken together, and that of private ethics. With these several limits however, it will be in a manner necessary to exhibit some idea, lest on the one hand we should seem to leave out any part of the subject that does belong to as untouched, or on the other hand, to deviate on any side into a track which does not belong to us. In the course of this inquiry, that part of it I mean, which concerns the limits between the civil and the penal branch of law, it will be necessary to settle a number of points, of which the connection with the main question might not at first sight be suspected. To ascertain what sort of a thing a law is, what the parts are that are to be found in it, what it must contain in order to be complete, what the connection is between that part of a body of laws which belongs to the subject of procedure and the rest of the law at large. All these, it will be seen, are so many problems which must be solved before any satisfactory answer can be given to the main question above mentioned. Nor is this their only use, for it is evident enough that the notion of a complete law must first be fixed before the legislator can, in any case, know what it is he has to do, or when his work is done. Ethics at large may be defined, the art of directing men's actions to the production of the greatest possible quantity of happiness on the part of those whose interest is in view. What then are the actions which it can be in a man's power to direct? They must be either his own actions or those of other agents. Ethics in as far as it is the art of directing a man's own actions may be styled the art of self-government or private ethics. What other agents then are there which at the same time that they are under the influence of man's direction are susceptible of happiness? They are of two sorts. One are the human beings who are styled persons. Two are the animals which on account of their interests having been neglected by the insensibility of the ancient jurists stand degraded into the class of things. As to other human beings, the art of directing their actions to the above end is what we mean, or at least the only thing which upon the principle of utility we ought to mean by the art of government, which in as far as the measures it displays itself in are of a permanent nature, is generally distinguished by the name of legislation, as it is by that of administration when they are of temporary nature determined by the occurrences of the day. Now human creatures considered with respect to the maturity of their faculties are either in an adult or in a non-adult state. The art of government in as far as it concerns the direction of the actions of persons in a non-adult state may be termed the art of education. In as far as this business is entrusted with those who, in virtue of some private relationship, are in the main the best disposed to take upon them and the best able to discharge this office, it may be termed the art of private education. In as far as it is exercised by those whose province it is to superintend the conduct of the whole community, it may be termed the art of public education. As to ethics in general, a man's happiness will depend, in the first place, upon such parts of his behavior as none but himself are interested in. In the next place, upon such parts of it as may affect the happiness of those about him. In as far as his happiness depends upon the first mention part of his behavior, it is said to depend upon his duty to himself. Ethics then, in as far as it is the art of directing a man's actions in this respect, may be termed the art of discharging one's duty to oneself, and the quality which a man manifests by the discharge of this branch of duty, if duty it is to be called, is that of prudence. In as far as his happiness and that of any other person or persons whose interests are considered depends upon such parts of his behavior as may affect the interests of those about him, it may be said to depend upon his duty to others, or, to use a phrase now somewhat antiquated, his duty to his neighbor. Ethics then, in as far as it is the art of directing a man's actions in this respect, may be termed the art of discharging one's duty to one's neighbor. Now the happiness of one's neighbor may be consulted in two ways. One, in a negative way by forbearing to diminish it. Two, in a positive way by studying to increase it. A man's duty to his neighbor is accordingly partly negative and partly positive. To discharge the negative branch of it is probity. To discharge the positive branch, beneficence. It may here be asked, how is it that upon the principle of private ethics, legislation and religion out of the question, a man's happiness depends upon such part of his conduct as affect, immediately at least, the happiness of no one but himself. This is as much to ask, what motives, independent of such as legislation and religion may chance to furnish, can one man have to consult the happiness of another by what motives, or, which comes to the same thing, by what obligations can he be bound to obey the dictates of probity and beneficence? In answer to this, it cannot but be admitted that the only interests which a man at all times and upon all occasions is sure to find adequate motives for consulting are his own. Notwithstanding this, there are no occasions in which a man has not some motives for consulting the happiness of other men. In the first place, he has, on all occasions, the purely social motive of sympathy or benevolence. In the next place, he has, on most occasions, the semi-social motives of love of amity and love of reputation. The motive of sympathy will act upon him with more or less effect according to the bias of his sensibility. The two other motives, according to a variety of circumstances, principally according to the strength of his intellectual powers, the firmness and steadiness of his mind, the quantum of his moral sensibility, and the characters of the people he has to deal with. Now private ethics has happiness for its end, and legislation can have no other. Private ethics concerns every member, that is, the happiness and actions of every member, of any community that can be proposed, and legislation can concern no more. Thus far, then, private ethics and the art of legislation go hand in hand. The end they have, or ought to have, in view, is of the same nature. The persons whose happiness they ought to have in view, as also the persons whose conduct they ought to be occupied in directing, are precisely the same. The very acts they ought to be conversant about are even in a great measure the same. Where, then, lies the difference? In that the acts which they ought to be conversant about, though in a great measure, are not perfectly and throughout the same. There is no case in which a private man ought not to direct his own conduct to the production of his own happiness, and that of his fellow creatures. But there are cases in which the legislator ought not, in a direct way at least, and by means of punishment applied immediately to particular individual acts, to attempt to direct the conduct of the several other members of the community. Every act which promises to be beneficial upon the whole to the community, himself included, each individual ought to perform of himself. But it is not every such act that the legislator ought to compel him to perform. Every act which promises to be pernicious upon the whole to the community, himself included, each individual ought to abstain from of him. But it is not every such act that the legislator ought to compel him to abstain from. Where, then, is the line to be drawn? We shall have not far to seek for it. The business is to give an idea of the cases in which ethics ought, and in which legislation ought not, in a direct manner at least, to interfere. If legislation interferes in a direct manner, it must be by punishment. Now the cases in which punishment, meaning the punishment of the political sanction, ought not to be inflicted, have been already stated. If, then, there be any of these cases in which, although legislation ought not, private ethics does, or ought to interfere, these cases will serve to point out the limits between the two arts or branches of science. These cases, it may be remembered, are of four sorts. One, where punishment would be groundless. Two, where it would be inefficacious. Three, where it would be unprofitable. Four, where it would be needless. Let us look over all these cases, and see whether in any of them there is room for the interference of private ethics, at the same time that there is none for the direct interference of legislation. One, first, then, as to the cases where punishment would be groundless. In these cases, it is evident that the restrictive interference of ethics would be groundless, too. It is because, upon the whole, there is no evil in the act that legislation ought not to endeavor to prevent it. No more, for the same reason, ought private ethics. Two, as to the cases in which punishment would be inefficacious, these, we may observe, may be divided into two sets or classes. The first do not depend at all upon the nature of the act. They turn only upon a defect in the timing of the punishment. The punishment in question is no more than what, for anything that appears, ought to have been applied to the act in question. It ought, however, to have been applied at a different time. We delicate, not till after it had been properly denounced. These are cases of an ex-post facto law, of a judicial sentence beyond the law, and of a law not sufficiently promulgated. The acts here in question, then might, for anything that appears, come properly under the department, even of coercive legislation. Of course do they under that of private ethics. As to the other set of cases in which punishment would be inefficacious, neither do these depend upon the nature of the act, that is, the sort of act. They turn only upon some extraneous circumstances with which an act of any sort may chance to be accompanied. These, however, are of such a nature as not only to exclude the application of legal punishment, but in general to leave little room for the influence of private ethics. These are the cases where the will could not be deterred from any act, even by the extraordinary force of artificial punishment, as in the cases of extreme infancy, insanity, and perfect intoxication. Of course, therefore, it could not by such slender and precarious force as could be applied by private ethics. The case is in this respect the same, under the circumstances of unintentionality, with respect to the event of the action, unconsciousness with regard to the circumstances, and mis-supposal with regard to the existence of circumstances which have not existed, as also where the force, even of extraordinary punishment, is rendered inoperative by the superior force of a physical danger or threatened mischief. It is evident that in these cases, if the thunders of the law prove impotent, the whispers of simple morality can have but little influence. Three, as to the cases where punishment would be unprofitable. These are the cases which constitute the great field for the exclusive interference of private ethics. When a punishment is unprofitable, or in other words too expensive, it is because the evil of the punishment exceeds that of the offence. Now the evil of the punishment we may remember is distinguishable into four branches. One, the evil of coercion, including constraint or restraint, according as the act commanded, is of the positive kind or the negative. Two, the evil of apprehension. Three, the evil of sufferance. Four, the derivative evils resulting to persons in connection with those by whom the three above to mentioned original evils are sustained. Now with respect to those original evils, the persons who lie exposed to them may be two very different sets of persons. In the first place, persons who may have actually committed or been prompted to commit the acts really meant to be prohibited. In the next place, persons who may have performed or been prompted to perform such other acts as they fear may be in danger of being involved in the punishment designed only for the former. But of these two sets of acts, it is the former only that are pernicious. It is therefore the former only that it can be the business of private ethics to endeavor to prevent. The latter, being by the supposition not mischievous, to prevent them is what it can no more be the business of ethics to endeavor at than of legislation. It remains to show how it may happen that there should be acts really pernicious which, although they may very properly come under the censure of private ethics, may yet be no fit objects for the legislator to control. Punishment then, as applied to delinquency, may be unprofitable in both or either of two ways. One, by the expense it would amount to, even supposing the application of it to be confined altogether to delinquency. Two, by the danger there may be of its involving the innocent in the fate designed only for the guilty. First, then, with regard to the cases in which the expense of the punishment, as applied to the guilty, would outweigh the profit to be made by it. These cases, it is evident, depend upon a certain proportion between the evil of the punishment and the evil of the offence. Now were the offence of such a nature that a punishment which, in point of magnitude, should but just exceed the profit of it, would be sufficient to prevent it, it might be rather difficult, perhaps, to find an instance in which such punishment would clearly appear to be unprofitable. But the fact is that there are many cases in which a punishment, in order to have any chance of being efficacious, must, in point of magnitude, be raised a great deal above that level. Thus it is, wherever the danger of detection is, or what comes to the same thing, is likely to appear to be so small as to make the punishment appear in a high degree uncertain. In this case, it is necessary, as has been shown, if punishment be at all applied, to raise it in point of magnitude as much as it falls short in point of certainty. It is evident, however, that all this can be but guesswork, and that the effect of such proportion will be rendered precarious by a variety of circumstances, by the want of sufficient promulgation on the part of the law, by the particular circumstances of the temptation, and by the circumstances influencing the sensibility of the several individuals who are exposed to it. Let the seducing motives be strong, the offence, then, will at any rate be frequently committed. Now and then, indeed, owing to a coincidence of circumstances more or less extraordinary, it will be detected, and by that means punished. But for the purpose of example, which is the principle one, an act of punishment considered in itself is of no use. What use it can be of depends altogether upon the expectation it raises of similar punishment in future cases of similar delinquency. But this future punishment, it is evident, must always depend upon detection. If, then, the want of detection is such as must in general, especially to eyes fascinated by the force of the seducing motives, appear too improbable to be reckoned upon, the punishment, though it should be inflicted, may come to be of no use. Here, then, will be two opposite evils running on at the same time, yet neither of them reducing the quantum of the other, the evil of the disease and the evil of the painful and inefficacious remedy. It seems to be partly owing to some such considerations that fornication, for example, or the illicit commerce between the sexes, has commonly either gone altogether unpunished, or been punished in a degree inferior to that in which, on other accounts, legislators might have been disposed to punish it. Secondly, with regard to the cases in which political punishment, as applied to delinquency, may be unprofitable, in virtue of the danger that there may be of its involving the innocent in the fate designed only for the guilty, when should this danger then arise? From the difficulty, there may be of fixing the idea of the guilty action. That is, of subjecting it to such a definition as shall be clear and precise enough to guard effectively against misapplication. This difficulty may arise from either of two sources, the one permanent, to wit, the nature of the actions themselves, the other occasional, I mean the qualities of the men who may have to deal with those actions in the way of government. In as far as it arises from the latter of these sources, it may depend partly upon the use which the legislator may be able to make of language, partly upon the use which, according to the apprehension of the legislators, the judge may be disposed to make of it. As far as legislation is concerned, it will depend upon the degree of perfecting to which the arts of language may have been carried, in the first place, in the nation in general, in the next place, by the legislator in particular. It is to a sense of this difficulty, as it should seem, that we may attribute the caution with which most legislators have abstained from subjecting to censure on the part of the law. Such actions as come under the notion of rudeness, for example, or treachery, or ingratitude. The attempt to bring acts of so vague and questionable a nature under the control of law will argue either a very immature age in which the difficulties which give birth to that danger are not described, or a very enlightened age in which they are overcome. For the sake of obtaining the clearer idea of the limits between the art of legislation and private ethics, it may now be time to call to mind the distinctions above established with regard to ethics in general. The degree in which private ethics stands in need of the assistance of legislation is different in the three branches of duty above distinguished. Of the rules of moral duty, those which seem to stand least in need of the assistance of legislation are the rules of prudence. It can only be through some defect on the part of the understanding if a man be ever deficient in point of duty to himself. If he does wrong, there is nothing else that it can be owing to, but either some inadvertence or some misopposal with regard to the circumstances on which his happiness depends. It is a standing topic of complaint that a man knows too little of himself, be it so. But is it so certain that the legislator must know more? It is plain that of individuals the legislator can know nothing. Concerning those points of conduct which depend upon the particular circumstances of each individual, it is plain therefore that he can determine nothing to advantage. It is only with respect to those broad lines of conduct in which all persons, or very large and permanent descriptions of persons, may be in a way to engage that he can have any pretense for interfering. And even here the propriety of his interference will, in most circumstances, lie very open to dispute. At any rate he must never expect to produce a perfect compliance by the mere force of the sanction of which he is himself the author. All he can hope to do is to increase the efficacy of private ethics by giving strength and direction to the influence of the moral sanction. With what chance of success, for example, would a legislator go about to extirpate drunkenness and fornication by dint of legal punishment? Not all the tortures which ingenuity could invent would compass it. And before he had made any progress worth regarding, such a mass of evil would be produced by the punishment as would exceed a thousandfold the utmost possible mischief of the offence. The great difficulty would be in procuring evidence, an object which could not be attempted with any probability of success without spreading dismay through every family, tearing the bonds of sympathy asunder and rooting out the influence of all the social motives. All he can do then against offences of this nature, with any prospect of advantage in the way of direct legislation, is to subject them, in cases of notoriety, to a slight censure, so as to thereby cover them with a slight shade of artificial disrepute. It may be observed that with regard to this branch of duty legislators have, in general, been disposed to carry their interference full as far as is expedient. The great difficulty here is to persuade them to confine themselves within bounds. A thousand little passions and prejudices have led them to narrow the liberty of the subject in this line, in cases in which the punishment is either attended with no profit at all or with none that will make up for the expense. The mischief of this sort of interference is more particularly conspicuous in the article of religion. The reasoning in this case is of the following stamp. There are certain errors in matters of belief to which all mankind are prone, and for these errors in judgment, it is the determination of a being of infinite benevolence to punish them with an infinity of torments. But from these errors the legislator himself is necessarily free, for the men who happen to be at hand for him to consult with, being men perfectly enlightened, unfettered, and unbiased, have such advantages over all the rest of the world, that when they sit down to inquire out the truth relative to points so plain and so familiar as those in question they cannot fail to find it. This being the case, when the sovereign sees his people ready to plunge headlong into an abyss of fire, shall he not stretch out a hand to save them? Such for example seems to have been the train of reasoning and such the motives, which led Louis XIV into those coercive measures which he took for the conversion of heretics and the confirmation of true believers. The groundwork, pure sympathy and loving kindness, the superstructure, all the miseries which the most determined malevolence could have devised, but of this more fully in another place. The rules of poverty are those which in point of expediency stand most in need of assistance on the part of the legislator, and in which, in point of fact, his interference has been most extensive. There are few cases in which it would be expedient to punish a man for hurting himself, but there are few cases, if any, in which it would not be expedient to punish a man for injuring his neighbour. With regard to that branch of probity which is opposed to offences against property, private ethics depends in a manner for its very existence upon legislation. Legislation must first determine what things are to be regarded as each man's property before the general rule of ethics on this head can have any particular application. The case is the same with regard to offences against the state. Without legislation there would be no such thing as a state, no particular persons invested with powers to be exercised for the benefit of the rest. It is plain, therefore, that in this branch the interference of the legislator cannot anywhere be dispensed with. We must first know what are the dictates of legislation before we can know what are the dictates of private ethics. As to the rules of beneficence, these, as far as concerns matters of detail, must necessarily be abandoned in great measure to the jurisdiction of private ethics. In many cases, the beneficial quality of the act depends essentially upon the disposition of the agent. That is, upon the motive by which he appears to have been prompted to perform it, upon their belonging to the head of sympathy, love of amity, or love of reputation, and not to any head of self-regarding motives brought into play by the force of political constraint, in a word, upon their being such as denominate his conduct free and voluntary, according to one of the many senses given to those ambiguous expressions. The limits of the law on this head seem, however, to be capable of being extended a good deal farther than they seem ever to have been extended hitherto. In particular, in cases where the person is in danger, why should it not be made the duty of every man to save another from mischief when it can be done without prejudicing himself, as well as to abstain from bringing it on him? This accordingly is the idea pursued in the body of the work. To conclude this section, let us recapitulate and bring to a point the difference between private ethics, considered as an art or science on the one hand, and that branch of jurisprudence which contains the art or science of legislation on the other. Private ethics teaches how each man may dispose himself to pursue the course most conducive to his own happiness by means of such motives as offer of themselves. The art of legislation, which may be considered as one branch of the science of jurisprudence, teaches how a multitude of men composing a community may be disposed to pursue that course, which upon the whole is the most conducive to the happiness of the whole community, by means of motives to be applied by the legislator. We come now to exhibit the limits between penal and civil jurisprudence. For this purpose it may be of use to give a distinct, though summary view, of the principal branches into which jurisprudence, considered in its utmost extent, is wanted to be divided. Section two, jurisprudence, its branches. Jurisprudence is a fictitious entity, nor can any meaning be found for the word, but by placing it in company with some word, it shall be significant of a real entity. To know what is meant by jurisprudence, we must know, for example, what is meant by a book of jurisprudence. A book of jurisprudence can have but one or the other of two objects, one, to ascertain what the law is, two, to ascertain what it ought to be. In the former case it may be styled a book of expository jurisprudence, in the latter, a book of sensorial jurisprudence, or in other words, a book on the art of legislation. A book of expository jurisprudence is either authoritative or unauthoritative. It is styled authoritative when it is composed by him who, by representing the state of the law to be so and so, causeth it so to be, that is, of the legislator himself, unauthoritative when it is the work of any other person at large. Now law, or the law, taken indefinitely, is an abstract and collective term, which, when it means anything, can mean neither more nor less than the sum total of a number of individual laws taken together. It follows that of whatever other modifications the subject of a book of jurisprudence is susceptible, they must all of them be taken from some circumstance or other, of which such individual laws or the assemblages into which they may be sorted are susceptible. The circumstances that have given rise to the principal branches of jurisprudence we are want to hear of seem to be as follows. One, the extent of the laws in question in point of dominion. Two, the political quality of the persons whose conduct they undertake to regulate. Three, the time of their being in force. Four, the manner in which they are expressed. Five, the concern which they have with the article of punishment. In the first place, in point of extent, what is delivered concerning the laws in question may have reference either to the laws of such and such a nation, or nations in particular, or to the laws of all nations whatsoever. In the first case, the book may be said to relate to local, in the other, to universal jurisprudence. Now of the infinite variety of nations there are upon the earth, there are no two which agree exactly in their laws, certainly not in the whole, perhaps not even in any single article. And let them agree today, they would disagree tomorrow. This is evident enough with regard to the substance of the laws, and it would still be more extraordinary if they agreed in point of form, that is, if they were conceived in precisely the same strings of words. What is more, as the languages of nations are commonly different as well as their laws, it is seldom that, strictly speaking, they have so much as a single word in common. However, among the words that are appropriated to the subject of law, there are some that in all languages are pretty exactly correspondent to one another, which comes to the same thing nearly as if they were the same. Of this stamp, for example, are those which correspond to the words power, right, obligation, liberty, and many others. It follows that if there are any books which can, properly speaking, be styled books of universal jurisprudence, they must be looked for within very narrow limits. Among such as our expository, there can be none that are authoritative, nor even as far as the substance of the laws is concerned any that are unauthoritative. To be susceptible of an universal application, all that a book of the expository kind can have to treat of is the import of words. To be, strictly speaking, universal, it must confine itself to terminology. Accordingly, the definitions which there has been occasioned here and there to intersperse in the course of the present work, and particularly the definition hereafter given to the word law, may be considered as matter belonging to the head of universal jurisprudence. Thus far in strictness of speech, though in point of usage we're a man, in laying down what he apprehends to be the law, extends his view to a few of the nations with which his own is most connected, it is common enough to consider what he writes as relating to universal jurisprudence. It is in the sensorial line that there is the greatest room for disquisitions that apply to the circumstances of all nations alike. And in this line, what regards the substance of the laws in question is as susceptible of an universal application as what regards the words. That the laws of all nations, or even of any two nations should coincide in all points, would be as ineligible as it is impossible. Some leading points, however, there seem to be in respect of which the laws of all civilised nations might, without inconvenience, be the same. To mark out some of these points will, as far as it goes, be the business of the body of this work. In the second place, with regard to the political quality of the persons whose conduct is the object of the law, these may, on any given occasion, be considered as either members of the same state or as members of different states. In the first case, the law may be referred to the head of internal, in the second case, to that of international jurisprudence. Now, as to any transactions which may take place between individuals who are subjects of different states, these are regulated by internal laws and decided upon by the internal tribunals of the one or the other of those states. The case is the same where the sovereign of the one has any immediate transactions with a private member of the other, the sovereign reducing himself, to the condition of a private person, as often as he submits his cause to either tribunal, whether by claiming a benefit or defending himself against a burden. There remain then the mutual transactions between sovereigns as such for the subject of that branch of jurisprudence, which may be properly and exclusively termed international. With what degree of propriety rules for the conduct of persons of this description can come under the appellation of laws is a question that must rest till the nature of the thing called a law shall have been more particularly unfolded. It is evident enough that international jurisprudence may as well as internal be sensorial as well as expository, unauthoritative as well as authoritative. Internal jurisprudence again may either concern all the members of a state indiscriminately or such of them as are connected in the way of residence or otherwise with a particular district. Jurisprudence is accordingly sometimes distinguished international and provincial, but as the epithet provincial is hardly applicable to a district so small as many of those which have laws of their own are want to be, such as towns, parishes and manors, the term local where universal jurisprudence is plainly out of the question or the term particular, but this latter is not very characteristic, might either of them be more commodious. Thirdly, with respect to time. In a work of the expository kind, the laws that are in question may either be such as are still in force at the time when the book is writing or such as have ceased to be in force. In the latter case, the subject of it might be termed ancient in the former, present or living jurisprudence. That is, if the substantive jurisprudence and no other must at any rate be employed and that with an epithet in both cases. But the truth is that a book of the former kind is rather a book of history than a book of jurisprudence, and if the word jurisprudence be expressive of the subject, it is only with some such words as history or antiquities prefixed. And as the laws which are anywhere in question are supposed, if nothing appears to the contrary to be those which are in force, no such epithet as that of present or living commonly appears. Where a book is so circumcised that the laws from which the subject of it, though in force at the time of its being written are in force no longer, that book is neither a book of living jurisprudence nor a book on the history of jurisprudence. It is no longer the former and it was never the latter. It is evident that owing to the changes which from time to time must take place in a greater or less degree in every body of laws, every book of jurisprudence, which is of an expository nature, must in the course of a few years come to partake more or less of this condition. The most common and most useful object of a history of jurisprudence is to exhibit the circumstances that have attended the establishment of laws that are actually in force. But the exposition of the dead laws, which have been superseded, is inseparably interwoven with that of the living ones which have superseded them. The great use of both these branches of science is to furnish examples for the art of legislation. Fourthly, in point of expression, the laws in question may subsist either in the form of statute or in that of customary law. As to the difference between these two branches, which respects only the article of form or expression, it cannot properly be made appear till some progress has been made in the definition of a law. Lastly, the most intricate distinction of all, and that which comes most frequently on the carpet, is that which is made between the civil branch of jurisprudence and the penal, which latter is want in certain circumstances to receive the name of criminal. What is a penal code of laws? What is a civil code? Of what nature are their contents? Is it that there are two sorts of laws? The one penal, the other civil, so that the laws in a penal code are all penal laws? While the laws in a civil code are all civil laws? Or is it that in every law there is some matter which is of a penal nature and which therefore belongs to the penal code? And at the same time, are the matter which is of a civil nature and which therefore belongs to the civil code? Or is it that some laws belong to one code or the other exclusively while others are divided between the two? To answer these questions in any manner that shall be tolerably satisfactory, it will be necessary to ascertain what a law is, meaning one entire but single law, and what are the parts into which a law, as such, is capable of being distinguished, or in other words, to ascertain what the properties are that are to be found in every object which can, with propriety, receive the appellation of a law. This then will be the business of the third and fourth sections. What concerns the import of the word criminal as applied to law will be discussed separately in the fifth. Footnote 27. Here ends the original work in the state into which it was brought in 5 November 1780. What follows is now added in January 1789. The third, fourth, and fifth sections intended, as expressed in the text, to have been added to this chapter will not here nor now be given because to give them in a manner tolerably complete and satisfactory might require a considerable volume. This volume will form a work of itself, closing the series of works mentioned in the preface. What follows here may serve to give a slight intimation of the nature of the task which such a work will have to achieve. It will at the same time furnish not anything like a satisfactory answer to the questions mentioned in the text but a slight and general indication of the course to be taken for giving them such an answer. What is a law? What are the parts of a law? The subject of these questions it is to be observed is the logical, the ideal, the intellectual whole, not the physical one, the law and not the statute. An inquiry directed to the latter sort of object could neither admit of difficulty nor afford instruction. In this sense, whatever is given for law by the person or persons recognized as possessing the power of making laws is law. The metamorphosis of Ovid, if thus given, would be law. So much as was embraced by one and the same act of authentication, so much as received the touch of the scepter at one stroke is one law, a whole law and nothing more. A statute of George II made to substitute an or instead of an and in a former statute is a complete law. A statute containing an entire body of laws, perfect in all its parts, would not be more so. By the word law then, as often as it occurs in the succeeding pages is meant that ideal object of which the part, the whole or the multiple or an assemblage of parts, holes and multiples mixed together is exhibited by a statute, not the statute which exhibits them. Every law when complete is either of a coercive or an uncoercive nature. A coercive law is a command. An uncoercive or rather a discoercive law is the revocation in whole or in part of a coercive law. What has been termed declaratory law so far as it stands distinguished from either a coercive or discoercive law is not properly speaking a law. It is not the expression of an act of the will exercised at the time. It is a mere notification of the existence of a law either of the coercive or the discoercive kind as already subsisting of the existence of some document expressive of some act of the will exercised not at the time but at some form or period. If it does anything more than give information of this fact, we delicate of the prior existence of a law of either the coercive or the discoercive kind. It ceases pro tanto to be what is meant by a declaratory law and assuming either the coercive or the discoercive quality. Every coercive law creates an offense that is converts an act of some sort or other into an offense. It is only by doing so that it can impose obligation that it can produce coercion. A law confining itself to the creation of an offense and a law commanding a punishment to be administered in the case of the commission of such an offense are two distinct laws, not parts as they seem to have been generally accounted here the two of one and the same law. The acts they command are altogether different. The persons they are addressed to are altogether different. For instance, let no man steal and let the judge cause whoever is convicted of stealing to be hanged. They might be styled the former, a simply imperative law, the other a punatory, but the punatory, if it commands the punishment to be inflicted and does not merely permit it, is as truly imperative as the other. Only it is punatory besides which the other is not. A law of the discoercive kind, considered in itself, can have no punatory law belonging to it. To receive the assistance and support of a punatory in law, it must first receive that of a simply imperative or coercive law, and it is to this latter that the punatory law will attach itself and not to the discoercive one. Example, discoercive law. The sheriff has the power to hang all such as the judge preceding in due course of law shall order him to hang. Example of a coercive law made in support of the above discoercive one, let no man hinder the sheriff from hanging such as the judge preceding in due course of law shall order him to hang. Example of a punatory law made in support of the above coercive one, let the judge cause to be imprisoned who so ever attempts to hinder the sheriff from hanging one whom the judge preceding in due course of law has ordered him to hang. But those simply imperative law and the punatory law attached to it are so far distinct laws that the former contains nothing of the latter and the latter in its direct tenor contains nothing of the former. Yet by implication and that unnecessary one, the punatory does not involve and include the import of the simply imperative law to which it is appended. To say to the judge caused to be hanged whoever in due form of law is convicted of stealing is though not a direct yet as intelligible a way of intimating to men in general that they must not steal as to say to them directly do not steal and one sees how much more likely to be efficacious. It should not seem then that wherever a simply imperative law is to have a punatory or one appended to it, the former might be spared altogether. In which case, saving the exception which naturally should seem not likely to be a frequent one, of a law capable of answering its purpose without such an appendage, there should be no occasion in the whole body of the law for any other than punatory or in other words than penal laws. And this perhaps would be the case where it not for the necessity of a large quantity of matter of the expository kind of which we come now to speak. It will happen in the instance of many, probably of most, possibly of all, commands endued with the force of a public law that in the expression given to such a command it shall be necessary to have recourse to terms too complex in their signification to exhibit the requisite ideas without the assistance of a greater or less quantity of matter of an expository nature. Such terms, like the symbols used in algebraical notation, are rather substitutes and indexes to the terms capable of themselves of exhibiting the ideas in question than the real and immediate representatives of those ideas. Take for instance, the law, thou shalt not steal. Such a command were it to rest there could never sufficiently answer the purpose of a law. A word of so vague and unexplicit a meaning cannot otherwise perform this office than by giving a general intimation of a variety of propositions, each requiring to convey it to the apprehension a more particular and ample assemblage of terms. Stealing, for example, according to a definition not accurate enough for use, but sufficiently so for the present purpose is the taking of a thing which is another's by one who has no title so to do and is conscious of his having none. Even after this exposition, supposing it a correct one, can the law be regarded as completely expressed? Certainly not. For what is meant by a man's having a title to take a thing? To be complete, the law must have exhibited amongst a multitude of other things two catalogs, the one of events to which it has given the quality of conferring title in such a case, the other of the events to which it has given the quality of taking it away. What follows? That for a man to have stolen, for a man to have had no title to what he took, either no one of the articles contained in the first of those lists must have happened in his favor, or if there has, someone of the number of those contained in the second must have happened to his prejudice. Such then is the nature of a general law that while the imperative part of it, the punctum saliens as it may be termed of this artificial body, shall not take up above two or three words, its expository appendage, without which that imperative part could not rightly perform its office, may occupy a considerable volume. But this may equally be the case with a private order given in a family. Take for instance, one from a bookseller to his foreman. Remove from this shop to my new one my whole stock according to this printed catalogue. Remove from this shop to my new one my whole stock is the imperative matter of this order. The catalogue referred to contains the expository appendage. The same mass of expository matter may serve in common for, may appertain in common to, many commands, many masses of imperative matter. Thus, amongst other things, the catalogue of collative and ablative events, with respect to titles above spoken of, will belong in common to all or most of the laws constitutive of the various offenses against property. Thus in mathematical diagrams, one in the same base shall serve for a whole cluster of triangles. Such expository matter, being of a complexion so different from the imperative, it would be no wonder if the connection of the former with the latter should escape the observation, which indeed is perhaps pretty generally the case. And so long as any massive legislative matter presents itself, which is not itself imperative or the contrary, or of which the connection with matter of one of those two descriptions is not apprehended, so long and so far the truth of the proposition that every law is a command or its opposite may remain unsuspected or appear questionable. So long also may the incompleteness of the greater part of those masses of legislative matter, which were the complexion of complete laws upon the face of them, also the method to be taken for rendering them really complete remain undiscovered. A circumstance that will naturally contribute to increase the difficulty of the discovery is the great variety of ways in which the imporation of a law may be conveyed. The great variety of forms which the imperative part of a law may indiscriminately assume, some more directly, some less directly expressive of the imperative quality.  Whoso stealeth shall be punished so and so. If any man steal, he shall be punished so and so. Stealing is where a man does so and so. The punishment for stealing is so and so. To judges, so and so named, and so and so constituted, belong the cognizance of such and such offenses, be delicate stealing, and so on. These are but part of a multitude of forms of words in any of which the command by which stealing is prohibited might equally be couched. And it is manifest to what a degree in some of them the imperative quality is clouded and concealed from ordinary apprehension. After this explanation, a general proposition or two that may be laid down may help to afford some little insight into the structure and contents of a complete body of laws. So many different sorts of offenses created. So many different laws of the coercive kind. So many exceptions taken out of the descriptions of those offenses. So many laws of the discoercive kind. To class offenses, as have been attempted to be done in the preceding chapter, it is therefore to class laws. To exhibit a complete catalog of all the offenses created by law, including the whole mass of expository matter necessary for fixing and exhibiting the import of the terms contained in the several laws by which those offenses are respectively created would be to exhibit a complete collection of the laws in force, in a word, a complete body of law, a panameon, if so it might be termed. From the obscurity in which the limits of a law and the distinction betwixt a law of the civil or simply imperative kind and a punatory law of our naturally involved results the obscurity of the limits betwixt a civil and a penal code betwixt a civil branch of the law and the penal. The question, what parts of the total mass of legislative matter belong to the civil branch and what the penal supposes that diverse political states or at least that some one such state are to be found having as well a civil code as a penal code, each of them complete in its kind and marked out by certain limits. But no one such state has ever yet existed. To put a question to which a true answer can be given, we must substitute the foregoing question, some such a one as that which follows. Suppose two masses of legislative matter be drawn up at this time of day, the one under the name of a civil code, the other of a penal code, each meant to be complete in its kind. In what general way is it natural to suppose that the different sorts of matter as above distinguished would be distributed between them? To this question, the following answer seems likely to come as near as any other to the truth. The civil code would not consist of a collection of civil laws each complete in itself as well as clear of all penal ones. Neither would the penal code, since we have seen that it could not, consist of a collection of punitive laws each not only complete in itself but clear of all civil ones. But the civil code would consist chiefly of mere masses of expository matter. The imperative matter to which those masses of expository matter respectively appertained would be found not in that same code, not in the civil code nor in a pure state free from all admixture of punatory laws but in the penal code in a state of combination involved in manner as above explained in so many correspondent punatory laws. The penal code then would consist principally of punitive laws involving the imperative matter of the whole number of civil laws along with which would probably also be found various masses of expository matter appertaining not the civil but to the punatory laws. The body of penal law enacted by Empress Queen Maria Theresa agrees pretty well with this account. The mass of legislative matter published in French as well as German under the auspices of Frederick II of Prussia by the name of Code Frederick but never established with force of law appears for example to be almost wholly composed of masses of expository matter the relation of which to any imperative matter appears to have been but very imperfectly apprehended. In that enormous mass of confusion and inconsistency the ancient Roman or as it is termed by way of eminence the civil law, the imperative matter and even all traces of the imperative character seem at last to have been smothered in the expository. Esto had been the language of primeval simplicity Esto had been the language of the 12 tables. By the time of Justinian so thick was the darkness raised by clouds of commentators the penal law had been crammed into an odd corner of the civil. The whole catalogue of offences and even of crimes lay buried under a heap of obligations. Will was hid in opinion and the original Esto had transformed itself into Widditer in the mouths of even the most despotic sovereigns. Among the barbarous nations that grew up out of the ruins of the Roman Empire law emerging from under the mountain of expository rubbish re-assumed for a while the language of command and then she had simplicity at least if nothing else to recommend her. Besides the civil and the penal every complete body of law must contain a third branch, the constitutional. The constitutional branch is chiefly employed in conferring on particular classes of persons, powers to be exercised for the good the whole society or of considerable parts of it and prescribing duties to the persons invested with those powers. The powers are principally constituted in the first instance by dis-coercive or permissive laws operating as exceptions to certain laws of the coercive or imperative kind. Instance, a tax gatherer as such may on such and such an occasion take such and such things without any other title. The duties are created by imperative laws addressed to the persons on whom the powers are conferred. Instance, on such and such an occasion such and such a tax gatherer shall take such and such things. Such and such a judge shall, in such and such a case, cause persons so and so offending to be hanged. The parts which perform the function of indicating who the individuals are, who in every case shall be considered as belonging to those classes, have neither a permissive complexion nor an imperative. They are so many masses of expository matter appertaining in common to all laws into the texture of which the names of those classes of persons have occasion to be inserted. Instance, imperative matter. Let the judge cause whoever in due course of law is convicted of stealing to be hanged. Nature of the expository matter. Who is the person meant by the word judge? He who has been invested with that office in such a manner and in respect of whom no event has happened of the number of those to which the effect is given of reducing him to the condition of one divested of that office. Thus it is that one in the same law, one in the same command will have its matter divided. Not only between the two great codes or main branches of the whole body of the laws, the civil and the penal, but amongst three such branches, the civil, the penal and the constitutional. In countries where a great part of the law exists in no other shape than that of which in England is called common law, but might be more expressively termed judiciary, there must be a great multitude of laws, the import of which cannot be sufficiently made out of practice without referring to this common law for more or less the expository matter belonging to them. Thus in England, the exposition of the word title, that basis of the whole fabric of the laws of property is nowhere else to be found. And as uncertainty is of the very essence of every particle of law so denominated, for the instant it is clothed in a certain authoritative form of words it changes its nature and passes over to the other denomination. Hence it is that a great part of the laws in being in such countries remain uncertain and incomplete. What are those countries? To this hour, everyone on the surface of the globe. Had the science of architecture no fixed nomenclature belonging to it, were there no settled names for distinguishing the different sorts of buildings nor the different parts of the same building from each other, what would it be? It would be what the science of legislation considered with respect to its form remains at present. Were there no architects who could distinguish a dwelling house from a barn or a sidewalk from a ceiling, what would architects be? They would be what all legislators are at present. From this very slight and imperfect sketch may be collected not an answer to the questions in the text but an intimation and that but an imperfect one of the course to be taken for giving such an answer and at any rate some idea of the difficulty as well as the necessity of the task. If it were thought necessary to recur to experience for proofs of this difficulty and this necessity, they need not be long wanting. Take for instance, so many well-meant endeavors on the part of popular bodies and so many well-meant recommendations in ingenious books to restrain supreme representative assemblies from making laws in such and such cases or to such and such an effect. Such laws to answer the intended purpose require a perfect mastery in the science of law considered in respect of its form in the sort of anatomy spoken of in the preface to this work. But a perfect or even a moderate insight into that science would prevent it being couched in those loose and inadequate terms in which they may be observed so frequently to be conceived as a perfect acquaintance with the dictates of utility on that head would in many, if not in most of those instances, discount sell the attempt. Keep to the letter and in attempting to prevent the making of bad laws you will find them prohibiting the making of the most necessary laws, perhaps even of all laws. Quit the letter and they express no more than if each man were to say, your law shall become ipso facto void as often as they contain anything which is not to my mind. Of such unhappy attempts, examples may be met with in the legislation of many nations, but in none more frequently than in that newly created nation, one of the most enlightened, if not the most enlightened at this day on the globe. Take, for instance, the Declaration of Rights enacted by the state of North Carolina in convention in or about the month of September, 1788 and said to be copied with a small exception from one in like manner enacted by the state of Virginia. The following, to go no farther, is the first and fundamental article that there are certain natural rights of which men, when they form a social compact cannot deprive or divest their posterity, among which are the enjoyment of life and liberty with the means of acquiring, possessing and protecting property and pursuing and obtaining happiness and safety. Not to dwell on the oversight of confining to posterity the benefit of the rights thus declared, what follows? That as against those whom the protection thus meant to be afforded includes every law or other order divesting a man of the enjoyment of life or liberty is void. Therefore, this is the case amongst others with every coercive law. Therefore, as against the persons thus protected every order, for example, to pay money on the score of taxation or of debt from individual to individual or otherwise is void. For the effect of it, if complied with is to deprive and divest him pro tanto of the enjoyment of liberty, we delicate the liberty of paying or not paying as he thinks proper. Not to mention the species opposed to imprisonment in the event of such a mode of coercions being resorted to. Likewise of property, which is itself a means of acquiring, possessing and protecting property and of pursuing and obtaining happiness and safety.  For the necessary effect of such an order is to deprive some of them of the enjoyment of life. The above mentioned consequences may suffice for examples amongst an endless train of similar ones.  May we now add, and in a law. End of chapter 17. End of an introduction to the principles of morals and legislation by Jeremy Bentham.


OK, it's 25, so we're going to start pretty soon. Get settled. So I hope everyone had a good lunch, and I hope you still have some energy for some codec-related adventures. So what I'll be talking about today is integrating hardware-accelerated video decoding with the Linux-based display stack. So who am I to tell you about this? I'm Paul Kschalkowski. I work at a company called Butlin, and we provide all kinds of services, development services, around embedded Linux in any form. So you can contact us for development, consulting, but we also do training, especially on-site. So if you have a team of engineers you need trained up on those topics, you can call us. And we try to work with a strong open-source focus. So all the things we do, we release in the free software licenses and so on. So this got me to contribute to a number of different aspects, especially the Linux kernel. So the one I'll be talking about today, especially, is the Cedrus VPU driver, but I also worked on the DRM driver for all-winner SOCs, which I will also be talking about a bit. And most recently, I developed a new display and rendering graphics with Linux training. So this is something we released publicly. It's about 200 slides. So if you're interested in graphics, you can check that out. It's released under a Creative Commons license. So you can share it with everyone and so on. And the office I live in is based in Toulouse, south-west of France, so not so far from here. So what's the actual purpose of this talk? A few things. First, I would like to present the kind of specific use case I had to deal with in terms of video decoding. So I will be presenting kind of some basics about that, how Linux kind of deal with it, and what's specific to our hardware and our use case, which is about all-winner SOCs. All of this is interesting, but we actually need these different components, the video parts, the display parts to kind of communicate together. So I will also be talking a bit about a pipeline, video pipeline integration over this talk. And this talk is kind of going to feel like a rant, because not many things worked out. And the purpose is also kind of to show you where things went wrong and the issues that I had, especially user space APIs and so on, so that if you kind of have to deal with the same situation, you kind of know where the blockers are and what the current situation of the whole thing is. So yeah, it's going to feel like a rant, but hopefully it will be seen as constructive criticism. And well, things could be worse. They definitely could be worse. So let's look on the bright side of things. So TLDR, it's all about pipelines, like going from video decoding to actually displaying frames. So I talked about video decoding. What is that exactly?  Like that's a huge size, huge files, if you need it to represent all the sequence of pictures as actual arrays of pixels. So in order to make this more bearable, we just compress these images and this series of images that make video files. So how does this kind of compression happen? So there are basically different layers of compressions that are happening. So this is kind of to give you some idea of what a codec is actually made of, what a video codec is actually all about in terms of compression and reducing data size. So one of the first things we can do is color space compression. So the idea is that we are not going to represent colors in terms of red, green, and blue. But instead, we're going to use a different color space called YUV. And this one is interesting because we separate the luminance component, which is kind of a grayscale representation of the image. We separate that from the colors. And the thing is the human eye actually has more sensitivity to luminosity compared to colors. So we can actually use the same color for multiple pixels but still have a single luminosity value for each pixel. So this gives us kind of a very precise luminosity to the picture. But the colors are less precise. But the human eye doesn't actually notice. So thanks to that, we can have one color sample for each block of four pixels, for instance. And that would look just as good for the human eye. So this already gives us some sort of reduction in terms of how we represent pictures. Then there is a second method that is used. This is basically what makes JPEG. This is spatial compression. So we're going to use a translation to frequency space a bit like it exists for audio files. The same thing happens in 2D for images. So in the case of codec, usually it's a discrete cosine transform. This brings us to frequency space. And once we're in frequency space, we can basically remove the high frequencies of the image that are also less noticeable to the eye. And this is kind of what gives the quality of the image. So we can filter high frequencies. We have a slightly worse quality. But then we have a lot less elements to represent. So this is spatial compression. Then with videos, we have something extra compared to, for instance, JPEG, where it's only the color compression and spatial compression. With videos, we also have temporal compressions. So this means that we're going to represent pictures, not only as fixed pixel representations, but basically as diffs between the previews and all the following pictures. So we're basically encoding diffs of what changed in the picture to kind of generate new images. And this is very convenient, because if you look at pretty much any video, you'll see that the whole scene is not everything is changing on it at the same time. Usually you have one object or this stream. If I'm just moving my hand, all this area doesn't move. So you don't need to actually encode that. You can just encode the diff of my hand moving. And then you have a much more efficient representation of the picture in terms of data and space usage. But that's not all. There is a final step called entropy compression, which is basically the same as what you find in a zip or any sort of archive compression format, where we basically just assign symbols to the bytes that are the most frequent in the representation of the image. And so we give those a smaller weight than those that are less common. And thanks to that, we can kind of reduce, once again, the size of the image. So what video codec implements are basically these four different layers of compression. So in order to properly decode those encoded picture, we need some metadata to kind of provide the details of how it was encoded and so how it's supposed to be decoded. So that's a number of different parameters that you need to actually carry on along with the compressed data itself. So this is called the metadata. And once we get both and we group them together, we have something called a bitstream. And that bitstream can actually also have other things like audio. If you have a video, it's nice to have audio as well. You can also have subtitles. And so the group of all of that is what we call a video in a container format. So the container contains the video stream, which is a bitstream with metadata and compressed data. But we can also find audio that might also be compressed and subtitles. So in the end, we have kind of this representation where we have a container with the encoded video and so on. So with these techniques, we have like reasonably small files that we can share with a long time of video. And so that's great. So that's for the video decoding part. Now, hardware video decoding is basically about having specialized hardware to kind of offload this operation of decoding, which is what I'm interested in today. The last thing is we work on embedded systems at Butlin. We're interested in that. And those don't really have a lot of CPU power to fire. So video decoding should be offloaded if possible. So there are basically two major types of hardware implementations to do video decoding. The first one is called stateful, where it's basically that you have a microcontroller attached to the actual hardware. And this microcontroller will do all the metadata parsing, all the engine configuration. It will also keep track of the different buffers that you need to do the temporal compression thing, where you calculate diffs between your pictures. So you kind of need to keep track which images you need to apply the diff to. So in the case of stateful decoders, this is done by a microcontroller. In the case of stateless decoders, this is up to the CPU, basically. So the CPU will still have to configure, like registers, to specify how the video was encoded so that it can be properly decoded. So in Linux, these hardware code, oh, sorry, there we go. In Linux, these are supported in the V4L2 framework. It's pretty new. It's something lots of people have been working on over the past year, or actually a bit longer. But yeah, it really came around last year. This is thanks to the V4L2 M2M framework, which allows supporting the stateful codecs. And we add to add a new API called the Request API for the stateless ones. And this Request API is basically a way to synchronize the compressed data buffers and the metadata that we provide directly to the kernel driver so that it can configure the registers. And so once we have all of this, we have our nice V4L2 drivers that we can use to decode videos. We get decoded pictures out, right? And so what do we do with these pictures? Well, we can either read them from the CPU using the well-known M-Map syscall. We just access the data. But more interestingly, if you want to pipe that to another device, for instance, your display engine, because you actually want to display the decoded video on a screen, then we can use a mechanism called DMABuff, which is just a way to provide a reference to the memory area where the decoded picture is and pass that between different drivers. So we don't have to do an extra copy. And in the end, the kind of expected result of all this is to be able to integrate a decoded video like here with a user interface on a desktop environment. So yeah, that's kind of the goal here. But actually in order to achieve that, there are lots of things to take in consideration. I mentioned that the pictures are not coded in RGB, but instead in a YUV color space. So we usually need to convert that YUV format, that YUV color space to RGB before it can be composed with the rest of the things to display. So that is one thing that is needed. We might also need to do scaling. For instance, if you want to display the video full screen, you have to have some way to kind of, yeah, do the scaling operation. And those are actually quite calculation intensive because you need to go over each pixel and apply an operation to do that. So doing this on the CPU is not very efficient. But thankfully, we have hardware to help with that as well. And this is where the display engine comes in. The display engine is not only responsible for sending pixels through your HDMI link or whatever. It also has this sort of scaling and color space conversion and composition abilities. So we should be able to use that. There's also the GPU that can pretty much do anything. So yeah, it's totally able to take our decoded data and do all the composition, scaling we need, and produce some final image that we can display on the actual monitor. And if we want to do that efficiently, what we really have to avoid is buffer copies. If you're going to use different hardware blocks to do different operations on our decoded buffers, we need to make sure that we don't copy the data between one device and another. They should always use a common area of memory to, as a sync and source for the next block. So that's for kind of the general idea. Now I'm going to present some specifics on Arduino platforms, which is what I actually worked on. So Arduino platforms, those are two examples of boards from our friends at Alimex and Libre Computer. So basically, Arduino SoCs we find in those kinds of single board computers. They are also cheap Chinese tablets that use them. Also random products out there, industrial ones included that yeah, might do pretty much anything. It's embedded, so you never know anything is embedded really, so yeah. The situation on this specific kind of family of platform, there are different Arduino SoCs, different platforms. The situation is the following. So we have this stateless video decoder hardware that we can use. It supports a whole bunch of codecs. We only focused on MPEG-2, H264 and H265 so far. But of course, we hope that soon the other ones will be supported. We have a display engine that can take multiple inputs and do composition of them. So that's quite useful for what we want to do. And we also have a GPU, which is a Mali 400 or 450 in those SoCs. But they also come with weird constraints that kind of got in the way and this is kind of the root of the problems we had. So first off, on an old platform, there is only a specific part of memory that the VPU can access. So we actually had to declare a reserved area of memory in that region so that we could use that to produce the final decoded pictures and also use that to provide the input bit stream. That's the actual contents from the video. But it's not only that, they also have this weird tiled output format. So it's basically that instead of pixels being laid out line by line, they are laid out in a different format. And this makes life a bit more complicated. Thankfully, the new ones didn't have any of those kind of constraints so we can use any memory. It still needs to be contiguous because we don't have any MMU attached. So we still need contiguous memory but at least we don't have this sort of limitation on the area that we can use. And there's no tiling that we have to use in particular. So just to give some visual interpretation of the tiling, this is called Linear of Raster Sky and Order. So this is the way pixels are laid out usually in memory. And on the right side, we see what our VPU actually produces. So the arrows are the increasing memory addresses. So you can see that in this case, it's not like full line, then another full line, then another full line, it's just a small block, then another small block, then another small block. And so it means that if we're going to use this data for another device, we need that other device to kind of understand this representation and be able to deal with it. So in order to add support for this, what I did at Butlin was to first add support for that on the display part. And of course, we also worked on the VPU driver itself. So the VPU driver is called Cedrus. It's in staging currently. It was merged in 5.1, Linux 5.1 with MPEG-2. And since then we added H264 in 5.3 and H265, which was just merged last week or so, and it will make it to 5.5. So that's for the V4L2 side, for the decoder side, and for the display side, we actually had to add a modifier, which is basically the indication that the scan order is not linear or raster, but that instead it's our weird custom tiled scan order. So the way to represent that in the DRM subsystem that deals with display is to specify a modifier. And we'll see that this modifier is actually quite important. Similarly, on V4L2 we also introduced a custom pixel format, tiled and V12, which is also a way to represent our weird tiled format. So basically we have all the things we need in Canon Space. We have drivers that can fit with the constraints that we have. And we wrote a test utility, V4L2 request test that we used to kind of bring up both sides. And we also have a VA API backend. So VA API is just a way to interact with the video decoding side from user space. So VA API will directly call the V4L2 driver side. Canon side. So how did we implement this actually? The first thing we did was to go with a full bare metal setup, right? We decided let's just write things from scratch. And so that's the test utility I mentioned just now, V4L2 request test. With that one we just implemented all the IOC tools to access the V4L2 side for decoding and the DRM side for display. And of course we used the DMABUFF API to be able to avoid copies. So this is called zero copy. DMABUFF basically exports an identifier of sorts of the memory area as a file descriptor. So we just pass this file descriptor between the two APIs, V4L2 and DRM. And in our case, this is the IOC tool we call on the V4L2 side. So basically we create a buffer. We provide it to the driver. It does the decoding and it gives us a destination buffer. And we call this XBUFF IOC tool on that destination buffer. Then we get file descriptor. And that file descriptor we can import into DRM to basically tell the display engine, oh, this is the memory you should be displaying. Yeah. So on the DRM side, the IOC tool is a prime FD2 handle. So it's just a way to import it in the framework. And what we implemented on the DRM side is also all the CSEs, so color space conversion, all the scaling and all the composition using DRM planes. So DRM planes are just basically overlays that you can add on top of your main frame buffer, I would say, your main picture.  And the display hardware itself will do the composition of only scratching that into one image that is finally sent to the display, to the HDMI link. So this is very useful because doing this all in hardware is very efficient. And so of course we wanted to use that. So that's what we implemented. The resulting pipeline is fairly simple. Yeah, we just take from V4L2 and give it to DRM. We use the planes. And things are quite fast. And there is no copy. There is no software-based or CPU-based operation. So this worked pretty well. But then we tried to use Xorg. And that's where it started getting a bit more complicated. First off, we went for a GPU-less setup because we didn't see any particular reason to involve the GPU in the mix at first. And so the idea was, can we use a DRM plane under X? Can we actually ask the display hardware to do that composition through the X API? And truth is, X has lots of limitations regarding to that. The X protocol only knows about RGB formats. And as I've mentioned, our decoded frame is in YUV, not RGB. So that didn't fit really well. But there are some extensions to allow that. The first one is quite old. It's called XV. And it supports YUV inputs. But the problem is, first off, you have to implement hardware-specific part of X, which is called the DDX for device-dependent X. You have to write a specific implementation for your device on that side just to be able to kind of pick up the YUV data and push it to a DRM plane. So yeah, that meant some work required. But it also doesn't support any zero-copy mechanism. So you also have to do a copy of the destination buffer of the VPU into another buffer to provide to XV. And another problem with it is that it has serious synchronization issues. So basically, if you use XV, say with VLC or any player, and you kind of move the window around, you'll see a bit of a lag of just the video contents not following Pixel Perfect, but having like a tenth of a second lag or even more. So it's really annoying, and it doesn't work great. Then there is also something called DRI3, which is a pretty new extension to Xorg. And it's supposed to kind of cover some of the downsides that existed with X in that regard. So one thing it does is DMABuff import. So we can actually pass the DMABuff descriptor to that API and then hope that the X server itself will properly import it with zero-copy and be able to display it on the actual display. But that one didn't support modifiers. So for the old generation platforms where we have to specify the modifier because it's not a linear scan order. It's really our tiled format. That just couldn't work out. And there's also the fact that currently DRI3 is only supported in Glamour, which is basically the GPU-accelerated X kind of backend. And so if we're not going to use a GPU, basically we cannot use this API. And even then, it doesn't really give us access to using the DRM planes. It's basically only allows importing that buffer with zero-copy to the GPU, not to the display engine. So that's not really something you could use. So in the end, for X, we kind of resorted to a fully software-based pipeline. So that wasn't great. We still needed to do entiling to be able to provide the buffer to X. So we had to implement that. It was actually already implemented, but we had to kind of bring it in to do this entiling operation of putting the pixels back in the expected order. We did that with Neon, so it's on ARM, so it's SIMD. It's much faster than doing it purely CPU-based implementation. But then we also needed to care about the color space conversion, and also scaling, and also composition with the rest of the buffer of the display. And that also involved copies. So we couldn't get a zero-copy setup with that. This led to a very suboptimal kind of setup. You can see the pipeline here. So in VA API, that's basically where we did the entiling step fully in software. That still was surprisingly good. We managed to play 720p videos at 30 frames per second with purely software composition. And the real bottleneck was actually scaling. Like, as soon as scaling is involved, then the performance would be like 10 frames per second or less, which is really not good enough for watching a video. So then we tried to bring in the GPU in the mix, because, hey, it's supposed to be able to do anything. So yeah, it might as well do that. Using the GPU, in our case, means using a non-free blob that ARM provides. And the way to kind of integrate that with Xorg is to use a device-dependent X thing called ARMsoc. And that one doesn't actually accelerate composition. That's one thing it's not going to help with. But it still allows us to kind of use the GPU with OpenGL to accelerate operations. So first, what I tried to do was to import the decoded buffer into the GPU using a YUV import, and then do entiling on the GPU. But that didn't work out, because the non-free blob doesn't support YUV import. So no luck. Then I tried importing it as a 8-bit component. So just reading itch bytes separately and doing the calculation with that on the GPU. It worked well on Intel. I actually developed an entire shader on Intel. It was pretty fine. But when I moved it to the Mali, it didn't work. For a reason, I still don't fully understand. I found some bug reports of people having the same issue, but no one really knows what's going on. Why that doesn't work. And zero copy was also impossible for the 8-bit direct access. So the bottom line is we basically didn't manage to use the GPU for anything, and it was just a waste of time. Maybe if that was on a blob, but instead of free software implementation, maybe we could have kind of dig into it and see, OK, are there actual limitations on the hardware, or is it just a lack of documentation or lack of implementation? Yeah, that could have helped. So nowadays, we have a free driver for the Mali, which is called LIMA. It's still kind of experimental, but it's making good progress. So maybe it would work better with that. We didn't check again by the time LIMA got merged. So another thing is Wayland, right? Like, Wayland is awesome. No one knows that. So if you don't already use Wayland, please do. It's the future. But we didn't really have time to investigate it at the time of this project. But I kind of still wanted to dig around and see, like, would it be feasible? And Wayland has kind of a conflicted relationship with GOM planes. So it doesn't expose them to applications. So an application doesn't know that the system has a plane available to do composition. I mean, at least applications don't know that. So if you have a video player, it cannot say, oh, I want to use a plane for that. Instead, it's considered to be an implementation detail of the Wayland compositor. So the compositor might actually end up using a plane if it can, and if things work out. But there is no guarantee about that. So that's something to kind of remember. Now, as for zero copy, there is a Wayland protocol extension that supports that, called Linux DMA buff. And it has a dedicated interface as well. But that one does support modifiers. So it's cool for our old generation hardware, but we need to pass that information along up to the display engine. But it turns out that the way it's implemented, because the rendering backend is GPU-based if you want to do a DMA buff. Then you also need the GPU to know about this modifier and to support it, which was definitely not the case for us because the GPU blobs we have are not, they don't support the modifier, and they don't support DMA buff import for that. So odds are, it wouldn't have worked with Wayland either for us. Now, the kind of final pipeline that we worked on and which actually worked, which actually provide nice results, is using Kodi. So for those of you who don't know what Kodi is, it's a media center implementation, a really nice one, fully free software, as far as I know. It's really nice. And it uses OpenGL to do UI rendering, and this uses the GBM EGL backend. So if you're familiar with OpenGL bits, EGL is the kind of window integration, window manager integration for OpenGL, and GBM basically allows using DRM directly for that. So it means you don't need a display server. And it's nice because then the application, Kodi, is directly in touch with the DRM subsystem, and so it can actually use planes, right? Planes are not hidden away by a display server, but they are actually directly usable. So what worked was to just use DRM planes in Kodi. There was already a fair support for that. I had to tweak a few bits to really get it to work. Also had to integrate our VA API plugin to retrieve the frames and do the zero copy export. But in the end it worked out, and so we were able to get Kodi up and running with our hardware, with our modifier, with everything we needed. So yeah, that's kind of the only non-test-based pipeline we got working. So it's almost time for this talk. Some general takeaway, or at least things that I've noticed and kind of wanted to share. First off, don't expect that DRM planes will be made available to your application. At best, they will be available to the Wayland compositor, but the Wayland compositor is not going to expose that in any way, so don't expect that you can use them directly from user space. I'm not gonna say whether this is a good or bad decision. It certainly makes sense in the context of Wayland. It certainly makes sense, but that's the way it is. Also Modifier Support is something pretty new in user space, only the Wayland extension did have a notion of modifiers. Basically any Xorg extension that you will find doesn't care about modifiers. So if your frame buffer is styled, basically you're screwed. Also not about the fact that most of the user space stack nowadays expects that you have OpenGL available to do everything, but in our case, using the GPU was really more a source of issues and headache rather than really something that solved an issue, because it was probably in part because it was a non-free, proprietary blob. Also because it didn't have the proper implementation for GMABuff and also that it had this weird limitation when I tried to import the decoded frame and that it just didn't work. So back to planes, there are actually some projects that exist to try and make the use of planes much easier. There's one called LibLift-off. It's still under discussion. It's a very new thing. I think Simon Serre is the person working on that. So if you're interested in actually exploiting planes and having a library that makes that easy, you can get in touch with that project. And more generally, there is a LibOutput project that kind of also aims to make display server implementations more robust by having a common code base. So LibOutput would help also use DRM planes in the context of compositors or things like this. So that's it for this presentation. I think we still have a few minutes for questions. So if you have any, feel free. And otherwise, thank you. No questions? Well, cheers. So does SOC have some sort of hardware scaler unit? So you wouldn't have to do the scaling in software? Yeah, so that one was integrated with the display engine. So when we use DRM planes, we have the opportunity to use the scaler that comes with it. So this is why we wanted to use DRM planes as much as possible. Because there are SOCs that have dedicated scalers. Did you ever look into that? And is it supported by Linux? So yes, there are some SOCs that have dedicated scalers. One example that comes to mind is the Rockchip RGA, which is exposed by a V4L2. So it's a memory to memory device. It actually uses the same API as the one we have for Codex. And that one, you can basically, yeah, give it any frame buffer as input and tell it to what it should scale, and then you'll get the scaled result as output. So that does exist, but I'm not aware of any projects kind of leveraging that from user space. So you have the kernel side is there, but yeah, I don't know if it's really used a lot. But, oh, it is used in JetStreamer, okay? Okay, and also it exists on Exynos and IMX6. So yeah, that does exist. Nice, thank you. Just anyone else? I don't bite, don't be shy. Yep, we have one over there. Thank you. So I think from my slightly limited experience with these things, I think the way they kind of expect you to use this a lot because a lot of these chipsets were actually designed for Android. And the way you do it there is by essentially using a GL extension, I think it's called external image, essentially abuse of the texture from the video surface. Is that something you kind of looked at? You know, like basically taking the memory from the VOD Coder and kind of piping it straight into a GL texture that you can then display like you would any other GL texture. That's the thing I tried to do and fail at doing. So that didn't work out for unknown reasons basically. Blob-related reasons. I'm sure that was just how it was supposed to be used, but okay. Thanks for that. So I'll just add to that. If you use some of the free software GPU drivers, you can directly import with DRM something something, the MA buffs into the GPU and get a GL texture out of it. Yeah, but that only works if the modifier is also supported by the GPU, which in our case was just not the case. The modifier was only supported by the VPU and DRM display, but not by the GPU. So this is another reason why the GPU didn't help solve the kind of pipeline issue. Okay, I think we're out of time, so thanks.