https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_pve_firewall

スキャンID：: 0afa4a48-74e3-4146-87cc-e043132284ae終了
送信済みURL：: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_pve_firewall
レポート終了日：: 2024年10月20日 23:39:37
リンク · 194件検出

ページから特定された発信リンク
リンク	テキスト
https://en.wikipedia.org/wiki/Hyper-converged_infrastructure	hyper-converged infrastructure
https://www.gnu.org/licenses/agpl-3.0.html	GNU Affero General Public License, version 3
https://forum.proxmox.com/	Proxmox VE Community Forum
http://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user	Proxmox VE User List
http://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel	Proxmox VE development discussion
https://proxmox.com/en/proxmox-virtual-environment/pricing	https://proxmox.com/en/proxmox-virtual-environment/pricing
https://bugzilla.proxmox.com	https://bugzilla.proxmox.com
https://corosync.github.io/corosync/	Corosync
https://kanaka.github.io/noVNC/	noVNC
https://ceph.com/	Ceph
JavaScript変数 · 4件検出

ページのウィンドウオブジェクトにロードされたグローバルのJavaScript変数は関数以外の場所で宣言された変数で、現在のスコープ内であればコードのどこからでもアクセス可能です
名前	規模
onbeforetoggle	object
documentPictureInPicture	object
onscrollend	object
asciidoc	object
コンソールログメッセージ · 0件検出

Webコンソールにログ記録されたメッセージ
HTML

未加工のHTMLページ本文
<!DOCTYPE html><html lang="en"><head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="generator" content="AsciiDoc 10.2.0">
<title>Proxmox VE Administration Guide</title>
<style type="text/css">
/* Shared CSS for AsciiDoc xhtml11 and html5 backends */

/* Default font. */
body {
  font-family: Georgia,serif;
}

/* Title font. */
h1, h2, h3, h4, h5, h6,
div.title, caption.title,
thead, p.table.header,
#toctitle,
#author, #revnumber, #revdate, #revremark,
#footer {
  font-family: Arial,Helvetica,sans-serif;
}

body {
  margin: 1em 5% 1em 5%;
}

a {
  color: blue;
  text-decoration: underline;
}
a:visited {
  color: fuchsia;
}

em {
  font-style: italic;
  color: navy;
}

strong {
  font-weight: bold;
  color: #083194;
}

h1, h2, h3, h4, h5, h6 {
  color: #527bbd;
  margin-top: 1.2em;
  margin-bottom: 0.5em;
  line-height: 1.3;
}

h1, h2, h3 {
  border-bottom: 2px solid silver;
}
h2 {
  padding-top: 0.5em;
}
h3 {
  float: left;
}
h3 + * {
  clear: left;
}
h5 {
  font-size: 1.0em;
}

div.sectionbody {
  margin-left: 0;
}

hr {
  border: 1px solid silver;
}

p {
  margin-top: 0.5em;
  margin-bottom: 0.5em;
}

ul, ol, li > p {
  margin-top: 0;
}
ul > li     { color: #aaa; }
ul > li > * { color: black; }

.monospaced, code, pre {
  font-family: "Courier New", Courier, monospace;
  font-size: inherit;
  color: navy;
  padding: 0;
  margin: 0;
}
pre {
  white-space: pre-wrap;
}

#author {
  color: #527bbd;
  font-weight: bold;
  font-size: 1.1em;
}
#email {
}
#revnumber, #revdate, #revremark {
}

#footer {
  font-size: small;
  border-top: 2px solid silver;
  padding-top: 0.5em;
  margin-top: 4.0em;
}
#footer-text {
  float: left;
  padding-bottom: 0.5em;
}
#footer-badges {
  float: right;
  padding-bottom: 0.5em;
}

#preamble {
  margin-top: 1.5em;
  margin-bottom: 1.5em;
}
div.imageblock, div.exampleblock, div.verseblock,
div.quoteblock, div.literalblock, div.listingblock, div.sidebarblock,
div.admonitionblock {
  margin-top: 1.0em;
  margin-bottom: 1.5em;
}
div.admonitionblock {
  margin-top: 2.0em;
  margin-bottom: 2.0em;
  margin-right: 10%;
  color: #606060;
}

div.content { /* Block element content. */
  padding: 0;
}

/* Block element titles. */
div.title, caption.title {
  color: #527bbd;
  font-weight: bold;
  text-align: left;
  margin-top: 1.0em;
  margin-bottom: 0.5em;
}
div.title + * {
  margin-top: 0;
}

td div.title:first-child {
  margin-top: 0.0em;
}
div.content div.title:first-child {
  margin-top: 0.0em;
}
div.content + div.title {
  margin-top: 0.0em;
}

div.sidebarblock > div.content {
  background: #ffffee;
  border: 1px solid #dddddd;
  border-left: 4px solid #f0f0f0;
  padding: 0.5em;
}

div.listingblock > div.content {
  border: 1px solid #dddddd;
  border-left: 5px solid #f0f0f0;
  background: #f8f8f8;
  padding: 0.5em;
}

div.quoteblock, div.verseblock {
  padding-left: 1.0em;
  margin-left: 1.0em;
  margin-right: 10%;
  border-left: 5px solid #f0f0f0;
  color: #888;
}

div.quoteblock > div.attribution {
  padding-top: 0.5em;
  text-align: right;
}

div.verseblock > pre.content {
  font-family: inherit;
  font-size: inherit;
}
div.verseblock > div.attribution {
  padding-top: 0.75em;
  text-align: left;
}
/* DEPRECATED: Pre version 8.2.7 verse style literal block. */
div.verseblock + div.attribution {
  text-align: left;
}

div.admonitionblock .icon {
  vertical-align: top;
  font-size: 1.1em;
  font-weight: bold;
  text-decoration: underline;
  color: #527bbd;
  padding-right: 0.5em;
}
div.admonitionblock td.content {
  padding-left: 0.5em;
  border-left: 3px solid #dddddd;
}

div.exampleblock > div.content {
  border-left: 3px solid #dddddd;
  padding-left: 0.5em;
}

div.imageblock div.content { padding-left: 0; }
span.image img { border-style: none; vertical-align: text-bottom; }
a.image:visited { color: white; }

dl {
  margin-top: 0.8em;
  margin-bottom: 0.8em;
}
dt {
  margin-top: 0.5em;
  margin-bottom: 0;
  font-style: normal;
  color: navy;
}
dd > *:first-child {
  margin-top: 0.1em;
}

ul, ol {
    list-style-position: outside;
}
ol.arabic {
  list-style-type: decimal;
}
ol.loweralpha {
  list-style-type: lower-alpha;
}
ol.upperalpha {
  list-style-type: upper-alpha;
}
ol.lowerroman {
  list-style-type: lower-roman;
}
ol.upperroman {
  list-style-type: upper-roman;
}

div.compact ul, div.compact ol,
div.compact p, div.compact p,
div.compact div, div.compact div {
  margin-top: 0.1em;
  margin-bottom: 0.1em;
}

tfoot {
  font-weight: bold;
}
td > div.verse {
  white-space: pre;
}

div.hdlist {
  margin-top: 0.8em;
  margin-bottom: 0.8em;
}
div.hdlist tr {
  padding-bottom: 15px;
}
dt.hdlist1.strong, td.hdlist1.strong {
  font-weight: bold;
}
td.hdlist1 {
  vertical-align: top;
  font-style: normal;
  padding-right: 0.8em;
  color: navy;
}
td.hdlist2 {
  vertical-align: top;
}
div.hdlist.compact tr {
  margin: 0;
  padding-bottom: 0;
}

.comment {
  background: yellow;
}

.footnote, .footnoteref {
  font-size: 0.8em;
}

span.footnote, span.footnoteref {
  vertical-align: super;
}

#footnotes {
  margin: 20px 0 20px 0;
  padding: 7px 0 0 0;
}

#footnotes div.footnote {
  margin: 0 0 5px 0;
}

#footnotes hr {
  border: none;
  border-top: 1px solid silver;
  height: 1px;
  text-align: left;
  margin-left: 0;
  width: 20%;
  min-width: 100px;
}

div.colist td {
  padding-right: 0.5em;
  padding-bottom: 0.3em;
  vertical-align: top;
}
div.colist td img {
  margin-top: 0.3em;
}

@media print {
  #footer-badges { display: none; }
}

#toc {
  margin-bottom: 2.5em;
}

#toctitle {
  color: #527bbd;
  font-size: 1.1em;
  font-weight: bold;
  margin-top: 1.0em;
  margin-bottom: 0.1em;
}

div.toclevel0, div.toclevel1, div.toclevel2, div.toclevel3, div.toclevel4 {
  margin-top: 0;
  margin-bottom: 0;
}
div.toclevel2 {
  margin-left: 2em;
  font-size: 0.9em;
}
div.toclevel3 {
  margin-left: 4em;
  font-size: 0.9em;
}
div.toclevel4 {
  margin-left: 6em;
  font-size: 0.9em;
}

span.aqua { color: aqua; }
span.black { color: black; }
span.blue { color: blue; }
span.fuchsia { color: fuchsia; }
span.gray { color: gray; }
span.green { color: green; }
span.lime { color: lime; }
span.maroon { color: maroon; }
span.navy { color: navy; }
span.olive { color: olive; }
span.purple { color: purple; }
span.red { color: red; }
span.silver { color: silver; }
span.teal { color: teal; }
span.white { color: white; }
span.yellow { color: yellow; }

span.aqua-background { background: aqua; }
span.black-background { background: black; }
span.blue-background { background: blue; }
span.fuchsia-background { background: fuchsia; }
span.gray-background { background: gray; }
span.green-background { background: green; }
span.lime-background { background: lime; }
span.maroon-background { background: maroon; }
span.navy-background { background: navy; }
span.olive-background { background: olive; }
span.purple-background { background: purple; }
span.red-background { background: red; }
span.silver-background { background: silver; }
span.teal-background { background: teal; }
span.white-background { background: white; }
span.yellow-background { background: yellow; }

span.big { font-size: 2em; }
span.small { font-size: 0.6em; }

span.underline { text-decoration: underline; }
span.overline { text-decoration: overline; }
span.line-through { text-decoration: line-through; }

div.unbreakable { page-break-inside: avoid; }


/*
 * xhtml11 specific
 *
 * */

div.tableblock {
  margin-top: 1.0em;
  margin-bottom: 1.5em;
}
div.tableblock > table {
  border: 3px solid #527bbd;
}
thead, p.table.header {
  font-weight: bold;
  color: #527bbd;
}
p.table {
  margin-top: 0;
}
/* Because the table frame attribute is overridden by CSS in most browsers. */
div.tableblock > table[frame="void"] {
  border-style: none;
}
div.tableblock > table[frame="hsides"] {
  border-left-style: none;
  border-right-style: none;
}
div.tableblock > table[frame="vsides"] {
  border-top-style: none;
  border-bottom-style: none;
}


/*
 * html5 specific
 *
 * */

table.tableblock {
  margin-top: 1.0em;
  margin-bottom: 1.5em;
}
thead, p.tableblock.header {
  font-weight: bold;
  color: #527bbd;
}
p.tableblock {
  margin-top: 0;
}
table.tableblock {
  border-width: 3px;
  border-spacing: 0px;
  border-style: solid;
  border-color: #527bbd;
  border-collapse: collapse;
}
th.tableblock, td.tableblock {
  border-width: 1px;
  padding: 4px;
  border-style: solid;
  border-color: #527bbd;
}

table.tableblock.frame-topbot {
  border-left-style: hidden;
  border-right-style: hidden;
}
table.tableblock.frame-sides {
  border-top-style: hidden;
  border-bottom-style: hidden;
}
table.tableblock.frame-none {
  border-style: hidden;
}

th.tableblock.halign-left, td.tableblock.halign-left {
  text-align: left;
}
th.tableblock.halign-center, td.tableblock.halign-center {
  text-align: center;
}
th.tableblock.halign-right, td.tableblock.halign-right {
  text-align: right;
}

th.tableblock.valign-top, td.tableblock.valign-top {
  vertical-align: top;
}
th.tableblock.valign-middle, td.tableblock.valign-middle {
  vertical-align: middle;
}
th.tableblock.valign-bottom, td.tableblock.valign-bottom {
  vertical-align: bottom;
}


/*
 * manpage specific
 *
 * */

body.manpage h1 {
  padding-top: 0.5em;
  padding-bottom: 0.5em;
  border-top: 2px solid silver;
  border-bottom: 2px solid silver;
}
body.manpage h2 {
  border-style: none;
}
body.manpage div.sectionbody {
  margin-left: 3em;
}

@media print {
  body.manpage div#toc { display: none; }
}


@media screen {
  body {
    max-width: 50em; /* approximately 80 characters wide */
    margin-left: 16em;
  }

  #toc {
    position: fixed;
    top: 0;
    left: 0;
    bottom: 0;
    width: 13em;
    padding: 0.5em;
    padding-bottom: 1.5em;
    margin: 0;
    overflow: auto;
    border-right: 3px solid #f8f8f8;
    background-color: white;
  }

  #toc .toclevel1 {
    margin-top: 0.5em;
  }

  #toc .toclevel2 {
    margin-top: 0.25em;
    display: list-item;
    color: #aaaaaa;
  }

  #toctitle {
    margin-top: 0.5em;
  }
}
</style>

<style type="text/css">
div #toc {
    width: 19em;
    font-family: sans;
    border-right: 3px solid gainsboro;
}
.book, .article {
    margin-left: 22em
}
div #toc a:link, div #toc a:visited {
    color: black;
    text-decoration: none;
}
div #toc a:hover {
    text-decoration: underline;
}
div .toclevel1 {
    font-size: 1.1em;
    margin-bottom: 0.3em;
}

.monospaced, code, pre {
    overflow-wrap: break-word;
}
img {
    max-width: 100%;
}

@media only screen and (min-width: 85em) {
div #toc {
    width: 22em
}
.book, .article {
    margin-left: 25em
}
div .toclevel1 {
    font-size: 1.2em;
}
}
</style>

<style type="text/css">
:root {
    /* pre-defined colors */
    --pdt-grey-950: hsl(0deg, 0%, 95%);
    --pdt-grey-750: hsl(0deg, 0%, 75%);
    --pdt-grey-400: hsl(0deg, 0%, 40%);
    --pdt-grey-250: hsl(0deg, 0%, 25%);
    --pdt-grey-150: hsl(0deg, 0%, 15%);
    --pdt-grey-100: hsl(0deg, 0%, 10%);
    --pdt-primary-850: hsl(205deg, 100%, 85%);
    --pdt-primary-800: hsl(205deg, 100%, 80%);
    --pdt-primary-700: hsl(205deg, 100%, 70%);
    --pdt-secondary-850: hsl(250deg, 100%, 85%);
}

/* adjust admonition block spacing. this allows for a background on
 * admonition blocks that doesn't make the elements look to tightly
 * spaced.
 */
div.admonitionblock {
    border-radius: 3px;
    margin: 1.5em 0;
    padding: 0.5em 10% 0.5em 0.5em;
}

div.admonitionblock td.icon {
    padding-right: 0.5em;
}

div.admonitionblock td.icon > img {
    box-sizing: border-box;
    padding: 0.15em;
}

/* Fine-tune headings a bit */
h4 {
  font-size: 1.1em;
}
h5 {
  font-size: 1.05em;
}
h6 {
  font-size: 1.0em;
}

/* Support for heading anchor links */
h3 {
    border-bottom: unset;
}

h3 > span {
    display: inline-block;
    border-bottom: 2px solid silver;
}

a.headerlink {
    color: var(--pdt-grey-750);
    padding: 0 4px;
    text-decoration: none;
    visibility: hidden;
}

/* add it as an pseudo-element, so that it does not show up in the ToC */
a.headerlink::after {
    content: '\00b6';
    text-decoration: none;
}

h1:hover > a.headerlink,
h2:hover > a.headerlink,
h3:hover > a.headerlink,
h4:hover > a.headerlink,
h5:hover > a.headerlink,
h6:hover > a.headerlink {
  visibility: visible;
}

/* Dark mode theme */
@media screen and (prefers-color-scheme: dark) {
    :root {
        color-scheme: dark;
        --pdt-body-background: var(--pdt-grey-150);
        --pdt-text: var(--pdt-grey-950);
        --pdt-headline: var(--pdt-primary-800);
        --pdt-link: var(--pdt-primary-700);
        --pdt-link-visited: var(--pdt-secondary-850);
        --pdt-highlighted-text: var(--pdt-primary-850);
        --pdt-background-sidebar: var(--pdt-grey-100);
        --pdt-background-listings: var(--pdt-grey-100);
        --pdt-border: var(--pdt-grey-400);
        --pdt-border-alt: var(--pdt-grey-250);
        --pdt-table-border: var(--pdt-grey-400);
        --pdt-background-admonition: var(--pdt-grey-250);
    }

    body {
        color: var(--pdt-text);
        background-color: var(--pdt-body-background);
    }

    a {
        color: var(--pdt-link);
    }

    a:visited {
        color: var(--pdt-link-visited);
    }

    /* style headlines, titles etc. */
    h1,
    h2,
    h3,
    h4,
    h5,
    h6,
    thead,
    #author,
    #toctitle,
    div.title,
    td.hdlist1,
    caption.title,
    p.tableblock.header {
        color: var(--pdt-headline);
    }

    h1,
    h2,
    h3,
    #footer {
        border-color: var(--pdt-border);
    }

    /* formatted colored text */
    dt,
    em,
    pre,
    code,
    strong,
    .monospaced {
        color: var(--pdt-highlighted-text);
    }

    /* style the table of contents sidebar */
    div #toc {
        color: var(--pdt-text);
        background-color: var(--pdt-background-sidebar);
        border-color: var(--pdt-border-alt);
    }

    div #toc a:link,
    div #toc a:visited {
        color: var(--pdt-text);
    }

    /* reduce the brigthness of images a bit and make it reversable
     * through hovering over them.
     */
    .image > img {
        filter: brightness(90%);
    }

    .image > img:hover {
        filter: none;
    }

    /* tables */
    th.tableblock,
    td.tableblock,
    table.tableblock {
        border-color: var(--pdt-table-border);
    }

    div.quoteblock,
    div.verseblock {
        color: var(--pdt-text);
        border-color: var(--pdt-border);
    }

    /* listings (e.g. code snippet blocks) */
    div.listingblock > div.content {
        background-color: var(--pdt-background-listings);
        border-color: var(--pdt-border-alt);
    }

    /* admonition blocks (e.g. notes, warnings etc.) */
    div.admonitionblock {
        color: var(--pdt-text);
        background-color: var(--pdt-background-admonition);
    }

    div.admonitionblock td.content {
        border-color: var(--pdt-border);
    }

    /* makes the admonition icons appear a bit more consistent, by
     * adding a white background the shadows in the icons look
     * "correct"
     */
    div.admonitionblock td.icon > img {
        background-color: white;
        border-radius: 100%;
        filter: brightness(95%);
    }

    /* invert the logo */
    #header > h1 > .image > img {
        filter: invert(100%) hue-rotate(180deg) brightness(90%);
    }

    /* fixes the black text on unorderd lists */
    ul > li > * {
        color: var(--pdt-text);
    }
}
</style>

<script type="text/javascript">
/*<![CDATA[*/
var asciidoc = {  // Namespace.

/////////////////////////////////////////////////////////////////////
// Table Of Contents generator
/////////////////////////////////////////////////////////////////////

/* Author: Mihai Bazon, September 2002
 * http://students.infoiasi.ro/~mishoo
 *
 * Table Of Content generator
 * Version: 0.4
 *
 * Feel free to use this script under the terms of the GNU General Public
 * License, as long as you do not remove or alter this notice.
 */

 /* modified by Troy D. Hanson, September 2006. License: GPL */
 /* modified by Stuart Rackham, 2006, 2009. License: GPL */

// toclevels = 1..4.
toc: function (toclevels) {

  function getText(el) {
    var text = "";
    for (var i = el.firstChild; i != null; i = i.nextSibling) {
      if (i.nodeType == 3 /* Node.TEXT_NODE */) // IE doesn't speak constants.
        text += i.data;
      else if (i.firstChild != null)
        text += getText(i);
    }
    return text;
  }

  function TocEntry(el, text, toclevel) {
    this.element = el;
    this.text = text;
    this.toclevel = toclevel;
  }

  function tocEntries(el, toclevels) {
    var result = new Array;
    var re = new RegExp('[hH]([1-'+(toclevels+1)+'])');
    // Function that scans the DOM tree for header elements (the DOM2
    // nodeIterator API would be a better technique but not supported by all
    // browsers).
    var iterate = function (el) {
      for (var i = el.firstChild; i != null; i = i.nextSibling) {
        if (i.nodeType == 1 /* Node.ELEMENT_NODE */) {
          var mo = re.exec(i.tagName);
          if (mo && (i.getAttribute("class") || i.getAttribute("className")) != "float") {
            result[result.length] = new TocEntry(i, getText(i), mo[1]-1);
          }
          iterate(i);
        }
      }
    }
    iterate(el);
    return result;
  }

  var toc = document.getElementById("toc");
  if (!toc) {
    return;
  }

  // Delete existing TOC entries in case we're reloading the TOC.
  var tocEntriesToRemove = [];
  var i;
  for (i = 0; i < toc.childNodes.length; i++) {
    var entry = toc.childNodes[i];
    if (entry.nodeName.toLowerCase() == 'div'
     && entry.getAttribute("class")
     && entry.getAttribute("class").match(/^toclevel/))
      tocEntriesToRemove.push(entry);
  }
  for (i = 0; i < tocEntriesToRemove.length; i++) {
    toc.removeChild(tocEntriesToRemove[i]);
  }

  // Rebuild TOC entries.
  var entries = tocEntries(document.getElementById("content"), toclevels);
  for (var i = 0; i < entries.length; ++i) {
    var entry = entries[i];
    if (entry.element.id == "")
      entry.element.id = "_toc_" + i;
    var a = document.createElement("a");
    a.href = "#" + entry.element.id;
    a.appendChild(document.createTextNode(entry.text));
    var div = document.createElement("div");
    div.appendChild(a);
    div.className = "toclevel" + entry.toclevel;
    toc.appendChild(div);
  }
  if (entries.length == 0)
    toc.parentNode.removeChild(toc);
},


/////////////////////////////////////////////////////////////////////
// Footnotes generator
/////////////////////////////////////////////////////////////////////

/* Based on footnote generation code from:
 * http://www.brandspankingnew.net/archive/2005/07/format_footnote.html
 */

footnotes: function () {
  // Delete existing footnote entries in case we're reloading the footnodes.
  var i;
  var noteholder = document.getElementById("footnotes");
  if (!noteholder) {
    return;
  }
  var entriesToRemove = [];
  for (i = 0; i < noteholder.childNodes.length; i++) {
    var entry = noteholder.childNodes[i];
    if (entry.nodeName.toLowerCase() == 'div' && entry.getAttribute("class") == "footnote")
      entriesToRemove.push(entry);
  }
  for (i = 0; i < entriesToRemove.length; i++) {
    noteholder.removeChild(entriesToRemove[i]);
  }

  // Rebuild footnote entries.
  var cont = document.getElementById("content");
  var spans = cont.getElementsByTagName("span");
  var refs = {};
  var n = 0;
  for (i=0; i<spans.length; i++) {
    if (spans[i].className == "footnote") {
      n++;
      var note = spans[i].getAttribute("data-note");
      if (!note) {
        // Use [\s\S] in place of . so multi-line matches work.
        // Because JavaScript has no s (dotall) regex flag.
        note = spans[i].innerHTML.match(/\s*\[([\s\S]*)]\s*/)[1];
        spans[i].innerHTML =
          "[<a id='_footnoteref_" + n + "' href='#_footnote_" + n +
          "' title='View footnote' class='footnote'>" + n + "</a>]";
        spans[i].setAttribute("data-note", note);
      }
      noteholder.innerHTML +=
        "<div class='footnote' id='_footnote_" + n + "'>" +
        "<a href='#_footnoteref_" + n + "' title='Return to text'>" +
        n + "</a>. " + note + "</div>";
      var id =spans[i].getAttribute("id");
      if (id != null) refs["#"+id] = n;
    }
  }
  if (n == 0)
    noteholder.parentNode.removeChild(noteholder);
  else {
    // Process footnoterefs.
    for (i=0; i<spans.length; i++) {
      if (spans[i].className == "footnoteref") {
        var href = spans[i].getElementsByTagName("a")[0].getAttribute("href");
        href = href.match(/#.*/)[0];  // Because IE return full URL.
        n = refs[href];
        spans[i].innerHTML =
          "[<a href='#_footnote_" + n +
          "' title='View footnote' class='footnote'>" + n + "</a>]";
      }
    }
  }
},

install: function(toclevels) {
  var timerId;

  function reinstall() {
    asciidoc.footnotes();
    if (toclevels) {
      asciidoc.toc(toclevels);
    }
  }

  function reinstallAndRemoveTimer() {
    clearInterval(timerId);
    reinstall();
  }

  timerId = setInterval(reinstall, 500);
  if (document.addEventListener)
    document.addEventListener("DOMContentLoaded", reinstallAndRemoveTimer, false);
  else
    window.onload = reinstallAndRemoveTimer;
}

}
asciidoc.install(2);
/*]]>*/
</script>
</head>
<body class="book" style="max-width:55em">
<div id="header">
<h1><span class="image">
<img alt="./images/proxmox-logo.svg" title="Proxmox VE Administration Guide" src="data:image/svg+xml;base64,
PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiIHN0YW5kYWxvbmU9Im5vIj8+Cjwh
LS0gQ3JlYXRlZCB3aXRoIElua3NjYXBlIChodHRwOi8vd3d3Lmlua3NjYXBlLm9yZy8pIC0tPgoK
PHN2ZwogICB4bWxuczpkYz0iaHR0cDovL3B1cmwub3JnL2RjL2VsZW1lbnRzLzEuMS8iCiAgIHht
bG5zOmNjPSJodHRwOi8vY3JlYXRpdmVjb21tb25zLm9yZy9ucyMiCiAgIHhtbG5zOnJkZj0iaHR0
cDovL3d3dy53My5vcmcvMTk5OS8wMi8yMi1yZGYtc3ludGF4LW5zIyIKICAgeG1sbnM6c3ZnPSJo
dHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyIKICAgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIw
MDAvc3ZnIgogICB4bWxuczpzb2RpcG9kaT0iaHR0cDovL3NvZGlwb2RpLnNvdXJjZWZvcmdlLm5l
dC9EVEQvc29kaXBvZGktMC5kdGQiCiAgIHhtbG5zOmlua3NjYXBlPSJodHRwOi8vd3d3Lmlua3Nj
YXBlLm9yZy9uYW1lc3BhY2VzL2lua3NjYXBlIgogICB3aWR0aD0iODAwIgogICBoZWlnaHQ9IjEy
OS4wMzkyOSIKICAgaWQ9InN2ZzMwMTgiCiAgIHZlcnNpb249IjEuMSIKICAgaW5rc2NhcGU6dmVy
c2lvbj0iMC40OC4zLjEgcjk4ODYiCiAgIHNvZGlwb2RpOmRvY25hbWU9IlByb3htb3hfbG9nb19z
dGFuZGFyZF9oZXguc3ZnIgogICBpbmtzY2FwZTpleHBvcnQtZmlsZW5hbWU9IlM6XFByb3htb3gt
TG9nby1ORVdcU3RhbmRhcmQtTG9nb1xQTkdcUHJveG1veF9sb2dvX3N0YW5kYXJkX2hleF84MDBw
eC5wbmciCiAgIGlua3NjYXBlOmV4cG9ydC14ZHBpPSI5MCIKICAgaW5rc2NhcGU6ZXhwb3J0LXlk
cGk9IjkwIj4KICA8c29kaXBvZGk6bmFtZWR2aWV3CiAgICAgaWQ9ImJhc2UiCiAgICAgcGFnZWNv
bG9yPSIjZmZmZmZmIgogICAgIGJvcmRlcmNvbG9yPSIjNjY2NjY2IgogICAgIGJvcmRlcm9wYWNp
dHk9IjEuMCIKICAgICBpbmtzY2FwZTpwYWdlb3BhY2l0eT0iMC4wIgogICAgIGlua3NjYXBlOnBh
Z2VzaGFkb3c9IjIiCiAgICAgaW5rc2NhcGU6em9vbT0iMi44IgogICAgIGlua3NjYXBlOmN4PSIz
NDkuMDE2MDIiCiAgICAgaW5rc2NhcGU6Y3k9IjY0LjUwNTI2NSIKICAgICBpbmtzY2FwZTpkb2N1
bWVudC11bml0cz0icHgiCiAgICAgaW5rc2NhcGU6Y3VycmVudC1sYXllcj0ibGF5ZXIxIgogICAg
IHNob3dncmlkPSJmYWxzZSIKICAgICBpbmtzY2FwZTp3aW5kb3ctd2lkdGg9IjI1NjAiCiAgICAg
aW5rc2NhcGU6d2luZG93LWhlaWdodD0iMTM3NyIKICAgICBpbmtzY2FwZTp3aW5kb3cteD0iLTgi
CiAgICAgaW5rc2NhcGU6d2luZG93LXk9Ii04IgogICAgIGlua3NjYXBlOndpbmRvdy1tYXhpbWl6
ZWQ9IjEiCiAgICAgc2hvd2d1aWRlcz0idHJ1ZSIKICAgICBpbmtzY2FwZTpndWlkZS1iYm94PSJ0
cnVlIgogICAgIG9iamVjdHRvbGVyYW5jZT0iNCIKICAgICBncmlkdG9sZXJhbmNlPSIyIgogICAg
IGd1aWRldG9sZXJhbmNlPSIxIgogICAgIGZpdC1tYXJnaW4tdG9wPSI1IgogICAgIGZpdC1tYXJn
aW4tbGVmdD0iNSIKICAgICBmaXQtbWFyZ2luLXJpZ2h0PSI1IgogICAgIGZpdC1tYXJnaW4tYm90
dG9tPSI1IgogICAgIGlua3NjYXBlOnNob3dwYWdlc2hhZG93PSJmYWxzZSIgLz4KICA8ZGVmcwog
ICAgIGlkPSJkZWZzMzAyMCI+CiAgICA8Y2xpcFBhdGgKICAgICAgIGNsaXBQYXRoVW5pdHM9InVz
ZXJTcGFjZU9uVXNlIgogICAgICAgaWQ9ImNsaXBQYXRoMzEwMi0xIj4KICAgICAgPHJlY3QKICAg
ICAgICAgc3R5bGU9ImZpbGw6I2U1NzAwMDtmaWxsLW9wYWNpdHk6MTtmaWxsLXJ1bGU6ZXZlbm9k
ZDtzdHJva2U6bm9uZSIKICAgICAgICAgaWQ9InJlY3QzMTA0LTciCiAgICAgICAgIHdpZHRoPSI0
MzYuNDAxODkiCiAgICAgICAgIGhlaWdodD0iMzI2LjQwOTkxIgogICAgICAgICB4PSItODIuOTk5
OTE2IgogICAgICAgICB5PSItMzQ3LjcxMzg3IgogICAgICAgICB0cmFuc2Zvcm09Im1hdHJpeCgw
LjczNDQ5MTYxLDAuNjc4NjE3NzYsLTAuNzg0OTcxOTMsMC42MTk1MzEzMywwLDApIiAvPgogICAg
PC9jbGlwUGF0aD4KICAgIDxjbGlwUGF0aAogICAgICAgY2xpcFBhdGhVbml0cz0idXNlclNwYWNl
T25Vc2UiCiAgICAgICBpZD0iY2xpcFBhdGg0NzQ2Ij4KICAgICAgPHJlY3QKICAgICAgICAgc3R5
bGU9ImZpbGw6I2U1NzAwMDtmaWxsLW9wYWNpdHk6MTtmaWxsLXJ1bGU6ZXZlbm9kZDtzdHJva2U6
bm9uZSIKICAgICAgICAgaWQ9InJlY3Q0NzQ4IgogICAgICAgICB3aWR0aD0iNDM2LjQwMTg5Igog
ICAgICAgICBoZWlnaHQ9IjMyNi40MDk5MSIKICAgICAgICAgeD0iLTgyLjk5OTkxNiIKICAgICAg
ICAgeT0iLTM0Ny43MTM4NyIKICAgICAgICAgdHJhbnNmb3JtPSJtYXRyaXgoMC43MzQ0OTE2MSww
LjY3ODYxNzc2LC0wLjc4NDk3MTkzLDAuNjE5NTMxMzMsMCwwKSIgLz4KICAgIDwvY2xpcFBhdGg+
CiAgICA8Y2xpcFBhdGgKICAgICAgIGlkPSJjbGlwUGF0aDI5OTktMi0zLTQtMSIKICAgICAgIGNs
aXBQYXRoVW5pdHM9InVzZXJTcGFjZU9uVXNlIj4KICAgICAgPHBhdGgKICAgICAgICAgaW5rc2Nh
cGU6Y29ubmVjdG9yLWN1cnZhdHVyZT0iMCIKICAgICAgICAgaWQ9InBhdGgzMDAxLTgtOC02LTAi
CiAgICAgICAgIGQ9Im0gMCwxNDQwMCAxNDQwMCwwIEwgMTQ0MDAsMCAwLDAgMCwxNDQwMCB6IiAv
PgogICAgPC9jbGlwUGF0aD4KICAgIDxjbGlwUGF0aAogICAgICAgY2xpcFBhdGhVbml0cz0idXNl
clNwYWNlT25Vc2UiCiAgICAgICBpZD0iY2xpcFBhdGgzMTAyLTQiPgogICAgICA8cmVjdAogICAg
ICAgICBzdHlsZT0iZmlsbDojZTU3MDAwO2ZpbGwtb3BhY2l0eToxO2ZpbGwtcnVsZTpldmVub2Rk
O3N0cm9rZTpub25lIgogICAgICAgICBpZD0icmVjdDMxMDQtMSIKICAgICAgICAgd2lkdGg9IjQz
Ni40MDE4OSIKICAgICAgICAgaGVpZ2h0PSIzMjYuNDA5OTEiCiAgICAgICAgIHg9Ii04Mi45OTk5
MTYiCiAgICAgICAgIHk9Ii0zNDcuNzEzODciCiAgICAgICAgIHRyYW5zZm9ybT0ibWF0cml4KDAu
NzM0NDkxNjEsMC42Nzg2MTc3NiwtMC43ODQ5NzE5MywwLjYxOTUzMTMzLDAsMCkiIC8+CiAgICA8
L2NsaXBQYXRoPgogICAgPGNsaXBQYXRoCiAgICAgICBjbGlwUGF0aFVuaXRzPSJ1c2VyU3BhY2VP
blVzZSIKICAgICAgIGlkPSJjbGlwUGF0aDMwMDgiPgogICAgICA8cmVjdAogICAgICAgICBzdHls
ZT0iZmlsbDojZTU3MDAwO2ZpbGwtb3BhY2l0eToxO2ZpbGwtcnVsZTpldmVub2RkO3N0cm9rZTpu
b25lIgogICAgICAgICBpZD0icmVjdDMwMTAiCiAgICAgICAgIHdpZHRoPSI0MzYuNDAxODkiCiAg
ICAgICAgIGhlaWdodD0iMzI2LjQwOTkxIgogICAgICAgICB4PSItODIuOTk5OTE2IgogICAgICAg
ICB5PSItMzQ3LjcxMzg3IgogICAgICAgICB0cmFuc2Zvcm09Im1hdHJpeCgwLjczNDQ5MTYxLDAu
Njc4NjE3NzYsLTAuNzg0OTcxOTMsMC42MTk1MzEzMywwLDApIiAvPgogICAgPC9jbGlwUGF0aD4K
ICAgIDxjbGlwUGF0aAogICAgICAgY2xpcFBhdGhVbml0cz0idXNlclNwYWNlT25Vc2UiCiAgICAg
ICBpZD0iY2xpcFBhdGgzMTAyIj4KICAgICAgPHJlY3QKICAgICAgICAgc3R5bGU9ImZpbGw6I2U1
NzAwMDtmaWxsLW9wYWNpdHk6MTtmaWxsLXJ1bGU6ZXZlbm9kZDtzdHJva2U6bm9uZSIKICAgICAg
ICAgaWQ9InJlY3QzMTA0IgogICAgICAgICB3aWR0aD0iNDM2LjQwMTg5IgogICAgICAgICBoZWln
aHQ9IjMyNi40MDk5MSIKICAgICAgICAgeD0iLTgyLjk5OTkxNiIKICAgICAgICAgeT0iLTM0Ny43
MTM4NyIKICAgICAgICAgdHJhbnNmb3JtPSJtYXRyaXgoMC43MzQ0OTE2MSwwLjY3ODYxNzc2LC0w
Ljc4NDk3MTkzLDAuNjE5NTMxMzMsMCwwKSIgLz4KICAgIDwvY2xpcFBhdGg+CiAgICA8Y2xpcFBh
dGgKICAgICAgIGNsaXBQYXRoVW5pdHM9InVzZXJTcGFjZU9uVXNlIgogICAgICAgaWQ9ImNsaXBQ
YXRoNDgxNCI+CiAgICAgIDxyZWN0CiAgICAgICAgIHN0eWxlPSJmaWxsOiNlNTcwMDA7ZmlsbC1v
cGFjaXR5OjE7ZmlsbC1ydWxlOmV2ZW5vZGQ7c3Ryb2tlOm5vbmUiCiAgICAgICAgIGlkPSJyZWN0
NDgxNiIKICAgICAgICAgd2lkdGg9IjQzNi40MDE4OSIKICAgICAgICAgaGVpZ2h0PSIzMjYuNDA5
OTEiCiAgICAgICAgIHg9Ii04Mi45OTk5MTYiCiAgICAgICAgIHk9Ii0zNDcuNzEzODciCiAgICAg
ICAgIHRyYW5zZm9ybT0ibWF0cml4KDAuNzM0NDkxNjEsMC42Nzg2MTc3NiwtMC43ODQ5NzE5Myww
LjYxOTUzMTMzLDAsMCkiIC8+CiAgICA8L2NsaXBQYXRoPgogICAgPGNsaXBQYXRoCiAgICAgICBp
ZD0iY2xpcFBhdGgyOTk5LTItMy00IgogICAgICAgY2xpcFBhdGhVbml0cz0idXNlclNwYWNlT25V
c2UiPgogICAgICA8cGF0aAogICAgICAgICBpbmtzY2FwZTpjb25uZWN0b3ItY3VydmF0dXJlPSIw
IgogICAgICAgICBpZD0icGF0aDMwMDEtOC04LTYiCiAgICAgICAgIGQ9Im0gMCwxNDQwMCAxNDQw
MCwwIEwgMTQ0MDAsMCAwLDAgMCwxNDQwMCB6IiAvPgogICAgPC9jbGlwUGF0aD4KICA8L2RlZnM+
CiAgPG1ldGFkYXRhCiAgICAgaWQ9Im1ldGFkYXRhMzAyMyI+CiAgICA8cmRmOlJERj4KICAgICAg
PGNjOldvcmsKICAgICAgICAgcmRmOmFib3V0PSIiPgogICAgICAgIDxkYzpmb3JtYXQ+aW1hZ2Uv
c3ZnK3htbDwvZGM6Zm9ybWF0PgogICAgICAgIDxkYzp0eXBlCiAgICAgICAgICAgcmRmOnJlc291
cmNlPSJodHRwOi8vcHVybC5vcmcvZGMvZGNtaXR5cGUvU3RpbGxJbWFnZSIgLz4KICAgICAgICA8
ZGM6dGl0bGU+PC9kYzp0aXRsZT4KICAgICAgPC9jYzpXb3JrPgogICAgPC9yZGY6UkRGPgogIDwv
bWV0YWRhdGE+CiAgPGcKICAgICB0cmFuc2Zvcm09InRyYW5zbGF0ZSgxNDUxLjEzNzQsMzguOTMy
MjkzKSIKICAgICBpZD0ibGF5ZXIxIgogICAgIGlua3NjYXBlOmdyb3VwbW9kZT0ibGF5ZXIiCiAg
ICAgaW5rc2NhcGU6bGFiZWw9IkxheWVyIDEiPgogICAgPGcKICAgICAgIGlkPSJnMzM3OCIKICAg
ICAgIHRyYW5zZm9ybT0ibWF0cml4KDAuMjgwMjY3MTksMCwwLDAuMjgwMjY3MTksLTEwNDAuODMy
NSw2MS4yNTQyOTQpIgogICAgICAgaW5rc2NhcGU6ZXhwb3J0LWZpbGVuYW1lPSJTOlxsdWNpYVxO
ZXctTG9nby1TeW1ib2xcU3RhbmRhcmQtTG9nb1xQTkdcUHJveG1veF9sb2dvX3N0YW5kYXJkX2hl
eF8zMDBweC5wbmciCiAgICAgICBpbmtzY2FwZTpleHBvcnQteGRwaT0iOTAiCiAgICAgICBpbmtz
Y2FwZTpleHBvcnQteWRwaT0iOTAiPgogICAgICA8ZwogICAgICAgICBpZD0iZzMwMTYtOSIKICAg
ICAgICAgc3R5bGU9ImZpbGw6IzAwMDAwMCIKICAgICAgICAgdHJhbnNmb3JtPSJtYXRyaXgoMC44
NTI4NjU5NCwwLDAsMC44NTI4NjU5NCwtMjIwNC43OTY0LC05NzAuMDY0OTUpIj4KICAgICAgICA8
ZwogICAgICAgICAgIHN0eWxlPSJmb250LXNpemU6MTQ0cHg7Zm9udC1zdHlsZTpub3JtYWw7Zm9u
dC12YXJpYW50Om5vcm1hbDtmb250LXdlaWdodDpub3JtYWw7Zm9udC1zdHJldGNoOm5vcm1hbDts
aW5lLWhlaWdodDoxMjUlO2xldHRlci1zcGFjaW5nOjBweDt3b3JkLXNwYWNpbmc6MHB4O2ZpbGw6
IzAwMDAwMDtmaWxsLW9wYWNpdHk6MTtzdHJva2U6bm9uZTtmb250LWZhbWlseTpIZWxpb247LWlu
a3NjYXBlLWZvbnQtc3BlY2lmaWNhdGlvbjpIZWxpb24iCiAgICAgICAgICAgaWQ9InRleHQzMDkz
LTUiCiAgICAgICAgICAgY2xpcC1wYXRoPSJ1cmwoI2NsaXBQYXRoMzEwMi00KSIKICAgICAgICAg
ICB0cmFuc2Zvcm09Im1hdHJpeCgtMC45OTc5MTk3OSwwLDAsMC45OTc5MTk3OSwxNDUyLjM1ODYs
NzQ2LjA0Nzg0KSI+CiAgICAgICAgICA8cGF0aAogICAgICAgICAgICAgZD0iTSAyNzYuMzA0NDMs
MjI2LjYyMzEgNDY2Ljg5MjI3LDE3LjAyMDYwNSBDIDQ1OS40OTkwMyw5LjYyODA4NzEgNDUwLjg4
NzYyLDMuODE5NDMyMSA0NDEuMDU3OTYsLTAuNDA1Mzc2OTUgNDMxLjIyNzU2LC00LjYyOTUxNzEg
NDIwLjY2NjQsLTYuNzgyMzcyMSA0MDkuMzc0MzcsLTYuODYzOTQ3OSAzOTcuMzgwOTksLTYuNzcy
MjAzMSAzODYuMzkzMjgsLTQuMzk1OTMyNiAzNzYuNDExMjMsMC4yNjQ4NzA5NiAzNjYuNDI4Niw0
LjkyNjM0MTcgMzU3Ljc1NjIzLDExLjMyMzk4IDM1MC4zOTQxMywxOS40NTc4MDUgTCAyNzYuMzAz
MjcsMTAwLjM3MjgyIDIwMi42OTk4NSwxOS40NTc4MDUgQyAxOTUuMDczMzUsMTEuMzIzOTcgMTg2
LjE5Nzg3LDQuOTI2MzI0OCAxNzYuMDczNDQsMC4yNjQ4NDkzIDE2NS45NDg4MSwtNC4zOTU5NTkg
MTU1LjAwMTc1LC02Ljc3MjIyMjQgMTQzLjIzMjIsLTYuODYzOTQ3OSAxMzEuOTM5NzgsLTYuNzgy
MzYxNiAxMjEuMzc4NiwtNC42Mjk1MDUgMTExLjU0ODYsLTAuNDA1MzY5NTEgMTAxLjcxODU1LDMu
ODE5NDM0NSA5My4xMDcxMjIsOS42MjgwODcxIDg1LjcxNDI4NywxNy4wMjA2MDUgTCAyNzYuMzA3
NjksMjI2LjYxNzUyIgogICAgICAgICAgICAgaWQ9InBhdGgzMDk4LTUiCiAgICAgICAgICAgICBp
bmtzY2FwZTpjb25uZWN0b3ItY3VydmF0dXJlPSIwIgogICAgICAgICAgICAgc29kaXBvZGk6bm9k
ZXR5cGVzPSJjY2NjY2NjY2NjY2NjIgogICAgICAgICAgICAgc3R5bGU9ImZpbGw6IzAwMDAwMDtm
aWxsLW9wYWNpdHk6MSIgLz4KICAgICAgICA8L2c+CiAgICAgICAgPGcKICAgICAgICAgICB0cmFu
c2Zvcm09Im1hdHJpeCgwLjk5NzkxOTc5LDAsMCwtMC45OTc5MTk3OSw5MDAuODk2MDQsMTIzMC4z
NTc2KSIKICAgICAgICAgICBjbGlwLXBhdGg9InVybCgjY2xpcFBhdGgzMTAyLTQpIgogICAgICAg
ICAgIGlkPSJnMzE5Ni05IgogICAgICAgICAgIHN0eWxlPSJmb250LXNpemU6MTQ0cHg7Zm9udC1z
dHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDtmb250LXdlaWdodDpub3JtYWw7Zm9udC1z
dHJldGNoOm5vcm1hbDtsaW5lLWhlaWdodDoxMjUlO2xldHRlci1zcGFjaW5nOjBweDt3b3JkLXNw
YWNpbmc6MHB4O2ZpbGw6IzAwMDAwMDtmaWxsLW9wYWNpdHk6MTtzdHJva2U6bm9uZTtmb250LWZh
bWlseTpIZWxpb247LWlua3NjYXBlLWZvbnQtc3BlY2lmaWNhdGlvbjpIZWxpb24iPgogICAgICAg
ICAgPHBhdGgKICAgICAgICAgICAgIHN0eWxlPSJmaWxsOiMwMDAwMDA7ZmlsbC1vcGFjaXR5OjEi
CiAgICAgICAgICAgICBzb2RpcG9kaTpub2RldHlwZXM9ImNjY2NjY2NjY2NjY2MiCiAgICAgICAg
ICAgICBpbmtzY2FwZTpjb25uZWN0b3ItY3VydmF0dXJlPSIwIgogICAgICAgICAgICAgaWQ9InBh
dGgzMTk4LTEiCiAgICAgICAgICAgICBkPSJNIDI3Ni4zMDQ0MywyMjYuNjIzMSA0NjYuODkyMjcs
MTcuMDIwNjA1IEMgNDU5LjQ5OTAzLDkuNjI4MDg3MSA0NTAuODg3NjIsMy44MTk0MzIxIDQ0MS4w
NTc5NiwtMC40MDUzNzY5NSA0MzEuMjI3NTYsLTQuNjI5NTE3MSA0MjAuNjY2NCwtNi43ODIzNzIx
IDQwOS4zNzQzNywtNi44NjM5NDc5IDM5Ny4zODA5OSwtNi43NzIyMDMxIDM4Ni4zOTMyOCwtNC4z
OTU5MzI2IDM3Ni40MTEyMywwLjI2NDg3MDk2IDM2Ni40Mjg2LDQuOTI2MzQxNyAzNTcuNzU2MjMs
MTEuMzIzOTggMzUwLjM5NDEzLDE5LjQ1NzgwNSBMIDI3Ni4zMDMyNywxMDAuMzcyODIgMjAyLjY5
OTg1LDE5LjQ1NzgwNSBDIDE5NS4wNzMzNSwxMS4zMjM5NyAxODYuMTk3ODcsNC45MjYzMjQ4IDE3
Ni4wNzM0NCwwLjI2NDg0OTMgMTY1Ljk0ODgxLC00LjM5NTk1OSAxNTUuMDAxNzUsLTYuNzcyMjIy
NCAxNDMuMjMyMiwtNi44NjM5NDc5IDEzMS45Mzk3OCwtNi43ODIzNjE2IDEyMS4zNzg2LC00LjYy
OTUwNSAxMTEuNTQ4NiwtMC40MDUzNjk1MSAxMDEuNzE4NTUsMy44MTk0MzQ1IDkzLjEwNzEyMiw5
LjYyODA4NzEgODUuNzE0Mjg3LDE3LjAyMDYwNSBMIDI3Ni4zMDc2OSwyMjYuNjE3NTIiIC8+CiAg
ICAgICAgPC9nPgogICAgICAgIDxwYXRoCiAgICAgICAgICAgc3R5bGU9ImZvbnQtc2l6ZToxMDU5
LjYxMjc5Mjk3cHg7Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDtmb250LXdl
aWdodDpub3JtYWw7Zm9udC1zdHJldGNoOm5vcm1hbDtsaW5lLWhlaWdodDoxMjUlO2xldHRlci1z
cGFjaW5nOjBweDt3b3JkLXNwYWNpbmc6MHB4O2ZpbGw6I2U1NzAwMDtmaWxsLW9wYWNpdHk6MTtz
dHJva2U6bm9uZTtmb250LWZhbWlseTpIZWxpb247LWlua3NjYXBlLWZvbnQtc3BlY2lmaWNhdGlv
bjpIZWxpb24iCiAgICAgICAgICAgZD0ibSAxMTYwLjYyOTIsOTg4LjIwNTM4IDAsMCAtMTQzLjI2
NjUsLTE1Ny4xMTIzMiBjIC04LjMzMywtOC44ODcyMSAtMTguMDMwNjEsLTE1Ljg3NzU0IC0yOS4w
OTI3OSwtMjAuOTcwNzggLTExLjA2MjUyLC01LjA5MjU1IC0yMy4wMjM2NiwtNy42ODg4OSAtMzUu
ODgzNDMsLTcuNzg5MTQgLTEyLjMzODQzLDAuMDg5MiAtMjMuODc3OTIsMi40NDE0NyAtMzQuNjE4
NDksNy4wNTY4NSAtMTAuNzQwNjIsNC42MTYxMyAtMjAuMTQ5NzIsMTAuOTYyODQgLTI4LjIyNzM2
LDE5LjA0MDE2IEwgMTAzNC45MzgzLDk4OC4yMDc3NyA4ODkuNTQwNjMsMTE0Ny45ODU0IGMgOC4w
Nzc2Miw4LjMzMjkgMTcuNDg2NzQsMTQuODM0OSAyOC4yMjczNiwxOS41MDYyIDEwLjc0MDU3LDQu
NjcxMiAyMi4yODAwNiw3LjA0NTcgMzQuNjE4NDksNy4xMjM0IDEyLjg5MzA4LC0wLjEgMjQuOTIw
NzMsLTIuNjk2MiAzNi4wODMwNywtNy43ODkxIDExLjE2MjIzLC01LjA5MjkgMjAuNzkzMjUsLTEy
LjA4MzMgMjguODkzMTUsLTIwLjk3MDggbCAxNDMuMjY2MiwtMTU3LjY0OTE3IgogICAgICAgICAg
IGlkPSJwYXRoMzI3OC00IgogICAgICAgICAgIGlua3NjYXBlOmNvbm5lY3Rvci1jdXJ2YXR1cmU9
IjAiCiAgICAgICAgICAgc29kaXBvZGk6bm9kZXR5cGVzPSJjY2NjY2NjY2NzY3NjYyIgLz4KICAg
ICAgICA8cGF0aAogICAgICAgICAgIHNvZGlwb2RpOm5vZGV0eXBlcz0iY2NjY2NjY2Njc2NzY2Mi
CiAgICAgICAgICAgaW5rc2NhcGU6Y29ubmVjdG9yLWN1cnZhdHVyZT0iMCIKICAgICAgICAgICBp
ZD0icGF0aDMyODAtNSIKICAgICAgICAgICBkPSJtIDExOTIuNjIwOCw5ODguMjA1MzggMCwwIDE0
My4yNjY1LC0xNTcuMTEyMzIgYyA4LjMzMywtOC44ODcyMSAxOC4wMzA2LC0xNS44Nzc1NCAyOS4w
OTI4LC0yMC45NzA3OCAxMS4wNjI1LC01LjA5MjU1IDIzLjAyMzcsLTcuNjg4ODkgMzUuODgzNCwt
Ny43ODkxNCAxMi4zMzg1LDAuMDg5MiAyMy44NzgsMi40NDE0NyAzNC42MTg1LDcuMDU2ODUgMTAu
NzQwNyw0LjYxNjEzIDIwLjE0OTgsMTAuOTYyODQgMjguMjI3NCwxOS4wNDAxNiBsIC0xNDUuMzk3
NywxNTkuNzc3NjIgMTQ1LjM5NzcsMTU5Ljc3NzYzIGMgLTguMDc3Niw4LjMzMjkgLTE3LjQ4Njcs
MTQuODM0OSAtMjguMjI3NCwxOS41MDYyIC0xMC43NDA1LDQuNjcxMiAtMjIuMjgsNy4wNDU3IC0z
NC42MTg0LDcuMTIzNCAtMTIuODkzMSwtMC4xIC0yNC45MjA4LC0yLjY5NjIgLTM2LjA4MzEsLTcu
Nzg5MSAtMTEuMTYyMywtNS4wOTI5IC0yMC43OTMzLC0xMi4wODMzIC0yOC44OTMyLC0yMC45NzA4
IEwgMTE5Mi42MjExLDk4OC4yMDU5MyIKICAgICAgICAgICBzdHlsZT0iZm9udC1zaXplOjEwNTku
NjEyNzkyOTdweDtmb250LXN0eWxlOm5vcm1hbDtmb250LXZhcmlhbnQ6bm9ybWFsO2ZvbnQtd2Vp
Z2h0Om5vcm1hbDtmb250LXN0cmV0Y2g6bm9ybWFsO2xpbmUtaGVpZ2h0OjEyNSU7bGV0dGVyLXNw
YWNpbmc6MHB4O3dvcmQtc3BhY2luZzowcHg7ZmlsbDojZTU3MDAwO2ZpbGwtb3BhY2l0eToxO3N0
cm9rZTpub25lO2ZvbnQtZmFtaWx5OkhlbGlvbjstaW5rc2NhcGUtZm9udC1zcGVjaWZpY2F0aW9u
OkhlbGlvbiIgLz4KICAgICAgPC9nPgogICAgICA8ZwogICAgICAgICBpZD0iZzMzNjgiPgogICAg
ICAgIDxnCiAgICAgICAgICAgaWQ9InRleHQzMjIzIgogICAgICAgICAgIHN0eWxlPSJmb250LXNp
emU6Mzg3LjUyMjA2NDIxcHg7Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDtm
b250LXdlaWdodDpub3JtYWw7Zm9udC1zdHJldGNoOm5vcm1hbDt0ZXh0LWFsaWduOnN0YXJ0O2xp
bmUtaGVpZ2h0OjEyNSU7bGV0dGVyLXNwYWNpbmc6MHB4O3dvcmQtc3BhY2luZzowcHg7d3JpdGlu
Zy1tb2RlOmxyLXRiO3RleHQtYW5jaG9yOnN0YXJ0O2ZpbGw6IzAwMDAwMDtmaWxsLW9wYWNpdHk6
MTtzdHJva2U6bm9uZTtmb250LWZhbWlseTpIZWxpb247LWlua3NjYXBlLWZvbnQtc3BlY2lmaWNh
dGlvbjpIZWxpb24iPgogICAgICAgICAgPHBhdGgKICAgICAgICAgICAgIGlua3NjYXBlOmNvbm5l
Y3Rvci1jdXJ2YXR1cmU9IjAiCiAgICAgICAgICAgICBpZD0icGF0aDMzNTQiCiAgICAgICAgICAg
ICBkPSJtIC02ODYuMTgxMSwtMjYyLjM4NzM3IC0xNzcuODcwMzQsMCBjIC02Ljk4MzQsMC4xOTQw
MyAtMTIuODc2ODksMi43MTI4OCAtMTcuNjgwNDgsNy41NTY1OCAtNC44MDM2LDQuODQ0MjIgLTcu
MzA2MzEsMTAuODUwNzMgLTcuNTA4MTMsMTguMDE5NTUgbCAwLDI0NS42ODU4MjIzIGMgMTkuMDg1
MiwtMC40NzYzMjg5IDM0Ljk3MzM4LC03LjA4MDI2MiA0Ny42NjQ2LC0xOS44MTE4MTkzIDEyLjY5
MTEyLC0xMi43MzE1MTcgMTkuMjc4OSwtMjguNzMyNzI0IDE5Ljc2MzM3LC00OC4wMDM2NjkgbCAx
MzUuNjMwOTgsMCBjIDE5LjI4NjgyLC0wLjQ3NjI2MSAzNS4zNTI2MSwtNy4wODAxOTQgNDguMTk3
NDIsLTE5LjgxMTgxOSAxMi44NDQzMiwtMTIuNzMxNDUgMTkuNTEyODQsLTI4LjczMjY1NSAyMC4w
MDU1OCwtNDguMDAzNjY1IGwgMCwtNjcuNDI3OTggYyAtMC40OTI3NCwtMTkuMjg2ODIgLTcuMTYx
MjUsLTM1LjM1MjYyIC0yMC4wMDU1NywtNDguMTk3NDQgLTEyLjg0NDgyLC0xMi44NDQzMiAtMjgu
OTEwNjEsLTE5LjUxMjgzIC00OC4xOTc0MywtMjAuMDA1NTYgeiBtIC0xMzUuNjMwOTgsMTQ0LjE1
NjM1IDAsLTc1Ljk1MzM1IDExOC45Njc3NCwwIGMgMC42OTQxMSwtMC4zNDY5NCAzLjQ3MTMxLDAu
MzQ3MzYgOC4zMzE2MSwyLjA4MjkxIDQuODU5OTEsMS43MzU5NSA3LjYzNzEyLDYuNTk2MDUgOC4z
MzE2MywxNC41ODAzMyBsIDAsNDIuMjM5MzYgYyAwLjM0Njk0LDAuNzEwNiAtMC4zNDczNiwzLjU1
MjM4IC0yLjA4MjkxLDguNTI1MzggLTEuNzM1OTUsNC45NzMyNiAtNi41OTYwNiw3LjgxNTA1IC0x
NC41ODAzMyw4LjUyNTM3IHoiIC8+CiAgICAgICAgICA8cGF0aAogICAgICAgICAgICAgaW5rc2Nh
cGU6Y29ubmVjdG9yLWN1cnZhdHVyZT0iMCIKICAgICAgICAgICAgIGlkPSJwYXRoMzM1NiIKICAg
ICAgICAgICAgIGQ9Im0gLTMwOC4wNDUyMSwtMTQzLjQxOTYzIDAsLTUwLjc2NDc0IGMgLTAuNDc2
NiwtMTkuMjg2ODIgLTcuMDgwNTMsLTM1LjM1MjYyIC0xOS44MTE4MiwtNDguMTk3NDQgLTEyLjcz
MTc5LC0xMi44NDQzMiAtMjguNzMzLC0xOS41MTI4MyAtNDguMDAzNjcsLTIwLjAwNTU2IGwgLTE3
Ny44NzAzNCwwIGMgLTcuMTUyOTMsMC4xOTQwMyAtMTMuMDk0ODUsMi43MTI4OCAtMTcuODI1Nzgs
Ny41NTY1OCAtNC43MzA5NSw0Ljg0NDIyIC03LjE4NTIyLDEwLjg1MDczIC03LjM2MjgzLDE4LjAx
OTU1IGwgMCwyNDUuNjg1ODIyMyBjIDE5LjEwMTM1LC0wLjQ3NjMyODkgMzUuMDU0MTIsLTcuMDgw
MjYyIDQ3Ljg1ODM2LC0xOS44MTE4MTkzIDEyLjgwNDE0LC0xMi43MzE1MTcgMTkuNDU2NTEsLTI4
LjczMjcyNCAxOS45NTcxMywtNDguMDAzNjY5IGwgMCwtMTcuMDUwNzUxIDc3LjExNTksMCAzOS4x
MzkyMiw1NS44MDI0NTkgYyA2LjI4MDgyLDkuMDA5Nzk0IDE0LjE2MDMzLDE2LjA4MTk3MzIgMjMu
NjM4NTQsMjEuMjE2NTU5NSA5LjQ3NzgxLDUuMTM0NjAyMSAyMC4wNjk5Myw3Ljc1MDMzOTcgMzEu
Nzc2NDEsNy44NDcyMjA4IDUuNjQyOTYsLTAuMDA4MDggMTEuMTE2NjMsLTAuNjcwMDg3MiAxNi40
MjEwMiwtMS45ODYwMzM3IDUuMzAzODgsLTEuMzE1OTQyNSAxMC4yOTMxNiwtMy4yMzczNzY1IDE0
Ljk2Nzg2LC01Ljc2NDMwNzggbCAtNTQuNjM5OTEsLTc3Ljg5MDkzMTggYyAxNS44Mzk1MiwtMy40
MzEwNTIgMjguODIxMzMsLTExLjE5NzUzNSAzOC45NDU0NiwtMjMuMjk5NDY5IDEwLjEyMzYyLC0x
Mi4xMDE3MiAxNS4zNTUxLC0yNi41NTI4NiAxNS42OTQ0NSwtNDMuMzUzNDcgeiBtIC0yMDMuMDU4
OTUsOC4xMzc4NiAwLC01OC45MDI2IDExOC4xOTI3MSwwIGMgMC43MTAyNiwtMC4zNDY5NCAzLjU1
MjA1LDAuMzQ3MzYgOC41MjUzNywyLjA4MjkxIDQuOTcyOTQsMS43MzU5NSA3LjgxNDczLDYuNTk2
MDUgOC41MjUzOCwxNC41ODAzMyBsIDAsMjUuNTc2MTMgYyAwLjM1NTAyLDAuNjk0NDUgLTAuMzU1
NDMsMy40NzE2NiAtMi4xMzEzNSw4LjMzMTYxIC0xLjc3NjMyLDQuODYwMjUgLTYuNzQ5NDUsNy42
Mzc0NiAtMTQuOTE5NCw4LjMzMTYyIHoiIC8+CiAgICAgICAgICA8cGF0aAogICAgICAgICAgICAg
aW5rc2NhcGU6Y29ubmVjdG9yLWN1cnZhdHVyZT0iMCIKICAgICAgICAgICAgIGlkPSJwYXRoMzM1
OCIKICAgICAgICAgICAgIGQ9Im0gLTY2LjI5NzE5MywtMjYyLjM4NzM3IC0xMzUuMjQzNDU3LDAg
YyAtMTkuMjcwOTUsMC40OTI3NSAtMzUuMjcyMTcsNy4xNjEyNiAtNDguMDAzNjgsMjAuMDA1NTgg
LTEyLjczMTU2LDEyLjg0NDgxIC0xOS4zMzU0OSwyOC45MTA2MSAtMTkuODExODEsNDguMTk3NDIg
bCAwLDEzNS4yNDM0NjQgYyAwLjQ3NjMyLDE5LjI3MDk0NSA3LjA4MDI1LDM1LjI3MjE1NCAxOS44
MTE4MSw0OC4wMDM2NzUgMTIuNzMxNTIsMTIuNzMxNTYxIDI4LjczMjczLDE5LjMzNTQ5MjEgNDgu
MDAzNjgsMTkuODExODEzMyBsIDEzNS4yNDM0NTcsMCBDIC00Ny4wMjY1MTksOC4zOTgyNTM0IC0z
MS4wMjUzMSwxLjc5NDMyMDMgLTE4LjI5MzUxOCwtMTAuOTM3MjM3IC01LjU2MjIyNzksLTIzLjY2
ODc1NCAxLjA0MTcwMzMsLTM5LjY2OTk2MSAxLjUxODI5NTMsLTU4Ljk0MDkwNiBsIDAsLTEzNS4y
NDM0NjQgYyAtMC40NzY1OTk3LC0xOS4yODY4MiAtNy4wODA1MzI4LC0zNS4zNTI2MiAtMTkuODEx
ODE5MywtNDguMTk3NDQgLTEyLjczMTc4OCwtMTIuODQ0MzIgLTI4LjczMjk5NSwtMTkuNTEyODMg
LTQ4LjAwMzY2OSwtMjAuMDA1NTYgeiBtIDAsMTg2LjM5NTcxMyBjIDAuMzU1MDIzLDAuNzEwNTM2
IC0wLjM1NTQyNywzLjU1MjMyNSAtMi4xMzEzNSw4LjUyNTM3NiAtMS43NzYzMjQsNC45NzMyMDMg
LTYuNzQ5NDUzLDcuODE0OTkyIC0xNC45MTk0MDEsOC41MjUzNzUgbCAtMTAxLjE0MTk1NiwwIGMg
LTAuNzEwNTMsMC4zNTUyOTQgLTMuNTUyMzIsLTAuMzU1MTU2IC04LjUyNTM4LC0yLjEzMTM1IC00
Ljk3MzIxLC0xLjc3NjA1NCAtNy44MTUsLTYuNzQ5MTgyIC04LjUyNTM3LC0xNC45MTk0MDEgbCAw
LC0xMDEuNTI5NDczIGMgLTAuMzU1MywtMC42OTQxMSAwLjM1NTE1LC0zLjQ3MTMyIDIuMTMxMzQs
LTguMzMxNjIgMS43NzYwNCwtNC44NTk5MiA2Ljc0OTE4LC03LjYzNzEyIDE0LjkxOTQxLC04LjMz
MTYyIGwgMTAxLjE0MTk1NiwwIGMgMC43MTAyNjUsLTAuMzQ2OTQgMy41NTIwNTQsMC4zNDczNiA4
LjUyNTM3NiwyLjA4MjkxIDQuOTcyOTMyLDEuNzM1OTUgNy44MTQ3MjEsNi41OTYwNSA4LjUyNTM3
NSwxNC41ODAzMyB6IiAvPgogICAgICAgICAgPHBhdGgKICAgICAgICAgICAgIGlua3NjYXBlOmNv
bm5lY3Rvci1jdXJ2YXR1cmU9IjAiCiAgICAgICAgICAgICBpZD0icGF0aDMzNjAiCiAgICAgICAg
ICAgICBzdHlsZT0iZmlsbDojZTU3MDAwO2ZpbGwtb3BhY2l0eToxIgogICAgICAgICAgICAgZD0i
bSAzNDMuMjQ1ODEsLTI0My4wMTE1MiBjIC01Ljg3NzY0LC01Ljg3NzA4IC0xMi43MjM3NywtMTAu
NDk0OTkgLTIwLjUzODQsLTEzLjg1Mzc0IC03LjgxNTIsLTMuMzU4MjEgLTE2LjIxMTQsLTUuMDY5
NzQgLTI1LjE4ODYxLC01LjEzNDU5IC05LjU1MDkzLDAuMDcyOSAtMTguMzUwNzksMS45NjIwOCAt
MjYuMzk5Niw1LjY2NzQ0IC04LjA0OTI4LDMuNzA1ODkgLTE1LjAwODQzLDguNzkyMDUgLTIwLjg3
NzQ5LDE1LjI1ODQ4IGwgLTU4LjUxNTA4LDY0LjMyNzgzIC01OC45MDI1OSwtNjQuMzI3ODMgYyAt
Ni4wMjI3NSwtNi42MzU5OCAtMTIuOTY1NzYsLTExLjc3MDU4IC0yMC44MjkwNSwtMTUuNDAzODEg
LTcuODYzNDMsLTMuNjMyNyAtMTYuNTUwMjYyLC01LjQ3MzQxIC0yNi4wNjA1MTcsLTUuNTIyMTEg
LTguOTc3NTE5LDAuMDY0OSAtMTcuMzczNzE0LDEuNzc2MzkgLTI1LjE4ODYxLDUuMTM0NiAtNy44
MTQ5MzYsMy4zNTg3NCAtMTQuNjYxMDY0LDcuOTc2NjUgLTIwLjUzODQwNSwxMy44NTM3MyBMIDE0
NS45OTk2MiwtMTI2Ljc1NjM5IDQwLjIwNzQ1OCwtMTAuNTAxMjcxIGMgNS44NzczMzMsNi4wNjMw
MzcxIDEyLjcyMzQ2MSwxMC43OTM5NjcwNiAyMC41Mzg0MDUsMTQuMTkyODAzOCA3LjgxNDkwNCwz
LjM5ODg0NyAxNi4yMTEwOTksNS4xMjY1Mjg1IDI1LjE4ODYxLDUuMTgzMDQ5NSA5LjM4MTA5MSwt
MC4wNzI2NTkgMTguMTMyNTA3LC0xLjk2MTgwMzcgMjYuMjU0Mjg3LC01LjY2NzQ0MDEgOC4xMjE2
MywtMy43MDU2MjUwOCAxNS4xMjkyMywtOC43OTE3ODA2IDIxLjAyMjgsLTE1LjI1ODQ4MjIgbCA1
OC41MTUwNywtNjQuMzI3ODM0IDU4LjUxNTA4LDY0LjMyNzgzNCBjIDUuODY5MDYsNi40NjY3MDE2
IDEyLjgyODIxLDExLjU1Mjg1NTE5IDIwLjg3NzQ5LDE1LjI1ODQ3NjQgOC4wNDg4MSwzLjcwNTYz
MjYgMTYuODQ4NjcsNS41OTQ3NzkzIDI2LjM5OTYsNS42Njc0NDU5IDguOTc3MjEsLTAuMDU2NTEz
IDE3LjM3MzQsLTEuNzg0MTkyOSAyNS4xODg2MSwtNS4xODMwNDM4IDcuODE0NjQsLTMuMzk4ODQw
NTIgMTQuNjYwNzcsLTguMTI5NzcyNCAyMC41Mzg0LC0xNC4xOTI4MDk1IEwgMjM3LjQ1MzY1LC0x
MjYuNzU2MzkgeiIgLz4KICAgICAgICAgIDxwYXRoCiAgICAgICAgICAgICBpbmtzY2FwZTpjb25u
ZWN0b3ItY3VydmF0dXJlPSIwIgogICAgICAgICAgICAgaWQ9InBhdGgzMzYyIgogICAgICAgICAg
ICAgZD0ibSA2OTUuODE4MDMsLTI2Mi4zODczNyAtNDIuNjI2ODgsMCBjIC0xMy45NDI4MSwwLjE5
NDAzIC0yNi4zNTk0OSwzLjk3MjMyIC0zNy4yNTAwOCwxMS4zMzQ4NyAtMTAuODkxMDcsNy4zNjMw
OCAtMTkuMTQxOTQsMTcuMTQ3ODcgLTI0Ljc1MjY2LDI5LjM1NDQyIGwgMC4zODc1MiwtMC43NzUw
MyAtMzkuOTE0MjYsODcuNTc4ODYgLTM5LjUyNjc0LC04Ny41Nzg4NiAwLDAuNzc1MDMgYyAtNS40
NDE1MiwtMTIuMjA2NTUgLTEzLjY0Mzk1LC0yMS45OTEzNCAtMjQuNjA3MzQsLTI5LjM1NDQyIC0x
MC45NjM1OSwtNy4zNjI1NiAtMjMuNDI4NzEsLTExLjE0MDg0IC0zNy4zOTUzOSwtMTEuMzM0ODcg
bCAtNDIuMjM5MzYsMCBjIC03LjMzODYyLDAuMTk0MDMgLTEzLjM5MzU3LDIuNzEyODggLTE4LjE2
NDg3LDcuNTU2NTggLTQuNzcxMzEsNC44NDQyMiAtNy4yNDE3MywxMC44NTA3MyAtNy40MTEyNiwx
OC4wMTk1NSBsIDAsMjQ1LjY4NTgyMjMgYyAxOS4xMDEzNSwtMC40NzYzMjg5IDM1LjA1NDExLC03
LjA4MDI2MiA0Ny44NTgzNSwtMTkuODExODE5MyAxMi44MDQxNSwtMTIuNzMxNTE3IDE5LjQ1NjUy
LC0yOC43MzI3MjQgMTkuOTU3MTQsLTQ4LjAwMzY2OSBsIDAsLTEyOS44MTgyMjQgYyAwLjAyNDEs
LTEuNzY3ODQgMC41NTY5OSwtMy4xNzI1OSAxLjU5ODUxLC00LjIxNDI1IDEuMDQxMzgsLTEuMDQx
MjUgMi40NDYxMywtMS41NzQwOCA0LjIxNDI1LC0xLjU5ODUgMS4wMDkwOCwwLjAzMjUgMS45OTQw
MiwwLjM1NTQyIDIuOTU0ODIsMC45Njg3OCAwLjk2MDY0LDAuNjEzNzcgMS42NTQ5NCwxLjMyNDIy
IDIuMDgyOSwyLjEzMTM1IGwgNzUuMTc4MzEsMTY1Ljg1NzMxMyBjIDEuNDIwNzQsMi45NjI5MTkg
My41MTk3OSw1LjM2ODc0OSA2LjI5NzE1LDcuMjE3NDk2IDIuNzc3MDUsMS44NDg4MDMgNS44NDQ4
OSwyLjgwMTQ1MSA5LjIwMzUzLDIuODU3OTQ4IDMuMzM0MDksLTAuMDQwMzQgNi4zNTM0OSwtMC45
MjgzOTQgOS4wNTgyMSwtMi42NjQxNzcgMi43MDQzNiwtMS43MzU3MyA0Ljg1MTg1LC00LjA3Njk3
NyA2LjQ0MjQ4LC03LjAyMzc1IGwgNzUuMTc4MzEsLTE2Ni4yNDQ4MyBjIDAuNTg5MDksLTAuODA3
MTIgMS4zNDc5NywtMS41MTc1NyAyLjI3NjY2LC0yLjEzMTM1IDAuOTI4MTcsLTAuNjEzMzcgMS45
Nzc2OSwtMC45MzYzIDMuMTQ4NTgsLTAuOTY4NzggMS41ODIxLDAuMDI0NCAyLjg3MzgyLDAuNTU3
MjYgMy44NzUxNywxLjU5ODUxIDEuMDAwODEsMS4wNDE2NSAxLjUxNzUsMi40NDY0IDEuNTUwMDcs
NC4yMTQyNCBsIDAsMTI5LjgxODIyNCBjIDAuNTAwMjcsMTkuMjcwOTQ1IDcuMTUyNjQsMzUuMjcy
MTU0IDE5Ljk1NzEyLDQ4LjAwMzY3NSAxMi44MDM5LDEyLjczMTU2MSAyOC43NTY2NywxOS4zMzU0
OTIxIDQ3Ljg1ODM2LDE5LjgxMTgxMzMgbCAwLC0yNDUuNjg1ODIyMyBjIC0wLjE3Nzk0LC03LjE2
ODgyIC0yLjYzMjIxLC0xMy4xNzUzMyAtNy4zNjI4MiwtMTguMDE5NTUgLTQuNzMxMjcsLTQuODQz
NyAtMTAuNjczMTksLTcuMzYyNTUgLTE3LjgyNTc4LC03LjU1NjU4IHoiIC8+CiAgICAgICAgICA8
cGF0aAogICAgICAgICAgICAgaW5rc2NhcGU6Y29ubmVjdG9yLWN1cnZhdHVyZT0iMCIKICAgICAg
ICAgICAgIGlkPSJwYXRoMzM2NCIKICAgICAgICAgICAgIGQ9Im0gOTYzLjA1ODQsLTI2Mi4zODcz
NyAtMTM1LjI0MzQ2LDAgYyAtMTkuMjcwOTUsMC40OTI3NSAtMzUuMjcyMTYsNy4xNjEyNiAtNDgu
MDAzNjgsMjAuMDA1NTggLTEyLjczMTU2LDEyLjg0NDgxIC0xOS4zMzU0OSwyOC45MTA2MSAtMTku
ODExODEsNDguMTk3NDIgbCAwLDEzNS4yNDM0NjQgYyAwLjQ3NjMyLDE5LjI3MDk0NSA3LjA4MDI1
LDM1LjI3MjE1NCAxOS44MTE4MSw0OC4wMDM2NzUgMTIuNzMxNTMsMTIuNzMxNTYxIDI4LjczMjcz
LDE5LjMzNTQ5MjEgNDguMDAzNjgsMTkuODExODEzMyBsIDEzNS4yNDM0NiwwIGMgMTkuMjcwNjcs
LTAuNDc2MzI4OSAzNS4yNzE4OCwtNy4wODAyNjIgNDguMDAzNywtMTkuODExODE5MyAxMi43MzEz
LC0xMi43MzE1MTcgMTkuMzM1MiwtMjguNzMyNzI0IDE5LjgxMTgsLTQ4LjAwMzY2OSBsIDAsLTEz
NS4yNDM0NjQgYyAtMC40NzY2LC0xOS4yODY4MiAtNy4wODA1LC0zNS4zNTI2MiAtMTkuODExOCwt
NDguMTk3NDQgLTEyLjczMTgyLC0xMi44NDQzMiAtMjguNzMzMDMsLTE5LjUxMjgzIC00OC4wMDM3
LC0yMC4wMDU1NiB6IG0gMCwxODYuMzk1NzEzIGMgMC4zNTUwMiwwLjcxMDUzNiAtMC4zNTU0Mywz
LjU1MjMyNSAtMi4xMzEzNSw4LjUyNTM3NiAtMS43NzYzMyw0Ljk3MzIwMyAtNi43NDk0Niw3Ljgx
NDk5MiAtMTQuOTE5NCw4LjUyNTM3NSBsIC0xMDEuMTQxOTYsMCBjIC0wLjcxMDUzLDAuMzU1Mjk0
IC0zLjU1MjMyLC0wLjM1NTE1NiAtOC41MjUzOCwtMi4xMzEzNSAtNC45NzMyMSwtMS43NzYwNTQg
LTcuODE1LC02Ljc0OTE4MiAtOC41MjUzNywtMTQuOTE5NDAxIGwgMCwtMTAxLjUyOTQ3MyBjIC0w
LjM1NTI5LC0wLjY5NDExIDAuMzU1MTUsLTMuNDcxMzIgMi4xMzEzNCwtOC4zMzE2MiAxLjc3NjA0
LC00Ljg1OTkyIDYuNzQ5MTgsLTcuNjM3MTIgMTQuOTE5NDEsLTguMzMxNjIgbCAxMDEuMTQxOTYs
MCBjIDAuNzEwMjYsLTAuMzQ2OTQgMy41NTIwNSwwLjM0NzM2IDguNTI1MzcsMi4wODI5MSA0Ljk3
MjkzLDEuNzM1OTUgNy44MTQ3Miw2LjU5NjA1IDguNTI1MzgsMTQuNTgwMzMgeiIgLz4KICAgICAg
ICAgIDxwYXRoCiAgICAgICAgICAgICBpbmtzY2FwZTpjb25uZWN0b3ItY3VydmF0dXJlPSIwIgog
ICAgICAgICAgICAgaWQ9InBhdGgzMzY2IgogICAgICAgICAgICAgc3R5bGU9ImZpbGw6I2U1NzAw
MDtmaWxsLW9wYWNpdHk6MSIKICAgICAgICAgICAgIGQ9Im0gMTM3Mi42MDEzLC0yNDMuMDExNTIg
YyAtNS44Nzc3LC01Ljg3NzA4IC0xMi43MjM4LC0xMC40OTQ5OSAtMjAuNTM4NCwtMTMuODUzNzQg
LTcuODE1MiwtMy4zNTgyMSAtMTYuMjExNCwtNS4wNjk3NCAtMjUuMTg4NiwtNS4xMzQ1OSAtOS41
NTEsMC4wNzI5IC0xOC4zNTA4LDEuOTYyMDggLTI2LjM5OTYsNS42Njc0NCAtOC4wNDkzLDMuNzA1
ODkgLTE1LjAwODUsOC43OTIwNSAtMjAuODc3NSwxNS4yNTg0OCBsIC01OC41MTUxLDY0LjMyNzgz
IC01OC45MDI2LC02NC4zMjc4MyBjIC02LjAyMjcsLTYuNjM1OTggLTEyLjk2NTgsLTExLjc3MDU4
IC0yMC44MjksLTE1LjQwMzgxIC03Ljg2MzUsLTMuNjMyNyAtMTYuNTUwMywtNS40NzM0MSAtMjYu
MDYwNiwtNS41MjIxMSAtOC45Nzc1LDAuMDY0OSAtMTcuMzczNywxLjc3NjM5IC0yNS4xODg2LDUu
MTM0NiAtNy44MTQ5LDMuMzU4NzQgLTE0LjY2MSw3Ljk3NjY1IC0yMC41Mzg0LDEzLjg1MzczIGwg
MTA1Ljc5MjIsMTE2LjI1NTEzIC0xMDUuNzkyMiwxMTYuMjU1MTE5IGMgNS44Nzc0LDYuMDYzMDM3
MSAxMi43MjM1LDEwLjc5Mzk2NzA2IDIwLjUzODQsMTQuMTkyODAzOCA3LjgxNDksMy4zOTg4NDcg
MTYuMjExMSw1LjEyNjUyODUgMjUuMTg4Niw1LjE4MzA0OTUgOS4zODExLC0wLjA3MjY1OSAxOC4x
MzI2LC0xLjk2MTgwMzcgMjYuMjU0MywtNS42Njc0NDAxIDguMTIxNywtMy43MDU2MjUwOCAxNS4x
MjkzLC04Ljc5MTc4MDYgMjEuMDIyOCwtMTUuMjU4NDgyMiBsIDU4LjUxNTEsLTY0LjMyNzgzNCA1
OC41MTUxLDY0LjMyNzgzNCBjIDUuODY5LDYuNDY2NzAxNiAxMi44MjgyLDExLjU1Mjg1NTE5IDIw
Ljg3NzUsMTUuMjU4NDc2NCA4LjA0ODgsMy43MDU2MzI2IDE2Ljg0ODYsNS41OTQ3NzkzIDI2LjM5
OTYsNS42Njc0NDU5IDguOTc3MiwtMC4wNTY1MTMgMTcuMzczNCwtMS43ODQxOTI5IDI1LjE4ODYs
LTUuMTgzMDQzOCA3LjgxNDYsLTMuMzk4ODQwNTIgMTQuNjYwNywtOC4xMjk3NzI0IDIwLjUzODQs
LTE0LjE5MjgwOTUgTCAxMjY2LjgwOTEsLTEyNi43NTYzOSB6IiAvPgogICAgICAgIDwvZz4KICAg
ICAgPC9nPgogICAgPC9nPgogIDwvZz4KPC9zdmc+Cg==">
</span> <br> Proxmox VE Administration Guide</h1>
<span id="author">Proxmox Server Solutions GmbH</span><br>
<span id="email" class="monospaced">&lt;<a href="mailto:[email protected]">[email protected]</a>&gt;</span><br>
<span id="revnumber">version 8.2.3,</span>
<span id="revdate">Wed Jul 31 16:58:41 CEST 2024</span>
<div id="toc">
  <a href="index.html" style="font-size: 0.8em;">↩Index</a>
  <div id="toctitle">Table of Contents</div>
  <noscript><p><b>JavaScript must be enabled in your browser to display the table of contents.</b></p></noscript>
<div class="toclevel1"><a href="#_introduction">
1. Introduction
 
</a></div><div class="toclevel2"><a href="#intro_central_management">
1.1. Central Management
 
</a></div><div class="toclevel2"><a href="#_flexible_storage">
1.2. Flexible Storage
 
</a></div><div class="toclevel2"><a href="#_integrated_backup_and_restore">
1.3. Integrated Backup and Restore
 
</a></div><div class="toclevel2"><a href="#_high_availability_cluster">
1.4. High Availability Cluster
 
</a></div><div class="toclevel2"><a href="#_flexible_networking">
1.5. Flexible Networking
 
</a></div><div class="toclevel2"><a href="#_integrated_firewall">
1.6. Integrated Firewall
 
</a></div><div class="toclevel2"><a href="#chapter_hyper_converged_infrastructure">
1.7. Hyper-converged Infrastructure
 
</a></div><div class="toclevel2"><a href="#_why_open_source">
1.8. Why Open Source
 
</a></div><div class="toclevel2"><a href="#_your_benefits_with_proxmox_ve">
1.9. Your benefits with Proxmox VE
 
</a></div><div class="toclevel2"><a href="#getting_help">
1.10. Getting Help
 
</a></div><div class="toclevel2"><a href="#intro_project_history">
1.11. Project History
 
</a></div><div class="toclevel2"><a href="#howto_improve_pve_docs">
1.12. Improving the Proxmox VE Documentation
 
</a></div><div class="toclevel2"><a href="#translation">
1.13. Translating Proxmox VE
 
</a></div><div class="toclevel1"><a href="#chapter_installation">
2. Installing Proxmox VE
 
</a></div><div class="toclevel2"><a href="#_system_requirements">
2.1. System Requirements
 
</a></div><div class="toclevel2"><a href="#installation_prepare_media">
2.2. Prepare Installation Media
 
</a></div><div class="toclevel2"><a href="#installation_installer">
2.3. Using the Proxmox VE Installer
 
</a></div><div class="toclevel2"><a href="#installation_unattended">
2.4. Unattended Installation
 
</a></div><div class="toclevel2"><a href="#_install_proxmox_ve_on_debian">
2.5. Install Proxmox VE on Debian
 
</a></div><div class="toclevel1"><a href="#chapter_system_administration">
3. Host System Administration
 
</a></div><div class="toclevel2"><a href="#sysadmin_package_repositories">
3.1. Package Repositories
 
</a></div><div class="toclevel2"><a href="#system_software_updates">
3.2. System Software Updates
 
</a></div><div class="toclevel2"><a href="#chapter_firmware_updates">
3.3. Firmware Updates
 
</a></div><div class="toclevel2"><a href="#sysadmin_network_configuration">
3.4. Network Configuration
 
</a></div><div class="toclevel2"><a href="#_time_synchronization">
3.5. Time Synchronization
 
</a></div><div class="toclevel2"><a href="#external_metric_server">
3.6. External Metric Server
 
</a></div><div class="toclevel2"><a href="#_disk_health_monitoring">
3.7. Disk Health Monitoring
 
</a></div><div class="toclevel2"><a href="#chapter_lvm">
3.8. Logical Volume Manager (LVM)
 
</a></div><div class="toclevel2"><a href="#chapter_zfs">
3.9. ZFS on Linux
 
</a></div><div class="toclevel2"><a href="#chapter_btrfs">
3.10. BTRFS
 
</a></div><div class="toclevel2"><a href="#proxmox_node_management">
3.11. Proxmox Node Management
 
</a></div><div class="toclevel2"><a href="#sysadmin_certificate_management">
3.12. Certificate Management
 
</a></div><div class="toclevel2"><a href="#sysboot">
3.13. Host Bootloader
 
</a></div><div class="toclevel2"><a href="#kernel_samepage_merging">
3.14. Kernel Samepage Merging (KSM)
 
</a></div><div class="toclevel1"><a href="#chapter_gui">
4. Graphical User Interface
 
</a></div><div class="toclevel2"><a href="#_features">
4.1. Features
 
</a></div><div class="toclevel2"><a href="#_login">
4.2. Login
 
</a></div><div class="toclevel2"><a href="#_gui_overview">
4.3. GUI Overview
 
</a></div><div class="toclevel2"><a href="#_content_panels">
4.4. Content Panels
 
</a></div><div class="toclevel2"><a href="#_tags">
4.5. Tags
 
</a></div><div class="toclevel1"><a href="#chapter_pvecm">
5. Cluster Manager
 
</a></div><div class="toclevel2"><a href="#_requirements">
5.1. Requirements
 
</a></div><div class="toclevel2"><a href="#_preparing_nodes">
5.2. Preparing Nodes
 
</a></div><div class="toclevel2"><a href="#pvecm_create_cluster">
5.3. Create a Cluster
 
</a></div><div class="toclevel2"><a href="#pvecm_join_node_to_cluster">
5.4. Adding Nodes to the Cluster
 
</a></div><div class="toclevel2"><a href="#_remove_a_cluster_node">
5.5. Remove a Cluster Node
 
</a></div><div class="toclevel2"><a href="#_quorum">
5.6. Quorum
 
</a></div><div class="toclevel2"><a href="#_cluster_network">
5.7. Cluster Network
 
</a></div><div class="toclevel2"><a href="#pvecm_redundancy">
5.8. Corosync Redundancy
 
</a></div><div class="toclevel2"><a href="#_role_of_ssh_in_proxmox_ve_clusters">
5.9. Role of SSH in Proxmox VE Clusters
 
</a></div><div class="toclevel2"><a href="#_corosync_external_vote_support">
5.10. Corosync External Vote Support
 
</a></div><div class="toclevel2"><a href="#_corosync_configuration">
5.11. Corosync Configuration
 
</a></div><div class="toclevel2"><a href="#_cluster_cold_start">
5.12. Cluster Cold Start
 
</a></div><div class="toclevel2"><a href="#pvecm_next_id_range">
5.13. Guest VMID Auto-Selection
 
</a></div><div class="toclevel2"><a href="#_guest_migration">
5.14. Guest Migration
 
</a></div><div class="toclevel1"><a href="#chapter_pmxcfs">
6. Proxmox Cluster File System (pmxcfs)
 
</a></div><div class="toclevel2"><a href="#_posix_compatibility">
6.1. POSIX Compatibility
 
</a></div><div class="toclevel2"><a href="#_file_access_rights">
6.2. File Access Rights
 
</a></div><div class="toclevel2"><a href="#_technology">
6.3. Technology
 
</a></div><div class="toclevel2"><a href="#_file_system_layout">
6.4. File System Layout
 
</a></div><div class="toclevel2"><a href="#_recovery">
6.5. Recovery
 
</a></div><div class="toclevel1"><a href="#chapter_storage">
7. Proxmox VE Storage
 
</a></div><div class="toclevel2"><a href="#_storage_types">
7.1. Storage Types
 
</a></div><div class="toclevel2"><a href="#_storage_configuration">
7.2. Storage Configuration
 
</a></div><div class="toclevel2"><a href="#_volumes">
7.3. Volumes
 
</a></div><div class="toclevel2"><a href="#_using_the_command_line_interface">
7.4. Using the Command-line Interface
 
</a></div><div class="toclevel2"><a href="#storage_directory">
7.5. Directory Backend
 
</a></div><div class="toclevel2"><a href="#storage_nfs">
7.6. NFS Backend
 
</a></div><div class="toclevel2"><a href="#storage_cifs">
7.7. CIFS Backend
 
</a></div><div class="toclevel2"><a href="#storage_pbs">
7.8. Proxmox Backup Server
 
</a></div><div class="toclevel2"><a href="#storage_glusterfs">
7.9. GlusterFS Backend
 
</a></div><div class="toclevel2"><a href="#storage_zfspool">
7.10. Local ZFS Pool Backend
 
</a></div><div class="toclevel2"><a href="#storage_lvm">
7.11. LVM Backend
 
</a></div><div class="toclevel2"><a href="#storage_lvmthin">
7.12. LVM thin Backend
 
</a></div><div class="toclevel2"><a href="#storage_open_iscsi">
7.13. Open-iSCSI initiator
 
</a></div><div class="toclevel2"><a href="#storage_iscsidirect">
7.14. User Mode iSCSI Backend
 
</a></div><div class="toclevel2"><a href="#ceph_rados_block_devices">
7.15. Ceph RADOS Block Devices (RBD)
 
</a></div><div class="toclevel2"><a href="#storage_cephfs">
7.16. Ceph Filesystem (CephFS)
 
</a></div><div class="toclevel2"><a href="#storage_btrfs">
7.17. BTRFS Backend
 
</a></div><div class="toclevel2"><a href="#storage_zfs">
7.18. ZFS over ISCSI Backend
 
</a></div><div class="toclevel1"><a href="#chapter_pveceph">
8. Deploy Hyper-Converged Ceph Cluster
 
</a></div><div class="toclevel2"><a href="#_introduction_2">
8.1. Introduction
 
</a></div><div class="toclevel2"><a href="#_terminology">
8.2. Terminology
 
</a></div><div class="toclevel2"><a href="#_recommendations_for_a_healthy_ceph_cluster">
8.3. Recommendations for a Healthy Ceph Cluster
 
</a></div><div class="toclevel2"><a href="#pve_ceph_install_wizard">
8.4. Initial Ceph Installation &amp; Configuration
 
</a></div><div class="toclevel2"><a href="#pve_ceph_monitors">
8.5. Ceph Monitor
 
</a></div><div class="toclevel2"><a href="#pve_ceph_manager">
8.6. Ceph Manager
 
</a></div><div class="toclevel2"><a href="#pve_ceph_osds">
8.7. Ceph OSDs
 
</a></div><div class="toclevel2"><a href="#pve_ceph_pools">
8.8. Ceph Pools
 
</a></div><div class="toclevel2"><a href="#pve_ceph_device_classes">
8.9. Ceph CRUSH &amp; device classes
 
</a></div><div class="toclevel2"><a href="#_ceph_client">
8.10. Ceph Client
 
</a></div><div class="toclevel2"><a href="#pveceph_fs">
8.11. CephFS
 
</a></div><div class="toclevel2"><a href="#_ceph_maintenance">
8.12. Ceph maintenance
 
</a></div><div class="toclevel2"><a href="#_ceph_monitoring_and_troubleshooting">
8.13. Ceph Monitoring and Troubleshooting
 
</a></div><div class="toclevel1"><a href="#chapter_pvesr">
9. Storage Replication
 
</a></div><div class="toclevel2"><a href="#_supported_storage_types">
9.1. Supported Storage Types
 
</a></div><div class="toclevel2"><a href="#pvesr_schedule_time_format">
9.2. Schedule Format
 
</a></div><div class="toclevel2"><a href="#_error_handling">
9.3. Error Handling
 
</a></div><div class="toclevel2"><a href="#_managing_jobs">
9.4. Managing Jobs
 
</a></div><div class="toclevel2"><a href="#_command_line_interface_examples">
9.5. Command-line Interface Examples
 
</a></div><div class="toclevel1"><a href="#chapter_virtual_machines">
10. QEMU/KVM Virtual Machines
 
</a></div><div class="toclevel2"><a href="#_emulated_devices_and_paravirtualized_devices">
10.1. Emulated devices and paravirtualized devices
 
</a></div><div class="toclevel2"><a href="#qm_virtual_machines_settings">
10.2. Virtual Machines Settings
 
</a></div><div class="toclevel2"><a href="#qm_migration">
10.3. Migration
 
</a></div><div class="toclevel2"><a href="#qm_copy_and_clone">
10.4. Copies and Clones
 
</a></div><div class="toclevel2"><a href="#qm_templates">
10.5. Virtual Machine Templates
 
</a></div><div class="toclevel2"><a href="#_vm_generation_id">
10.6. VM Generation ID
 
</a></div><div class="toclevel2"><a href="#qm_import_virtual_machines">
10.7. Importing Virtual Machines
 
</a></div><div class="toclevel2"><a href="#qm_cloud_init">
10.8. Cloud-Init Support
 
</a></div><div class="toclevel2"><a href="#qm_pci_passthrough">
10.9. PCI(e) Passthrough
 
</a></div><div class="toclevel2"><a href="#_hookscripts">
10.10. Hookscripts
 
</a></div><div class="toclevel2"><a href="#qm_hibernate">
10.11. Hibernation
 
</a></div><div class="toclevel2"><a href="#resource_mapping">
10.12. Resource Mapping
 
</a></div><div class="toclevel2"><a href="#_managing_virtual_machines_with_span_class_monospaced_qm_span">
10.13. Managing Virtual Machines with qm
 
</a></div><div class="toclevel2"><a href="#qm_configuration">
10.14. Configuration
 
</a></div><div class="toclevel2"><a href="#_locks">
10.15. Locks
 
</a></div><div class="toclevel1"><a href="#chapter_pct">
11. Proxmox Container Toolkit
 
</a></div><div class="toclevel2"><a href="#_technology_overview">
11.1. Technology Overview
 
</a></div><div class="toclevel2"><a href="#pct_supported_distributions">
11.2. Supported Distributions
 
</a></div><div class="toclevel2"><a href="#pct_container_images">
11.3. Container Images
 
</a></div><div class="toclevel2"><a href="#pct_settings">
11.4. Container Settings
 
</a></div><div class="toclevel2"><a href="#_security_considerations">
11.5. Security Considerations
 
</a></div><div class="toclevel2"><a href="#_guest_operating_system_configuration">
11.6. Guest Operating System Configuration
 
</a></div><div class="toclevel2"><a href="#pct_container_storage">
11.7. Container Storage
 
</a></div><div class="toclevel2"><a href="#_backup_and_restore">
11.8. Backup and Restore
 
</a></div><div class="toclevel2"><a href="#_managing_containers_with_span_class_monospaced_pct_span">
11.9. Managing Containers with pct
 
</a></div><div class="toclevel2"><a href="#pct_migration">
11.10. Migration
 
</a></div><div class="toclevel2"><a href="#pct_configuration">
11.11. Configuration
 
</a></div><div class="toclevel2"><a href="#_locks_2">
11.12. Locks
 
</a></div><div class="toclevel1"><a href="#chapter_pvesdn">
12. Software-Defined Network
 
</a></div><div class="toclevel2"><a href="#pvesdn_overview">
12.1. Introduction
 
</a></div><div class="toclevel2"><a href="#pvesdn_support_status">
12.2. Support Status
 
</a></div><div class="toclevel2"><a href="#pvesdn_installation">
12.3. Installation
 
</a></div><div class="toclevel2"><a href="#pvesdn_main_configuration">
12.4. Configuration Overview
 
</a></div><div class="toclevel2"><a href="#pvesdn_tech_and_config_overview">
12.5. Technology &amp; Configuration
 
</a></div><div class="toclevel2"><a href="#pvesdn_config_zone">
12.6. Zones
 
</a></div><div class="toclevel2"><a href="#pvesdn_config_vnet">
12.7. VNets
 
</a></div><div class="toclevel2"><a href="#pvesdn_config_subnet">
12.8. Subnets
 
</a></div><div class="toclevel2"><a href="#pvesdn_config_controllers">
12.9. Controllers
 
</a></div><div class="toclevel2"><a href="#pvesdn_config_ipam">
12.10. IPAM
 
</a></div><div class="toclevel2"><a href="#pvesdn_config_dns">
12.11. DNS
 
</a></div><div class="toclevel2"><a href="#pvesdn_config_dhcp">
12.12. DHCP
 
</a></div><div class="toclevel2"><a href="#pvesdn_setup_examples">
12.13. Examples
 
</a></div><div class="toclevel2"><a href="#pvesdn_notes">
12.14. Notes
 
</a></div><div class="toclevel1"><a href="#chapter_pve_firewall">
13. Proxmox VE Firewall
 
</a></div><div class="toclevel2"><a href="#_zones">
13.1. Zones
 
</a></div><div class="toclevel2"><a href="#_configuration_files">
13.2. Configuration Files
 
</a></div><div class="toclevel2"><a href="#_firewall_rules">
13.3. Firewall Rules
 
</a></div><div class="toclevel2"><a href="#pve_firewall_security_groups">
13.4. Security Groups
 
</a></div><div class="toclevel2"><a href="#pve_firewall_ip_aliases">
13.5. IP Aliases
 
</a></div><div class="toclevel2"><a href="#pve_firewall_ip_sets">
13.6. IP Sets
 
</a></div><div class="toclevel2"><a href="#pve_firewall_services_commands">
13.7. Services and Commands
 
</a></div><div class="toclevel2"><a href="#pve_firewall_default_rules">
13.8. Default firewall rules
 
</a></div><div class="toclevel2"><a href="#_logging_of_firewall_rules">
13.9. Logging of firewall rules
 
</a></div><div class="toclevel2"><a href="#_tips_and_tricks">
13.10. Tips and Tricks
 
</a></div><div class="toclevel2"><a href="#_notes_on_ipv6">
13.11. Notes on IPv6
 
</a></div><div class="toclevel2"><a href="#_ports_used_by_proxmox_ve">
13.12. Ports used by Proxmox VE
 
</a></div><div class="toclevel2"><a href="#_nftables">
13.13. nftables
 
</a></div><div class="toclevel1"><a href="#user_mgmt">
14. User Management
 
</a></div><div class="toclevel2"><a href="#pveum_users">
14.1. Users
 
</a></div><div class="toclevel2"><a href="#pveum_groups">
14.2. Groups
 
</a></div><div class="toclevel2"><a href="#pveum_tokens">
14.3. API Tokens
 
</a></div><div class="toclevel2"><a href="#pveum_resource_pools">
14.4. Resource Pools
 
</a></div><div class="toclevel2"><a href="#pveum_authentication_realms">
14.5. Authentication Realms
 
</a></div><div class="toclevel2"><a href="#pveum_tfa_auth">
14.6. Two-Factor Authentication
 
</a></div><div class="toclevel2"><a href="#pveum_permission_management">
14.7. Permission Management
 
</a></div><div class="toclevel2"><a href="#_command_line_tool">
14.8. Command-line Tool
 
</a></div><div class="toclevel2"><a href="#_real_world_examples">
14.9. Real World Examples
 
</a></div><div class="toclevel1"><a href="#chapter_ha_manager">
15. High Availability
 
</a></div><div class="toclevel2"><a href="#_requirements_3">
15.1. Requirements
 
</a></div><div class="toclevel2"><a href="#ha_manager_resources">
15.2. Resources
 
</a></div><div class="toclevel2"><a href="#_management_tasks">
15.3. Management Tasks
 
</a></div><div class="toclevel2"><a href="#_how_it_works_2">
15.4. How It Works
 
</a></div><div class="toclevel2"><a href="#_ha_simulator">
15.5. HA Simulator
 
</a></div><div class="toclevel2"><a href="#_configuration_16">
15.6. Configuration
 
</a></div><div class="toclevel2"><a href="#ha_manager_fencing">
15.7. Fencing
 
</a></div><div class="toclevel2"><a href="#ha_manager_start_failure_policy">
15.8. Start Failure Policy
 
</a></div><div class="toclevel2"><a href="#ha_manager_error_recovery">
15.9. Error Recovery
 
</a></div><div class="toclevel2"><a href="#ha_manager_package_updates">
15.10. Package Updates
 
</a></div><div class="toclevel2"><a href="#ha_manager_node_maintenance">
15.11. Node Maintenance
 
</a></div><div class="toclevel2"><a href="#ha_manager_crs">
15.12. Cluster Resource Scheduling
 
</a></div><div class="toclevel1"><a href="#chapter_vzdump">
16. Backup and Restore
 
</a></div><div class="toclevel2"><a href="#_backup_modes">
16.1. Backup Modes
 
</a></div><div class="toclevel2"><a href="#_backup_file_names">
16.2. Backup File Names
 
</a></div><div class="toclevel2"><a href="#_backup_file_compression">
16.3. Backup File Compression
 
</a></div><div class="toclevel2"><a href="#_backup_encryption">
16.4. Backup Encryption
 
</a></div><div class="toclevel2"><a href="#vzdump_jobs">
16.5. Backup Jobs
 
</a></div><div class="toclevel2"><a href="#vzdump_retention">
16.6. Backup Retention
 
</a></div><div class="toclevel2"><a href="#vzdump_protection">
16.7. Backup Protection
 
</a></div><div class="toclevel2"><a href="#vzdump_notes">
16.8. Backup Notes
 
</a></div><div class="toclevel2"><a href="#vzdump_restore">
16.9. Restore
 
</a></div><div class="toclevel2"><a href="#vzdump_configuration">
16.10. Configuration
 
</a></div><div class="toclevel2"><a href="#_hook_scripts">
16.11. Hook Scripts
 
</a></div><div class="toclevel2"><a href="#_file_exclusions">
16.12. File Exclusions
 
</a></div><div class="toclevel2"><a href="#_examples_10">
16.13. Examples
 
</a></div><div class="toclevel1"><a href="#chapter_notifications">
17. Notifications
 
</a></div><div class="toclevel2"><a href="#_overview">
17.1. Overview
 
</a></div><div class="toclevel2"><a href="#notification_targets">
17.2. Notification Targets
 
</a></div><div class="toclevel2"><a href="#notification_matchers">
17.3. Notification Matchers
 
</a></div><div class="toclevel2"><a href="#notification_events">
17.4. Notification Events
 
</a></div><div class="toclevel2"><a href="#_system_mail_forwarding">
17.5. System Mail Forwarding
 
</a></div><div class="toclevel2"><a href="#_permissions_2">
17.6. Permissions
 
</a></div><div class="toclevel2"><a href="#notification_mode">
17.7. Notification Mode
 
</a></div><div class="toclevel1"><a href="#_important_service_daemons">
18. Important Service Daemons
 
</a></div><div class="toclevel2"><a href="#_pvedaemon_proxmox_ve_api_daemon">
18.1. pvedaemon - Proxmox VE API Daemon
 
</a></div><div class="toclevel2"><a href="#_pveproxy_proxmox_ve_api_proxy_daemon">
18.2. pveproxy - Proxmox VE API Proxy Daemon
 
</a></div><div class="toclevel2"><a href="#_pvestatd_proxmox_ve_status_daemon">
18.3. pvestatd - Proxmox VE Status Daemon
 
</a></div><div class="toclevel2"><a href="#_spiceproxy_spice_proxy_service">
18.4. spiceproxy - SPICE Proxy Service
 
</a></div><div class="toclevel2"><a href="#_pvescheduler_proxmox_ve_scheduler_daemon">
18.5. pvescheduler - Proxmox VE Scheduler Daemon
 
</a></div><div class="toclevel1"><a href="#_useful_command_line_tools">
19. Useful Command-line Tools
 
</a></div><div class="toclevel2"><a href="#_pvesubscription_subscription_management">
19.1. pvesubscription - Subscription Management
 
</a></div><div class="toclevel2"><a href="#_pveperf_proxmox_ve_benchmark_script">
19.2. pveperf - Proxmox VE Benchmark Script
 
</a></div><div class="toclevel2"><a href="#_shell_interface_for_the_proxmox_ve_api">
19.3. Shell interface for the Proxmox VE API
 
</a></div><div class="toclevel1"><a href="#_frequently_asked_questions_2">
20. Frequently Asked Questions
 
</a></div><div class="toclevel1"><a href="#_bibliography">
21. Bibliography
 
</a></div><div class="toclevel1"><a href="#_command_line_interface">22. Appendix A: Command-line Interface</a></div><div class="toclevel2"><a href="#_output_format_options_span_class_monospaced_format_options_span">
22.1. Output format options [FORMAT_OPTIONS]
 
</a></div><div class="toclevel2"><a href="#_strong_pvesm_strong_proxmox_ve_storage_manager">
22.2. pvesm - Proxmox VE Storage Manager
 
</a></div><div class="toclevel2"><a href="#_strong_pvesubscription_strong_proxmox_ve_subscription_manager">
22.3. pvesubscription - Proxmox VE Subscription Manager
 
</a></div><div class="toclevel2"><a href="#_strong_pveperf_strong_proxmox_ve_benchmark_script">
22.4. pveperf - Proxmox VE Benchmark Script
 
</a></div><div class="toclevel2"><a href="#_strong_pveceph_strong_manage_ceph_services_on_proxmox_ve_nodes">
22.5. pveceph - Manage CEPH Services on Proxmox VE Nodes
 
</a></div><div class="toclevel2"><a href="#_strong_pvenode_strong_proxmox_ve_node_management">
22.6. pvenode - Proxmox VE Node Management
 
</a></div><div class="toclevel2"><a href="#_strong_pvesh_strong_shell_interface_for_the_proxmox_ve_api">
22.7. pvesh - Shell interface for the Proxmox VE API
 
</a></div><div class="toclevel2"><a href="#_strong_qm_strong_qemu_kvm_virtual_machine_manager">
22.8. qm - QEMU/KVM Virtual Machine Manager
 
</a></div><div class="toclevel2"><a href="#_strong_qmrestore_strong_restore_qemuserver_span_class_monospaced_vzdump_span_backups">
22.9. qmrestore - Restore QemuServer vzdump Backups
 
</a></div><div class="toclevel2"><a href="#_strong_pct_strong_proxmox_container_toolkit">
22.10. pct - Proxmox Container Toolkit
 
</a></div><div class="toclevel2"><a href="#_strong_pveam_strong_proxmox_ve_appliance_manager">
22.11. pveam -  Proxmox VE Appliance Manager
 
</a></div><div class="toclevel2"><a href="#_strong_pvecm_strong_proxmox_ve_cluster_manager">
22.12. pvecm - Proxmox VE Cluster Manager
 
</a></div><div class="toclevel2"><a href="#_strong_pvesr_strong_proxmox_ve_storage_replication">
22.13. pvesr - Proxmox VE Storage Replication
 
</a></div><div class="toclevel2"><a href="#_strong_pveum_strong_proxmox_ve_user_manager">
22.14. pveum - Proxmox VE User Manager
 
</a></div><div class="toclevel2"><a href="#_strong_vzdump_strong_backup_utility_for_vms_and_containers">
22.15. vzdump - Backup Utility for VMs and Containers
 
</a></div><div class="toclevel2"><a href="#_strong_ha_manager_strong_proxmox_ve_ha_manager">
22.16. ha-manager - Proxmox VE HA Manager
 
</a></div><div class="toclevel1"><a href="#_service_daemons">23. Appendix B: Service Daemons</a></div><div class="toclevel2"><a href="#_strong_pve_firewall_strong_proxmox_ve_firewall_daemon">
23.1. pve-firewall - Proxmox VE Firewall Daemon
 
</a></div><div class="toclevel2"><a href="#_strong_pvedaemon_strong_proxmox_ve_api_daemon">
23.2. pvedaemon - Proxmox VE API Daemon
 
</a></div><div class="toclevel2"><a href="#_strong_pveproxy_strong_proxmox_ve_api_proxy_daemon">
23.3. pveproxy - Proxmox VE API Proxy Daemon
 
</a></div><div class="toclevel2"><a href="#_strong_pvestatd_strong_proxmox_ve_status_daemon">
23.4. pvestatd - Proxmox VE Status Daemon
 
</a></div><div class="toclevel2"><a href="#_strong_spiceproxy_strong_spice_proxy_service">
23.5. spiceproxy - SPICE Proxy Service
 
</a></div><div class="toclevel2"><a href="#_strong_pmxcfs_strong_proxmox_cluster_file_system">
23.6. pmxcfs - Proxmox Cluster File System
 
</a></div><div class="toclevel2"><a href="#_strong_pve_ha_crm_strong_cluster_resource_manager_daemon">
23.7. pve-ha-crm - Cluster Resource Manager Daemon
 
</a></div><div class="toclevel2"><a href="#_strong_pve_ha_lrm_strong_local_resource_manager_daemon">
23.8. pve-ha-lrm - Local Resource Manager Daemon
 
</a></div><div class="toclevel2"><a href="#_strong_pvescheduler_strong_proxmox_ve_scheduler_daemon">
23.9. pvescheduler - Proxmox VE Scheduler Daemon
 
</a></div><div class="toclevel1"><a href="#_configuration_files_2">24. Appendix C: Configuration Files</a></div><div class="toclevel2"><a href="#datacenter_configuration_file">
24.1. Datacenter Configuration
 
</a></div><div class="toclevel1"><a href="#_calendar_events">25. Appendix D: Calendar Events</a></div><div class="toclevel2"><a href="#chapter_calendar_events">
25.1. Schedule Format
 
</a></div><div class="toclevel2"><a href="#_detailed_specification">
25.2. Detailed Specification
 
</a></div><div class="toclevel1"><a href="#_qemu_vcpu_list">26. Appendix E: QEMU vCPU List</a></div><div class="toclevel2"><a href="#chapter_qm_vcpu_list">
26.1. Introduction
 
</a></div><div class="toclevel2"><a href="#_intel_cpu_types">
26.2. Intel CPU Types
 
</a></div><div class="toclevel2"><a href="#_amd_cpu_types">
26.3. AMD CPU Types
 
</a></div><div class="toclevel1"><a href="#_firewall_macro_definitions">27. Appendix F: Firewall Macro Definitions</a></div><div class="toclevel1"><a href="#_markdown_primer">28. Appendix G: Markdown Primer</a></div><div class="toclevel2"><a href="#markdown_basics">
28.1. Markdown Basics
 
</a></div><div class="toclevel1"><a href="#_gnu_free_documentation_license">29. Appendix H: GNU Free Documentation License</a></div></div>
</div>
<div id="content">
<div class="sect1">
<h2 id="_introduction">
1. Introduction
 <a class="headerlink" href="#_introduction" title="Permalink to this heading"></a>
</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Proxmox VE is a platform to run virtual machines and containers. It is
based on Debian Linux, and completely open source. For maximum
flexibility, we implemented two virtualization technologies -
Kernel-based Virtual Machine (KVM) and container-based virtualization
(LXC).</p></div>
<div class="paragraph">
<p>One main design goal was to make administration as easy as
possible. You can use Proxmox VE on a single node, or assemble a cluster of
many nodes. All management tasks can be done using our web-based
management interface, and even a novice user can setup and install
Proxmox VE within minutes.</p></div>
<div class="imageblock" style="text-align:center;">
<div class="content">
<img alt="Proxmox Software Stack" src="data:image/svg+xml;base64,
PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiIHN0YW5kYWxvbmU9Im5vIj8+Cjwh
RE9DVFlQRSBzdmcgUFVCTElDICItLy9XM0MvL0RURCBTVkcgMS4xLy9FTiIKICJodHRwOi8vd3d3
LnczLm9yZy9HcmFwaGljcy9TVkcvMS4xL0RURC9zdmcxMS5kdGQiPgo8IS0tIEdlbmVyYXRlZCBi
eSBncmFwaHZpeiB2ZXJzaW9uIDIuMzguMCAoMjAxNDA0MTMuMjA0MSkKIC0tPgo8IS0tIFRpdGxl
OiBwdmVfc29mdHdhcmVfc3RhY2sgUGFnZXM6IDEgLS0+Cjxzdmcgd2lkdGg9IjUyNnB0IiBoZWln
aHQ9IjQwMXB0Igogdmlld0JveD0iMC4wMCAwLjAwIDUyNi4wMCA0MDEuMDAiIHhtbG5zPSJodHRw
Oi8vd3d3LnczLm9yZy8yMDAwL3N2ZyIgeG1sbnM6eGxpbms9Imh0dHA6Ly93d3cudzMub3JnLzE5
OTkveGxpbmsiPgo8ZyBpZD0iZ3JhcGgwIiBjbGFzcz0iZ3JhcGgiIHRyYW5zZm9ybT0ic2NhbGUo
MSAxKSByb3RhdGUoMCkgdHJhbnNsYXRlKDQgMzk3KSI+Cjx0aXRsZT5wdmVfc29mdHdhcmVfc3Rh
Y2s8L3RpdGxlPgo8cG9seWdvbiBmaWxsPSJub25lIiBzdHJva2U9Im5vbmUiIHBvaW50cz0iLTQs
NCAtNCwtMzk3IDUyMiwtMzk3IDUyMiw0IC00LDQiLz4KPCEtLSBzdGFjayAtLT4KPGcgaWQ9Im5v
ZGUxIiBjbGFzcz0ibm9kZSI+PHRpdGxlPnN0YWNrPC90aXRsZT4KPHBvbHlnb24gZmlsbD0iIzAw
NjE3ZiIgc3Ryb2tlPSJub25lIiBwb2ludHM9IjEyLC0yODkuNSAxMiwtMzg0LjUgNTA2LC0zODQu
NSA1MDYsLTI4OS41IDEyLC0yODkuNSIvPgo8dGV4dCB0ZXh0LWFuY2hvcj0ic3RhcnQiIHg9IjIx
MC41IiB5PSItMzY2LjUiIGZvbnQtZmFtaWx5PSJIZWx2ZXRpY2Esc2Fucy1TZXJpZiIgZm9udC1z
aXplPSIyMC4wMCIgZmlsbD0id2hpdGUiPlVzZXIgVG9vbHM8L3RleHQ+Cjxwb2x5Z29uIGZpbGw9
IiMwMDYxN2YiIHN0cm9rZT0ibm9uZSIgcG9pbnRzPSIxNCwtMjkxLjUgMTQsLTM1Ni41IDUwNCwt
MzU2LjUgNTA0LC0yOTEuNSAxNCwtMjkxLjUiLz4KPHBvbHlnb24gZmlsbD0iI2ZmOTEwMCIgc3Ry
b2tlPSJub25lIiBwb2ludHM9IjE5LC0zMjYuNSAxOSwtMzUxLjUgMTA5LC0zNTEuNSAxMDksLTMy
Ni41IDE5LC0zMjYuNSIvPgo8dGV4dCB0ZXh0LWFuY2hvcj0ic3RhcnQiIHg9IjU0IiB5PSItMzM1
LjMiIGZvbnQtZmFtaWx5PSJIZWx2ZXRpY2Esc2Fucy1TZXJpZiIgZm9udC1zaXplPSIxNC4wMCIg
ZmlsbD0id2hpdGUiPnFtPC90ZXh0Pgo8cG9seWdvbiBmaWxsPSIjZmY5MTAwIiBzdHJva2U9Im5v
bmUiIHBvaW50cz0iMTE0LC0zMjYuNSAxMTQsLTM1MS41IDIyNSwtMzUxLjUgMjI1LC0zMjYuNSAx
MTQsLTMyNi41Ii8+Cjx0ZXh0IHRleHQtYW5jaG9yPSJzdGFydCIgeD0iMTQ5IiB5PSItMzM1LjMi
IGZvbnQtZmFtaWx5PSJIZWx2ZXRpY2Esc2Fucy1TZXJpZiIgZm9udC1zaXplPSIxNC4wMCIgZmls
bD0id2hpdGUiPnB2ZXNtPC90ZXh0Pgo8cG9seWdvbiBmaWxsPSIjZmY5MTAwIiBzdHJva2U9Im5v
bmUiIHBvaW50cz0iMjMwLC0zMjYuNSAyMzAsLTM1MS41IDM1MSwtMzUxLjUgMzUxLC0zMjYuNSAy
MzAsLTMyNi41Ii8+Cjx0ZXh0IHRleHQtYW5jaG9yPSJzdGFydCIgeD0iMjY5LjUiIHk9Ii0zMzUu
MyIgZm9udC1mYW1pbHk9IkhlbHZldGljYSxzYW5zLVNlcmlmIiBmb250LXNpemU9IjE0LjAwIiBm
aWxsPSJ3aGl0ZSI+cHZldW08L3RleHQ+Cjxwb2x5Z29uIGZpbGw9IiNmZjkxMDAiIHN0cm9rZT0i
bm9uZSIgcG9pbnRzPSIzNTYsLTMyNi41IDM1NiwtMzUxLjUgNDk5LC0zNTEuNSA0OTksLTMyNi41
IDM1NiwtMzI2LjUiLz4KPHRleHQgdGV4dC1hbmNob3I9InN0YXJ0IiB4PSIzOTAuNSIgeT0iLTMz
NS4zIiBmb250LWZhbWlseT0iSGVsdmV0aWNhLHNhbnMtU2VyaWYiIGZvbnQtc2l6ZT0iMTQuMDAi
IGZpbGw9IndoaXRlIj5oYSYjNDU7bWFuYWdlcjwvdGV4dD4KPHBvbHlnb24gZmlsbD0iI2ZmOTEw
MCIgc3Ryb2tlPSJub25lIiBwb2ludHM9IjE5LC0yOTYuNSAxOSwtMzIxLjUgMTA5LC0zMjEuNSAx
MDksLTI5Ni41IDE5LC0yOTYuNSIvPgo8dGV4dCB0ZXh0LWFuY2hvcj0ic3RhcnQiIHg9IjU0LjUi
IHk9Ii0zMDUuMyIgZm9udC1mYW1pbHk9IkhlbHZldGljYSxzYW5zLVNlcmlmIiBmb250LXNpemU9
IjE0LjAwIiBmaWxsPSJ3aGl0ZSI+cGN0PC90ZXh0Pgo8cG9seWdvbiBmaWxsPSIjZmY5MTAwIiBz
dHJva2U9Im5vbmUiIHBvaW50cz0iMTE0LC0yOTYuNSAxMTQsLTMyMS41IDIyNSwtMzIxLjUgMjI1
LC0yOTYuNSAxMTQsLTI5Ni41Ii8+Cjx0ZXh0IHRleHQtYW5jaG9yPSJzdGFydCIgeD0iMTQ5IiB5
PSItMzA1LjMiIGZvbnQtZmFtaWx5PSJIZWx2ZXRpY2Esc2Fucy1TZXJpZiIgZm9udC1zaXplPSIx
NC4wMCIgZmlsbD0id2hpdGUiPnB2ZWNtPC90ZXh0Pgo8cG9seWdvbiBmaWxsPSIjZmY5MTAwIiBz
dHJva2U9Im5vbmUiIHBvaW50cz0iMjMwLC0yOTYuNSAyMzAsLTMyMS41IDM1MSwtMzIxLjUgMzUx
LC0yOTYuNSAyMzAsLTI5Ni41Ii8+Cjx0ZXh0IHRleHQtYW5jaG9yPSJzdGFydCIgeD0iMjY0LjUi
IHk9Ii0zMDUuMyIgZm9udC1mYW1pbHk9IkhlbHZldGljYSxzYW5zLVNlcmlmIiBmb250LXNpemU9
IjE0LjAwIiBmaWxsPSJ3aGl0ZSI+cHZlY2VwaDwvdGV4dD4KPHBvbHlnb24gZmlsbD0iI2ZmOTEw
MCIgc3Ryb2tlPSJub25lIiBwb2ludHM9IjM1NiwtMjk2LjUgMzU2LC0zMjEuNSA0OTksLTMyMS41
IDQ5OSwtMjk2LjUgMzU2LC0yOTYuNSIvPgo8dGV4dCB0ZXh0LWFuY2hvcj0ic3RhcnQiIHg9IjM5
MyIgeT0iLTMwNS4zIiBmb250LWZhbWlseT0iSGVsdmV0aWNhLHNhbnMtU2VyaWYiIGZvbnQtc2l6
ZT0iMTQuMDAiIGZpbGw9IndoaXRlIj5wdmUmIzQ1O2ZpcmV3YWxsPC90ZXh0Pgo8cG9seWdvbiBm
aWxsPSIjMDA2MTdmIiBzdHJva2U9Im5vbmUiIHBvaW50cz0iMTIsLTIxOC41IDEyLC0yODMuNSA1
MDYsLTI4My41IDUwNiwtMjE4LjUgMTIsLTIxOC41Ii8+Cjx0ZXh0IHRleHQtYW5jaG9yPSJzdGFy
dCIgeD0iMjIwLjUiIHk9Ii0yNjUuNSIgZm9udC1mYW1pbHk9IkhlbHZldGljYSxzYW5zLVNlcmlm
IiBmb250LXNpemU9IjIwLjAwIiBmaWxsPSJ3aGl0ZSI+U2VydmljZXM8L3RleHQ+Cjxwb2x5Z29u
IGZpbGw9IiMwMDYxN2YiIHN0cm9rZT0ibm9uZSIgcG9pbnRzPSIxNCwtMjIwLjUgMTQsLTI1NS41
IDUwNCwtMjU1LjUgNTA0LC0yMjAuNSAxNCwtMjIwLjUiLz4KPHBvbHlnb24gZmlsbD0iI2U1NzAw
MCIgc3Ryb2tlPSJub25lIiBwb2ludHM9IjE5LC0yMjUuNSAxOSwtMjUwLjUgMTA0LC0yNTAuNSAx
MDQsLTIyNS41IDE5LC0yMjUuNSIvPgo8dGV4dCB0ZXh0LWFuY2hvcj0ic3RhcnQiIHg9IjM0IiB5
PSItMjM0LjMiIGZvbnQtZmFtaWx5PSJIZWx2ZXRpY2Esc2Fucy1TZXJpZiIgZm9udC1zaXplPSIx
NC4wMCIgZmlsbD0id2hpdGUiPnB2ZXByb3h5PC90ZXh0Pgo8cG9seWdvbiBmaWxsPSIjZTU3MDAw
IiBzdHJva2U9Im5vbmUiIHBvaW50cz0iMTA5LC0yMjUuNSAxMDksLTI1MC41IDIxMSwtMjUwLjUg
MjExLC0yMjUuNSAxMDksLTIyNS41Ii8+Cjx0ZXh0IHRleHQtYW5jaG9yPSJzdGFydCIgeD0iMTI0
IiB5PSItMjM0LjMiIGZvbnQtZmFtaWx5PSJIZWx2ZXRpY2Esc2Fucy1TZXJpZiIgZm9udC1zaXpl
PSIxNC4wMCIgZmlsbD0id2hpdGUiPnB2ZWRhZW1vbjwvdGV4dD4KPHBvbHlnb24gZmlsbD0iI2U1
NzAwMCIgc3Ryb2tlPSJub25lIiBwb2ludHM9IjIxNiwtMjI1LjUgMjE2LC0yNTAuNSAyOTgsLTI1
MC41IDI5OCwtMjI1LjUgMjE2LC0yMjUuNSIvPgo8dGV4dCB0ZXh0LWFuY2hvcj0ic3RhcnQiIHg9
IjIzMSIgeT0iLTIzNC4zIiBmb250LWZhbWlseT0iSGVsdmV0aWNhLHNhbnMtU2VyaWYiIGZvbnQt
c2l6ZT0iMTQuMDAiIGZpbGw9IndoaXRlIj5wdmVzdGF0ZDwvdGV4dD4KPHBvbHlnb24gZmlsbD0i
I2U1NzAwMCIgc3Ryb2tlPSJub25lIiBwb2ludHM9IjMwMywtMjI1LjUgMzAzLC0yNTAuNSAzOTgs
LTI1MC41IDM5OCwtMjI1LjUgMzAzLC0yMjUuNSIvPgo8dGV4dCB0ZXh0LWFuY2hvcj0ic3RhcnQi
IHg9IjMxNy41IiB5PSItMjM0LjMiIGZvbnQtZmFtaWx5PSJIZWx2ZXRpY2Esc2Fucy1TZXJpZiIg
Zm9udC1zaXplPSIxNC4wMCIgZmlsbD0id2hpdGUiPnB2ZSYjNDU7aGEmIzQ1O2xybTwvdGV4dD4K
PHBvbHlnb24gZmlsbD0iI2U1NzAwMCIgc3Ryb2tlPSJub25lIiBwb2ludHM9IjQwMywtMjI1LjUg
NDAzLC0yNTAuNSA0OTksLTI1MC41IDQ5OSwtMjI1LjUgNDAzLC0yMjUuNSIvPgo8dGV4dCB0ZXh0
LWFuY2hvcj0ic3RhcnQiIHg9IjQxNy41IiB5PSItMjM0LjMiIGZvbnQtZmFtaWx5PSJIZWx2ZXRp
Y2Esc2Fucy1TZXJpZiIgZm9udC1zaXplPSIxNC4wMCIgZmlsbD0id2hpdGUiPnB2ZSYjNDU7Y2x1
c3RlcjwvdGV4dD4KPHBvbHlnb24gZmlsbD0iI2FiYmFiYSIgc3Ryb2tlPSJub25lIiBwb2ludHM9
IjIwLC05NS41IDIwLC0yMDQuNSAxMzMsLTIwNC41IDEzMywtOTUuNSAyMCwtOTUuNSIvPgo8dGV4
dCB0ZXh0LWFuY2hvcj0ic3RhcnQiIHg9IjYxIiB5PSItMTc4LjUiIGZvbnQtZmFtaWx5PSJIZWx2
ZXRpY2Esc2Fucy1TZXJpZiIgZm9udC1zaXplPSIyMC4wMCIgZmlsbD0id2hpdGUiPlZNPC90ZXh0
Pgo8cG9seWdvbiBmaWxsPSJ3aGl0ZSIgc3Ryb2tlPSJub25lIiBwb2ludHM9IjI1LC0xMzcuNSAy
NSwtMTYyLjUgNzQsLTE2Mi41IDc0LC0xMzcuNSAyNSwtMTM3LjUiLz4KPHRleHQgdGV4dC1hbmNo
b3I9InN0YXJ0IiB4PSIzNyIgeT0iLTE0Ni4zIiBmb250LWZhbWlseT0iSGVsdmV0aWNhLHNhbnMt
U2VyaWYiIGZvbnQtc2l6ZT0iMTQuMDAiIGZpbGw9IiMwMDYxN2YiPkFwcDwvdGV4dD4KPHBvbHln
b24gZmlsbD0id2hpdGUiIHN0cm9rZT0ibm9uZSIgcG9pbnRzPSI3OSwtMTM3LjUgNzksLTE2Mi41
IDEyOCwtMTYyLjUgMTI4LC0xMzcuNSA3OSwtMTM3LjUiLz4KPHRleHQgdGV4dC1hbmNob3I9InN0
YXJ0IiB4PSI5MSIgeT0iLTE0Ni4zIiBmb250LWZhbWlseT0iSGVsdmV0aWNhLHNhbnMtU2VyaWYi
IGZvbnQtc2l6ZT0iMTQuMDAiIGZpbGw9IiMwMDYxN2YiPkFwcDwvdGV4dD4KPHBvbHlnb24gZmls
bD0iIzAwNjE3ZiIgc3Ryb2tlPSJub25lIiBwb2ludHM9IjI1LC0xMDAuNSAyNSwtMTMyLjUgMTI4
LC0xMzIuNSAxMjgsLTEwMC41IDI1LC0xMDAuNSIvPgo8dGV4dCB0ZXh0LWFuY2hvcj0ic3RhcnQi
IHg9IjMyLjUiIHk9Ii0xMTEuNSIgZm9udC1mYW1pbHk9IkhlbHZldGljYSxzYW5zLVNlcmlmIiBm
b250LXNpemU9IjIwLjAwIiBmaWxsPSJ3aGl0ZSI+R3Vlc3QgT1M8L3RleHQ+Cjxwb2x5Z29uIGZp
bGw9IiNhYmJhYmEiIHN0cm9rZT0ibm9uZSIgcG9pbnRzPSIxMzksLTk1LjUgMTM5LC0yMDQuNSAy
NTIsLTIwNC41IDI1MiwtOTUuNSAxMzksLTk1LjUiLz4KPHRleHQgdGV4dC1hbmNob3I9InN0YXJ0
IiB4PSIxODAiIHk9Ii0xNzguNSIgZm9udC1mYW1pbHk9IkhlbHZldGljYSxzYW5zLVNlcmlmIiBm
b250LXNpemU9IjIwLjAwIiBmaWxsPSJ3aGl0ZSI+Vk08L3RleHQ+Cjxwb2x5Z29uIGZpbGw9Indo
aXRlIiBzdHJva2U9Im5vbmUiIHBvaW50cz0iMTQ0LC0xMzcuNSAxNDQsLTE2Mi41IDE5MywtMTYy
LjUgMTkzLC0xMzcuNSAxNDQsLTEzNy41Ii8+Cjx0ZXh0IHRleHQtYW5jaG9yPSJzdGFydCIgeD0i
MTU2IiB5PSItMTQ2LjMiIGZvbnQtZmFtaWx5PSJIZWx2ZXRpY2Esc2Fucy1TZXJpZiIgZm9udC1z
aXplPSIxNC4wMCIgZmlsbD0iIzAwNjE3ZiI+QXBwPC90ZXh0Pgo8cG9seWdvbiBmaWxsPSJ3aGl0
ZSIgc3Ryb2tlPSJub25lIiBwb2ludHM9IjE5OCwtMTM3LjUgMTk4LC0xNjIuNSAyNDcsLTE2Mi41
IDI0NywtMTM3LjUgMTk4LC0xMzcuNSIvPgo8dGV4dCB0ZXh0LWFuY2hvcj0ic3RhcnQiIHg9IjIx
MCIgeT0iLTE0Ni4zIiBmb250LWZhbWlseT0iSGVsdmV0aWNhLHNhbnMtU2VyaWYiIGZvbnQtc2l6
ZT0iMTQuMDAiIGZpbGw9IiMwMDYxN2YiPkFwcDwvdGV4dD4KPHBvbHlnb24gZmlsbD0iIzAwNjE3
ZiIgc3Ryb2tlPSJub25lIiBwb2ludHM9IjE0NCwtMTAwLjUgMTQ0LC0xMzIuNSAyNDcsLTEzMi41
IDI0NywtMTAwLjUgMTQ0LC0xMDAuNSIvPgo8dGV4dCB0ZXh0LWFuY2hvcj0ic3RhcnQiIHg9IjE1
MS41IiB5PSItMTExLjUiIGZvbnQtZmFtaWx5PSJIZWx2ZXRpY2Esc2Fucy1TZXJpZiIgZm9udC1z
aXplPSIyMC4wMCIgZmlsbD0id2hpdGUiPkd1ZXN0IE9TPC90ZXh0Pgo8cG9seWdvbiBmaWxsPSIj
ZmY5MTAwIiBzdHJva2U9Im5vbmUiIHBvaW50cz0iMTgsLTY1LjUgMTgsLTkxLjUgMjU0LC05MS41
IDI1NCwtNjUuNSAxOCwtNjUuNSIvPgo8dGV4dCB0ZXh0LWFuY2hvcj0ic3RhcnQiIHg9IjEwOC41
IiB5PSItNzMuNSIgZm9udC1mYW1pbHk9IkhlbHZldGljYSxzYW5zLVNlcmlmIiBmb250LXNpemU9
IjIwLjAwIiBmaWxsPSJ3aGl0ZSI+UUVNVTwvdGV4dD4KPHRleHQgdGV4dC1hbmNob3I9InN0YXJ0
IiB4PSIzMjAuNSIgeT0iLTE5Mi44IiBmb250LWZhbWlseT0iSGVsdmV0aWNhLHNhbnMtU2VyaWYi
IGZvbnQtc2l6ZT0iMTQuMDAiPiA8L3RleHQ+Cjx0ZXh0IHRleHQtYW5jaG9yPSJzdGFydCIgeD0i
MzIwLjUiIHk9Ii0xNzAuOCIgZm9udC1mYW1pbHk9IkhlbHZldGljYSxzYW5zLVNlcmlmIiBmb250
LXNpemU9IjE0LjAwIj4gPC90ZXh0Pgo8dGV4dCB0ZXh0LWFuY2hvcj0ic3RhcnQiIHg9IjMyMC41
IiB5PSItMTQ5LjMiIGZvbnQtZmFtaWx5PSJIZWx2ZXRpY2Esc2Fucy1TZXJpZiIgZm9udC1zaXpl
PSIxNC4wMCI+IDwvdGV4dD4KPHBvbHlnb24gZmlsbD0iI2FiYmFiYSIgc3Ryb2tlPSJub25lIiBw
b2ludHM9IjI2NiwtNjcuNSAyNjYsLTEzOS41IDM3OSwtMTM5LjUgMzc5LC02Ny41IDI2NiwtNjcu
NSIvPgo8dGV4dCB0ZXh0LWFuY2hvcj0ic3RhcnQiIHg9IjI3OC41IiB5PSItMTEzLjUiIGZvbnQt
ZmFtaWx5PSJIZWx2ZXRpY2Esc2Fucy1TZXJpZiIgZm9udC1zaXplPSIyMC4wMCIgZmlsbD0id2hp
dGUiPkNvbnRhaW5lcjwvdGV4dD4KPHBvbHlnb24gZmlsbD0id2hpdGUiIHN0cm9rZT0ibm9uZSIg
cG9pbnRzPSIyNzEsLTcyLjUgMjcxLC05Ny41IDMyMCwtOTcuNSAzMjAsLTcyLjUgMjcxLC03Mi41
Ii8+Cjx0ZXh0IHRleHQtYW5jaG9yPSJzdGFydCIgeD0iMjgzIiB5PSItODEuMyIgZm9udC1mYW1p
bHk9IkhlbHZldGljYSxzYW5zLVNlcmlmIiBmb250LXNpemU9IjE0LjAwIiBmaWxsPSIjMDA2MTdm
Ij5BcHA8L3RleHQ+Cjxwb2x5Z29uIGZpbGw9IndoaXRlIiBzdHJva2U9Im5vbmUiIHBvaW50cz0i
MzI1LC03Mi41IDMyNSwtOTcuNSAzNzQsLTk3LjUgMzc0LC03Mi41IDMyNSwtNzIuNSIvPgo8dGV4
dCB0ZXh0LWFuY2hvcj0ic3RhcnQiIHg9IjMzNyIgeT0iLTgxLjMiIGZvbnQtZmFtaWx5PSJIZWx2
ZXRpY2Esc2Fucy1TZXJpZiIgZm9udC1zaXplPSIxNC4wMCIgZmlsbD0iIzAwNjE3ZiI+QXBwPC90
ZXh0Pgo8cG9seWdvbiBmaWxsPSIjYWJiYWJhIiBzdHJva2U9Im5vbmUiIHBvaW50cz0iMzg1LC02
Ny41IDM4NSwtMTM5LjUgNDk4LC0xMzkuNSA0OTgsLTY3LjUgMzg1LC02Ny41Ii8+Cjx0ZXh0IHRl
eHQtYW5jaG9yPSJzdGFydCIgeD0iMzk3LjUiIHk9Ii0xMTMuNSIgZm9udC1mYW1pbHk9IkhlbHZl
dGljYSxzYW5zLVNlcmlmIiBmb250LXNpemU9IjIwLjAwIiBmaWxsPSJ3aGl0ZSI+Q29udGFpbmVy
PC90ZXh0Pgo8cG9seWdvbiBmaWxsPSJ3aGl0ZSIgc3Ryb2tlPSJub25lIiBwb2ludHM9IjM5MCwt
NzIuNSAzOTAsLTk3LjUgNDM5LC05Ny41IDQzOSwtNzIuNSAzOTAsLTcyLjUiLz4KPHRleHQgdGV4
dC1hbmNob3I9InN0YXJ0IiB4PSI0MDIiIHk9Ii04MS4zIiBmb250LWZhbWlseT0iSGVsdmV0aWNh
LHNhbnMtU2VyaWYiIGZvbnQtc2l6ZT0iMTQuMDAiIGZpbGw9IiMwMDYxN2YiPkFwcDwvdGV4dD4K
PHBvbHlnb24gZmlsbD0id2hpdGUiIHN0cm9rZT0ibm9uZSIgcG9pbnRzPSI0NDQsLTcyLjUgNDQ0
LC05Ny41IDQ5MywtOTcuNSA0OTMsLTcyLjUgNDQ0LC03Mi41Ii8+Cjx0ZXh0IHRleHQtYW5jaG9y
PSJzdGFydCIgeD0iNDU2IiB5PSItODEuMyIgZm9udC1mYW1pbHk9IkhlbHZldGljYSxzYW5zLVNl
cmlmIiBmb250LXNpemU9IjE0LjAwIiBmaWxsPSIjMDA2MTdmIj5BcHA8L3RleHQ+Cjxwb2x5Z29u
IGZpbGw9IiMwMDYxN2YiIHN0cm9rZT0ibm9uZSIgcG9pbnRzPSIxMiwtNy41IDEyLC01My41IDUw
NiwtNTMuNSA1MDYsLTcuNSAxMiwtNy41Ii8+Cjxwb2x5Z29uIGZpbGw9IiMwMDYxN2YiIHN0cm9r
ZT0ibm9uZSIgcG9pbnRzPSIxNCwtOS41IDE0LC01MS41IDUwNCwtNTEuNSA1MDQsLTkuNSAxNCwt
OS41Ii8+Cjxwb2x5Z29uIGZpbGw9IiNmZjkxMDAiIHN0cm9rZT0ibm9uZSIgcG9pbnRzPSIxOSwt
MTggMTksLTQzIDY0LC00MyA2NCwtMTggMTksLTE4Ii8+Cjx0ZXh0IHRleHQtYW5jaG9yPSJzdGFy
dCIgeD0iMjQiIHk9Ii0yNi44IiBmb250LWZhbWlseT0iSGVsdmV0aWNhLHNhbnMtU2VyaWYiIGZv
bnQtc2l6ZT0iMTQuMDAiIGZpbGw9IndoaXRlIj5LVk08L3RleHQ+Cjx0ZXh0IHRleHQtYW5jaG9y
PSJzdGFydCIgeD0iMTUwLjUiIHk9Ii0yNS41IiBmb250LWZhbWlseT0iSGVsdmV0aWNhLHNhbnMt
U2VyaWYiIGZvbnQtc2l6ZT0iMjAuMDAiIGZpbGw9IndoaXRlIj5MaW51eCBLZXJuZWw8L3RleHQ+
Cjxwb2x5Z29uIGZpbGw9IiNmZjkxMDAiIHN0cm9rZT0ibm9uZSIgcG9pbnRzPSIzNDksLTE4IDM0
OSwtNDMgNDI0LC00MyA0MjQsLTE4IDM0OSwtMTgiLz4KPHRleHQgdGV4dC1hbmNob3I9InN0YXJ0
IiB4PSIzNTQiIHk9Ii0yNi44IiBmb250LWZhbWlseT0iSGVsdmV0aWNhLHNhbnMtU2VyaWYiIGZv
bnQtc2l6ZT0iMTQuMDAiIGZpbGw9IndoaXRlIj5BcHBBcm1vcjwvdGV4dD4KPHBvbHlnb24gZmls
bD0iI2ZmOTEwMCIgc3Ryb2tlPSJub25lIiBwb2ludHM9IjQyOSwtMTggNDI5LC00MyA0OTksLTQz
IDQ5OSwtMTggNDI5LC0xOCIvPgo8dGV4dCB0ZXh0LWFuY2hvcj0ic3RhcnQiIHg9IjQzNCIgeT0i
LTI2LjgiIGZvbnQtZmFtaWx5PSJIZWx2ZXRpY2Esc2Fucy1TZXJpZiIgZm9udC1zaXplPSIxNC4w
MCIgZmlsbD0id2hpdGUiPmNncm91cHM8L3RleHQ+CjwvZz4KPC9nPgo8L3N2Zz4K">
</div>
</div>
<div class="sect2">
<h3 id="intro_central_management">
<span>1.1. Central Management</span>
 <a class="headerlink" href="#intro_central_management" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>While many people start with a single node, Proxmox VE can scale out to a
large set of clustered nodes. The cluster stack is fully integrated
and ships with the default installation.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
Unique Multi-Master Design
</dt>
<dd>
<p>
The integrated web-based management interface gives you a clean
overview of all your KVM guests and Linux containers and even of your
whole cluster. You can easily manage your VMs and containers, storage
or cluster from the GUI. There is no need to install a separate,
complex, and pricey management server.
</p>
</dd>
<dt class="hdlist1">
Proxmox Cluster File System (pmxcfs)
</dt>
<dd>
<p>
Proxmox VE uses the unique Proxmox Cluster file system (pmxcfs), a
database-driven file system for storing configuration files. This
enables you to store the configuration of thousands of virtual
machines. By using corosync, these files are replicated in real time
on all cluster nodes. The file system stores all data inside a
persistent database on disk, nonetheless, a copy of the data resides
in RAM which provides a maximum storage size of 30MB - more than
enough for thousands of VMs.
</p>
<div class="paragraph">
<p>Proxmox VE is the only virtualization platform using this unique
cluster file system.</p></div>
</dd>
<dt class="hdlist1">
Web-based Management Interface
</dt>
<dd>
<p>
Proxmox VE is simple to use. Management tasks can be done via the
included web based management interface - there is no need to install a
separate management tool or any additional management node with huge
databases. The multi-master tool allows you to manage your whole
cluster from any node of your cluster. The central web-based
management - based on the JavaScript Framework (ExtJS) - empowers
you to control all functionalities from the GUI and overview history
and syslogs of each single node. This includes running backup or
restore jobs, live-migration or HA triggered activities.
</p>
</dd>
<dt class="hdlist1">
Command Line
</dt>
<dd>
<p>
For advanced users who are used to the comfort of the Unix shell or
Windows Powershell, Proxmox VE provides a command-line interface to
manage all the components of your virtual environment. This command-line
interface has intelligent tab completion and full documentation
in the form of UNIX man pages.
</p>
</dd>
<dt class="hdlist1">
REST API
</dt>
<dd>
<p>
Proxmox VE uses a RESTful API. We choose JSON as primary data format,
and the whole API is formally defined using JSON Schema. This enables
fast and easy integration for third party management tools like custom
hosting environments.
</p>
</dd>
<dt class="hdlist1">
Role-based Administration
</dt>
<dd>
<p>
You can define granular access for all objects (like VMs, storages,
nodes, etc.) by using the role based user- and permission
management. This allows you to define privileges and helps you to
control access to objects. This concept is also known as access
control lists: Each permission specifies a subject (a user or group)
and a role (set of privileges) on a specific path.
</p>
</dd>
<dt class="hdlist1">
Authentication Realms
</dt>
<dd>
<p>
Proxmox VE supports multiple authentication sources like Microsoft
Active Directory, LDAP, Linux PAM standard authentication or the
built-in Proxmox VE authentication server.
</p>
</dd>
</dl></div>
</div>
<div class="sect2">
<h3 id="_flexible_storage">
<span>1.2. Flexible Storage</span>
 <a class="headerlink" href="#_flexible_storage" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>The Proxmox VE storage model is very flexible. Virtual machine images
can either be stored on one or several local storages or on shared
storage like NFS and on SAN. There are no limits, you may configure as
many storage definitions as you like. You can use all storage
technologies available for Debian Linux.</p></div>
<div class="paragraph">
<p>One major benefit of storing VMs on shared storage is the ability to
live-migrate running machines without any downtime, as all nodes in
the cluster have direct access to VM disk images.</p></div>
<div class="paragraph">
<p>We currently support the following Network storage types:</p></div>
<div class="ulist"><ul>
<li>
<p>
LVM Group (network backing with iSCSI targets)
</p>
</li>
<li>
<p>
iSCSI target
</p>
</li>
<li>
<p>
NFS Share
</p>
</li>
<li>
<p>
CIFS Share
</p>
</li>
<li>
<p>
Ceph RBD
</p>
</li>
<li>
<p>
Directly use iSCSI LUNs
</p>
</li>
<li>
<p>
GlusterFS
</p>
</li>
</ul></div>
<div class="paragraph">
<p>Local storage types supported are:</p></div>
<div class="ulist"><ul>
<li>
<p>
LVM Group (local backing devices like block devices, FC devices, DRBD, etc.)
</p>
</li>
<li>
<p>
Directory (storage on existing filesystem)
</p>
</li>
<li>
<p>
ZFS
</p>
</li>
</ul></div>
</div>
<div class="sect2">
<h3 id="_integrated_backup_and_restore">
<span>1.3. Integrated Backup and Restore</span>
 <a class="headerlink" href="#_integrated_backup_and_restore" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>The integrated backup tool (<span class="monospaced">vzdump</span>) creates consistent snapshots of
running Containers and KVM guests. It basically creates an archive of
the VM or CT data which includes the VM/CT configuration files.</p></div>
<div class="paragraph">
<p>KVM live backup works for all storage types including VM images on
NFS, CIFS, iSCSI LUN, Ceph RBD. The new backup format is optimized for storing
VM backups fast and effective (sparse files, out of order data, minimized I/O).</p></div>
</div>
<div class="sect2">
<h3 id="_high_availability_cluster">
<span>1.4. High Availability Cluster</span>
 <a class="headerlink" href="#_high_availability_cluster" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>A multi-node Proxmox VE HA Cluster enables the definition of highly
available virtual servers. The Proxmox VE HA Cluster is based on
proven Linux HA technologies, providing stable and reliable HA
services.</p></div>
</div>
<div class="sect2">
<h3 id="_flexible_networking">
<span>1.5. Flexible Networking</span>
 <a class="headerlink" href="#_flexible_networking" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Proxmox VE uses a bridged networking model. All VMs can share one
bridge as if virtual network cables from each guest were all plugged
into the same switch. For connecting VMs to the outside world, bridges
are attached to physical network cards and assigned a TCP/IP
configuration.</p></div>
<div class="paragraph">
<p>For further flexibility, VLANs (IEEE 802.1q) and network
bonding/aggregation are possible. In this way it is possible to build
complex, flexible virtual networks for the Proxmox VE hosts,
leveraging the full power of the Linux network stack.</p></div>
</div>
<div class="sect2">
<h3 id="_integrated_firewall">
<span>1.6. Integrated Firewall</span>
 <a class="headerlink" href="#_integrated_firewall" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>The integrated firewall allows you to filter network packets on
any VM or Container interface. Common sets of firewall rules can
be grouped into “security groups”.</p></div>
</div>
<div class="sect2">
<h3 id="chapter_hyper_converged_infrastructure">
<span>1.7. Hyper-converged Infrastructure</span>
 <a class="headerlink" href="#chapter_hyper_converged_infrastructure" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Proxmox VE is a virtualization platform that tightly integrates compute, storage and
networking resources, manages highly available clusters, backup/restore as well
as disaster recovery. All components are software-defined and compatible with
one another.</p></div>
<div class="paragraph">
<p>Therefore it is possible to administrate them like a single system via the
centralized web management interface. These capabilities make Proxmox VE an ideal
choice to deploy and manage an open source
<a href="https://en.wikipedia.org/wiki/Hyper-converged_infrastructure">hyper-converged infrastructure</a>.</p></div>
<div class="sect3">
<h4 id="_benefits_of_a_hyper_converged_infrastructure_hci_with_proxmox_ve">1.7.1. Benefits of a Hyper-Converged Infrastructure (HCI) with Proxmox VE
 <a class="headerlink" href="#_benefits_of_a_hyper_converged_infrastructure_hci_with_proxmox_ve" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>A hyper-converged infrastructure (HCI) is especially useful for deployments in
which a high infrastructure demand meets a low administration budget, for
distributed setups such as remote and branch office environments or for virtual
private and public clouds.</p></div>
<div class="paragraph">
<p>HCI provides the following advantages:</p></div>
<div class="ulist"><ul>
<li>
<p>
Scalability: seamless expansion of compute, network and storage devices (i.e.
  scale up servers and storage quickly and independently from each other).
</p>
</li>
<li>
<p>
Low cost: Proxmox VE is open source and integrates all components you need such as
  compute, storage, networking, backup, and management center. It can replace
  an expensive compute/storage infrastructure.
</p>
</li>
<li>
<p>
Data protection and efficiency: services such as backup and disaster recovery
  are integrated.
</p>
</li>
<li>
<p>
Simplicity: easy configuration and centralized administration.
</p>
</li>
<li>
<p>
Open Source: No vendor lock-in.
</p>
</li>
</ul></div>
</div>
<div class="sect3">
<h4 id="_hyper_converged_infrastructure_storage">1.7.2. Hyper-Converged Infrastructure: Storage
 <a class="headerlink" href="#_hyper_converged_infrastructure_storage" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Proxmox VE has tightly integrated support for deploying a hyper-converged storage
infrastructure. You can, for example, deploy and manage the following two
storage technologies by using the web interface only:</p></div>
<div class="ulist"><ul>
<li>
<p>
<strong>Ceph</strong>: a both self-healing and self-managing shared, reliable and highly
  scalable storage system. Checkout
  <a href="#chapter_pveceph">how to manage Ceph services on Proxmox VE nodes</a>
</p>
</li>
<li>
<p>
<strong>ZFS</strong>: a combined file system and logical volume manager with extensive
  protection against data corruption, various RAID modes, fast and cheap
  snapshots - among other features. Find out
  <a href="#chapter_zfs">how to leverage the power of ZFS on Proxmox VE nodes</a>.
</p>
</li>
</ul></div>
<div class="paragraph">
<p>Besides above, Proxmox VE has support to integrate a wide range of
additional storage technologies. You can find out about them in the
<a href="#chapter_storage">Storage Manager chapter</a>.</p></div>
</div>
</div>
<div class="sect2">
<h3 id="_why_open_source">
<span>1.8. Why Open Source</span>
 <a class="headerlink" href="#_why_open_source" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Proxmox VE uses a Linux kernel and is based on the Debian GNU/Linux
Distribution. The source code of Proxmox VE is released under the
<a href="https://www.gnu.org/licenses/agpl-3.0.html">GNU Affero General Public
License, version 3</a>. This means that you are free to inspect the
source code at any time or contribute to the project yourself.</p></div>
<div class="paragraph">
<p>At Proxmox we are committed to use open source software whenever
possible. Using open source software guarantees full access to all
functionalities - as well as high security and reliability. We think
that everybody should have the right to access the source code of a
software to run it, build on it, or submit changes back to the
project. Everybody is encouraged to contribute while Proxmox ensures
the product always meets professional quality criteria.</p></div>
<div class="paragraph">
<p>Open source software also helps to keep your costs low and makes your
core infrastructure independent from a single vendor.</p></div>
</div>
<div class="sect2">
<h3 id="_your_benefits_with_proxmox_ve">
<span>1.9. Your benefits with Proxmox VE</span>
 <a class="headerlink" href="#_your_benefits_with_proxmox_ve" title="Permalink to this heading"></a>
</h3>
<div class="ulist"><ul>
<li>
<p>
Open source software
</p>
</li>
<li>
<p>
No vendor lock-in
</p>
</li>
<li>
<p>
Linux kernel
</p>
</li>
<li>
<p>
Fast installation and easy-to-use
</p>
</li>
<li>
<p>
Web-based management interface
</p>
</li>
<li>
<p>
REST API
</p>
</li>
<li>
<p>
Huge active community
</p>
</li>
<li>
<p>
Low administration costs and simple deployment
</p>
</li>
</ul></div>
</div>
<div class="sect2">
<h3 id="getting_help">
<span>1.10. Getting Help</span>
 <a class="headerlink" href="#getting_help" title="Permalink to this heading"></a>
</h3>
<div class="sect3">
<h4 id="_proxmox_ve_wiki">1.10.1. Proxmox VE Wiki
 <a class="headerlink" href="#_proxmox_ve_wiki" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The primary source of information is the <a href="https://pve.proxmox.com/wiki/">Proxmox VE Wiki</a>. It combines the reference
documentation with user contributed content.</p></div>
</div>
<div class="sect3">
<h4 id="_community_support_forum">1.10.2. Community Support Forum
 <a class="headerlink" href="#_community_support_forum" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Proxmox VE itself is fully open source, so we always encourage our users to discuss
and share their knowledge using the <a href="https://forum.proxmox.com/">Proxmox VE Community Forum</a>. The forum is moderated by the
Proxmox support team, and has a large user base from all around the world.
Needless to say, such a large forum is a great place to get information.</p></div>
</div>
<div class="sect3">
<h4 id="_mailing_lists">1.10.3. Mailing Lists
 <a class="headerlink" href="#_mailing_lists" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>This is a fast way to communicate with the Proxmox VE community via email.</p></div>
<div class="ulist"><ul>
<li>
<p>
Mailing list for users:
  <a href="http://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user">Proxmox VE User List</a>
</p>
</li>
</ul></div>
<div class="paragraph">
<p>Proxmox VE is fully open source and contributions are welcome! The primary
communication channel for developers is the:</p></div>
<div class="ulist"><ul>
<li>
<p>
Mailing list for developers:
  <a href="http://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel">Proxmox VE development
  discussion</a>
</p>
</li>
</ul></div>
</div>
<div class="sect3">
<h4 id="_commercial_support">1.10.4. Commercial Support
 <a class="headerlink" href="#_commercial_support" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Proxmox Server Solutions GmbH also offers enterprise support available as
<a href="https://proxmox.com/en/proxmox-virtual-environment/pricing">Proxmox VE Subscription Service Plans</a>.
All users with a subscription get access to the Proxmox VE
<a href="#sysadmin_enterprise_repo">Enterprise Repository</a>, and—with a Basic, Standard
or Premium subscription—also to the Proxmox Customer Portal. The customer
portal provides help and support with guaranteed response times from the Proxmox VE
developers.</p></div>
<div class="paragraph">
<p>For volume discounts, or more information in general, please contact
<a href="mailto:[email protected]">[email protected]</a>.</p></div>
</div>
<div class="sect3">
<h4 id="_bug_tracker">1.10.5. Bug Tracker
 <a class="headerlink" href="#_bug_tracker" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Proxmox runs a public bug tracker at <a href="https://bugzilla.proxmox.com">https://bugzilla.proxmox.com</a>. If an issue
appears, file your report there. An issue can be a bug as well as a request for
a new feature or enhancement. The bug tracker helps to keep track of the issue
and will send a notification once it has been solved.</p></div>
</div>
</div>
<div class="sect2">
<h3 id="intro_project_history">
<span>1.11. Project History</span>
 <a class="headerlink" href="#intro_project_history" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>The project started in 2007, followed by a first stable version in
2008. At the time we used OpenVZ for containers, and KVM for virtual
machines. The clustering features were limited, and the user interface
was simple (server generated web page).</p></div>
<div class="paragraph">
<p>But we quickly developed new features using the
<a href="https://corosync.github.io/corosync/">Corosync</a> cluster stack, and the
introduction of the new Proxmox cluster file system (pmxcfs) was a big
step forward, because it completely hides the cluster complexity from
the user. Managing a cluster of 16 nodes is as simple as managing a
single node.</p></div>
<div class="paragraph">
<p>We also introduced a new REST API, with a complete declarative
specification written in JSON-Schema. This enabled other people to
integrate Proxmox VE into their infrastructure, and made it easy to provide
additional services.</p></div>
<div class="paragraph">
<p>Also, the new REST API made it possible to replace the original user
interface with a modern HTML5 application using JavaScript. We also
replaced the old Java based VNC console code with
<a href="https://kanaka.github.io/noVNC/">noVNC</a>. So you only need a web browser
to manage your VMs.</p></div>
<div class="paragraph">
<p>The support for various storage types is another big task. Notably,
Proxmox VE was the first distribution to ship ZFS on Linux by default in
2014. Another milestone was the ability to run and manage
<a href="https://ceph.com/">Ceph</a> storage on the hypervisor nodes. Such setups
are extremely cost effective.</p></div>
<div class="paragraph">
<p>When we started we were among the first companies providing
commercial support for KVM. The KVM project itself continuously
evolved, and is now a widely used hypervisor. New features arrive
with each release. We developed the KVM live backup feature, which
makes it possible to create snapshot backups on any storage type.</p></div>
<div class="paragraph">
<p>The most notable change with version 4.0 was the move from OpenVZ to
<a href="https://linuxcontainers.org/">LXC</a>. Containers are now deeply
integrated, and they can use the same storage and network features
as virtual machines.</p></div>
</div>
<div class="sect2">
<h3 id="howto_improve_pve_docs">
<span>1.12. Improving the Proxmox VE Documentation</span>
 <a class="headerlink" href="#howto_improve_pve_docs" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Contributions and improvements to the Proxmox VE documentation are always welcome.
There are several ways to contribute.</p></div>
<div class="paragraph">
<p>If you find errors or other room for improvement in this documentation, please
file a bug at the <a href="https://bugzilla.proxmox.com/">Proxmox bug tracker</a> to propose
a correction.</p></div>
<div class="paragraph">
<p>If you want to propose new content, choose one of the following options:</p></div>
<div class="ulist"><ul>
<li>
<p>
The wiki: For specific setups, how-to guides, or tutorials the wiki   is the
right option to contribute.
</p>
</li>
<li>
<p>
The reference documentation: For general content that will be   helpful to all
  users please propose your contribution for the   reference documentation. This
  includes all information about how to install, configure, use, and
  troubleshoot Proxmox VE features. The reference documentation is written in the
  <a href="https://en.wikipedia.org/wiki/AsciiDoc">asciidoc format</a>. To edit the
  documentation you need to clone the git repository at
  <span class="monospaced">git://git.proxmox.com/git/pve-docs.git</span>; then follow the
  <a href="https://git.proxmox.com/?p=pve-docs.git;a=blob_plain;f=README.adoc;hb=HEAD">README.adoc</a>
  document.
</p>
</li>
</ul></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">If you are interested in working on the Proxmox VE codebase, the
<a href="https://pve.proxmox.com/wiki/Developer_Documentation">Developer Documentation</a> wiki article will
show you where to start.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect2">
<h3 id="translation">
<span>1.13. Translating Proxmox VE</span>
 <a class="headerlink" href="#translation" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>The Proxmox VE user interface is in English by default. However, thanks to the
contributions of the community, translations to other languages are also available.
We welcome any support in adding new languages, translating the latest features, and
improving incomplete or inconsistent translations.</p></div>
<div class="paragraph">
<p>We use <a href="https://www.gnu.org/software/gettext/">gettext</a> for the management of the
translation files. Tools like <a href="https://poedit.net/">Poedit</a> offer a nice user
interface to edit the translation files, but you can use whatever editor you’re
comfortable with. No programming knowledge is required for translating.</p></div>
<div class="sect3">
<h4 id="i18n_with_git">1.13.1. Translating with git
 <a class="headerlink" href="#i18n_with_git" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The language files are available as a
<a href="https://git.proxmox.com/?p=proxmox-i18n.git">git repository</a>. If you are familiar
with git, please contribute according to our
<a href="https://pve.proxmox.com/wiki/Developer_Documentation">Developer Documentation</a>.</p></div>
<div class="paragraph">
<p>You can create a new translation by doing the following (replace &lt;LANG&gt; with the
language ID):</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># git clone git://git.proxmox.com/git/proxmox-i18n.git
# cd proxmox-i18n
# make init-&lt;LANG&gt;.po</pre>
</div></div>
<div class="paragraph">
<p>Or you can edit an existing translation, using the editor of your choice:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># poedit &lt;LANG&gt;.po</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="i18n_without_git">1.13.2. Translating without git
 <a class="headerlink" href="#i18n_without_git" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Even if you are not familiar with git, you can help translate Proxmox VE.
To start, you can download the language files
<a href="https://git.proxmox.com/?p=proxmox-i18n.git;a=tree">here</a>. Find the
language you want to improve, then right click on the "raw" link of this language
file and select <em>Save Link As…</em>. Make your changes to the file, and then
send your final translation directly to office(at)proxmox.com, together with a
signed
<a href="https://pve.proxmox.com/wiki/Developer_Documentation#Software_License_and_Copyright">contributor license agreement</a>.</p></div>
</div>
<div class="sect3">
<h4 id="_testing_the_translation">1.13.3. Testing the Translation
 <a class="headerlink" href="#_testing_the_translation" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>In order for the translation to be used in Proxmox VE, you must first translate
the <span class="monospaced">.po</span> file into a <span class="monospaced">.js</span> file. You can do this by invoking the following script,
which is located in the same repository:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># ./po2js.pl -t pve xx.po &gt;pve-lang-xx.js</pre>
</div></div>
<div class="paragraph">
<p>The resulting file <span class="monospaced">pve-lang-xx.js</span> can then be copied to the directory
<span class="monospaced">/usr/share/pve-i18n</span>, on your proxmox server, in order to test it out.</p></div>
<div class="paragraph">
<p>Alternatively, you can build a deb package by running the following command from
the root of the repository:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># make deb</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Important" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAALa0lEQVRogdWZa2wc1RXHfzM7O/te
P9e1vXHSmEdjx3YeDkkaF6REKRRCEDSEFNkRjdSWSsgC2iqoRWqLQKiqIBg1NOQDiMeHtkQIQkRR
S9S4aWwgCQoUgl232KkT28J21l6vd3d2dx79sDuTXdtre03Uqlc62tl53Pv/n3PuOffcKxiGwf9z
E//XAL5sk65WR0a6AaDrOjMtKwgCoigiCAKCIAhXa9wvTcAEHhoc5KMjR7jw3nucf+MN67k/GCS4
YQOr77yTpl27kGUZm81mXC0ywlLnQDbwEwcO0P2b3yz4jV5RwZq2Nm5/5BH8fj+SJGGz2b4UiSUR
MAzD0DSNEwcO8MdHHln8d0AcsAUCfOe551h7yy04HA7sdvuSrVEwAV3XjdDgIC/v2cOl06dznqWA
JCAAMvn9MwlEgF1PP8037rsPt9uN3W5fkjUKIqDrujF+4QLP3XQTU0NDOc8UIAa4y8sJtrQQqK/H
7XQSfv99Pn/nHQRRxNB16/0EEAUatm9n76uv4vP5cDgcBZNYNIF84I0MEK20lIZvf5sNe/dSWlqK
1+vFxBEZHubT3/+evqNHmR4dJRWP55BevW0bra+8gt/vx+l0IknSokksikA+8ElgClixYwc3799P
ZWUlTqczbz+x0VFOPvEEQ6dPE/niCwxdt0jUbd1K68svU1xcXBCJBQnMBz5eWkrL/v2s++Y3KSsr
QxQXlxc7f/5zBjo7CQ8NoadSKKQn943t7dyyf79FYjHuNC8BwzAMVVV56Z57+OzNN3PAJ8vK+FZH
B2s2b55X6/nahRMnOP7TnxLOKEXJyO2/+hVf37uXoqIinE6nmfzyksirMjNUvvGjH+WAVwG1vJxb
Dhxg1bp1uFwuMwQWJCu3bWPN3r24SkoAcGbA/OXXv2bws8+IRqOkUqlZGX1RBAzDMHRd51+nTvHe
wYNX7gPTQMO+faxav56SkpIlgTdlU3s7TXv2IGUs6AGioRDH2tuZmJggFouhqirGPCzyEUBVVX7X
1pZzPwI0fP/7bNmzh7KysrzADh8+zP3338++ffs4ePBg3vdEUWTDD3/IsuZmRLsdW4bEpd5eOp99
lnA4jKIoaJqWl8QsAqb2Txw4kDNp40BxXR0bWlspLy/PC+qZZ55hdHQUSZIYHByko6ODxx57bBZw
cy3kq6xk6y9/ib+qCgQBmbQ7nT50iIs9PUSjUZLJJIZhzEliLgKMDQzwp5/9zLqnkQ51W3/xCwKB
AE6nMy+B3t5exsfH6ezspLe3l3g8zuHDh2cBz5bKpibWtrXhLi0FwA3owNsPP8zExATxeBxVVecy
QC4Bc+L+taNjluvU33031dddN6/2BUGgpKSE/v5+JicncweaA3i2tDz8MIHrr0eU0gsQBzDW10ff
yZNEIhESicScrpRDQNd1xgYG+OC3v7XuJQBXIMDWn/yEioqKeUGMjY0hCAKKoszS1MjIyLzfAmx/
4gmKli0DQcBJOmicfOopJicnicfjJoG5LWBq/+Szz+a8oABNe/bgdrvndR1BEKiqqiIejxOLxWYR
CAaDC0alqjVrqGxowFNWhkDalUKff07f3/5mWSFTLFksLAK6rjN+4UKO9lXAUV7O2tbWeaNOthZl
WZ4FHlh0aN3y4IPIHk+6L8AGvNfRQTgctuZCthVEU/u6rnPutddyBk0A9bt3W4us+QY2fbyurg6v
15vTzw033LBoAsHmZoLNzdgcDgBcQKi/n391dTE9PU0ymcyxgkmAVCrF+4cO5WjfFgiwfoGwOTOy
lJWVUVtba1mkUAKCILClvR1veTkA9owVPnzllZy8YFnA1H5/dzfTw8PWgziwcutWfD4fsiwvCNyU
xsZG3G43lZWVVl933XVXQQSq161j+ebNlhXcwBdnzxIOh60lhmkFEUDTNP7++uvWgBrp6mrNHBk3
H/BsAtFoFEdm8GAwyMaNGwsiIAgCjbt346uosKwgAp8cPWolNj1THInmsqHn6FGLQApY1thIxcqV
+Hy+WX6+0ETesWMHLpcLWZZ54IEHCgYvCAKrbrsNyem0JrQEXMzkhKzlBaKu61z+97+ZHhnJmbwr
tm2zJu5CWp8pra2tNDU1sXPnTtrb25dEQBAEvrplC57MXJCB8XPniEQixONxa6Uq6brOQHd3jvvo
QP2ttxZUpMxsL7zwwpK+AyxLXrt9Oxe6uxFEEXvGZc4fO0bVD35AMplM1wuapjGYtbugAp5AgKLq
aquu/W/JTEs37dqFqihWzSADk8PD1jzQNC3tQtmrThUINDbm+H4h0tfXx6OPPsodd9xBW1sbPT09
BQPPluu2b7fWRxIQ6usjFotZBCRN06wZbbZAfT1FRUWWKQtpzz//PMPDw/T09DA1NcXo6CjHjx+f
11Xma95AwHJjEdB0HUVRrhCYuRGrAiXBoFUqFtq6u7sZHx9nJBMUurq65uxnsX0Hm5v56A9/ANIJ
LXrpEolEwgqlUqZQyPlo6OOPlzx5ly1bxnBWQmxpackBW6hSBEC02axrpqasNZGmaek8kN0kYPDj
j1FVdUlz4Mknn6Surs4Cb5aUhYZiUy59+GGOgm3V1WiadiUPCILAV9autV5wAaM9PXQeOWJprBBp
aGjgxIkTpFIpOjs7aWxsXBJwUwZOnUKZmgLS2zl4vUjSlV1XURRFqtavv3IDKAVe/O53efPxx69a
SCxUBs+c4dV77+Vyfz+xy5eBdG3iX7ECm81m9S2JokiwuZmiujrCPT1AuqgOAqcef5zzL77Imt27
uemhhyipqVnYZ5cw8bPbR0eOcOall7g8MEB4aIhEJAJk9qOAwKZNyLJsHpQgRKNRIxQK0XPmDH/+
3vdQQ6G8na/YtIk199zDypYWVmzceFWAT1y8yMWzZzn/9tv0vfsudqeTlKIQHRvL2QSOAqU7d/K1
W2/l2muvpba2Nl3iJhIJIxKJMDw8TO/Zs3zw4x+jTUzkHVCw2fCUleHw+QiuXYu/upqqhgb8VVUY
hkFNczPFweCs7z556y3r+vyxYwiiyD+PHwdBwNB1krEYhqYRC4UwdB2D9LJmmnRtXHz77VTfeCPL
ly+ntraWmpqa9MaaqqqGoiiEQiGGhoYYHBzk00OHiHR2LkqDNlnG7nJhd7nQswqN+ZogiuipFMlY
DDWRgBl5KEF6woqAUFFByc03U756NZWVlSxfvpyamhoCgQAejwdB13VDVVVisRihUIiRkRGGh4cZ
+sc/GO3qIt7VhZGJAle7mVpWuVKDCBng8jXX4N68Gc8111BUVER5eTnV1dVUV1cTCASuFFqGYaDr
upFKpYjH40xMTDA+Ps7Y2Bjj4+OEw2EmenqYPneO1OAgZCWpxTY9S9QMcLhyHGUDRI8HqbISZ309
zoYGnE4nHo8Hv99PaWkpgUCAiooK6/DE4XCkI1HW2a6hqiqKojA9Pc3k5KQl4XCY6elpotFoetuk
v59UKIR66RLToRBJTcMYHUWIRvMSMPO6rbY2/SsIOFatQpYk5NWrkSQJWZZxOBy43W68Xi9+v5+i
oiJKSkooLi7G5/PhdruRZdnads85HzD3hpLJJIqiEIvFiEajRCIRIpEI0WiUWCxGPB631iOKopBK
pazUbi4Oc7Y+snKCzWbDZrMhSRKSJGG3262w6HK5cLlceDwevF4vPp8Pr9eL2+3G5XLhcDjMkxuE
TNibdcBhnv9qmkYqlSKZTJJIJFAUhXg8jqIoKIpCIpGwSCSTSVRVtWSuk3qTiAnebrfngHc4HDid
TpxOJy6Xy7qWZdk6wZzrxCbvCY1JRNd1NE3LAWgSM69ngp+PgCiK1gF3NhFTzHvmO+Y3wCzw8xLI
JpL5tYCZpEyw2f/nWt1ag2UtM0x3Mq1iAp1j+bH0M7KFCGWDnfmbd8AMnuxfUzL/C0rp/wFnFd4n
EQn3XQAAAABJRU5ErkJggg==">
</td>
<td class="content">For either of these methods to work, you need to have the following
perl packages installed on your system. For Debian/Ubuntu:</td>
</tr></tbody></table>
</div>
<div class="literalblock">
<div class="content monospaced">
<pre># apt-get install perl liblocale-po-perl libjson-perl</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_sending_the_translation">1.13.4. Sending the Translation
 <a class="headerlink" href="#_sending_the_translation" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>You can send the finished translation (<span class="monospaced">.po</span> file) to the Proxmox team at the address
office(at)proxmox.com, along with a signed contributor license agreement.
Alternatively, if you have some developer experience, you can send it as a
patch to the Proxmox VE development mailing list. See
<a href="https://pve.proxmox.com/wiki/Developer_Documentation">Developer Documentation</a>.</p></div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="chapter_installation">
2. Installing Proxmox VE
 <a class="headerlink" href="#chapter_installation" title="Permalink to this heading"></a>
</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Proxmox VE is based on Debian. This is why the install disk images (ISO files)
provided by Proxmox include a complete Debian system as well as all necessary
Proxmox VE packages.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">See the <a href="#faq-support-table">support table in the FAQ</a> for the
relationship between Proxmox VE releases and Debian releases.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>The installer will guide you through the setup, allowing you to partition the
local disk(s), apply basic system configurations (for example, timezone,
language, network) and install all required packages. This process should not
take more than a few minutes. Installing with the provided ISO is the
recommended method for new and existing users.</p></div>
<div class="paragraph">
<p>Alternatively, Proxmox VE can be installed on top of an existing Debian system. This
option is only recommended for advanced users because detailed knowledge about
Proxmox VE is required.</p></div>
<div class="sect2">
<h3 id="_system_requirements">
<span>2.1. System Requirements</span>
 <a class="headerlink" href="#_system_requirements" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>We recommend using high quality server hardware, when running Proxmox VE in
production. To further decrease the impact of a failed host, you can run Proxmox VE in
a cluster with highly available (HA) virtual machines and containers.</p></div>
<div class="paragraph">
<p>Proxmox VE can use local storage (DAS), SAN, NAS, and distributed storage like Ceph
RBD. For details see <a href="#chapter_storage">chapter storage</a>.</p></div>
<div class="sect3">
<h4 id="install_minimal_requirements">2.1.1. Minimum Requirements, for Evaluation
 <a class="headerlink" href="#install_minimal_requirements" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>These minimum requirements are for evaluation purposes only and should not be
used in production.</p></div>
<div class="ulist"><ul>
<li>
<p>
CPU: 64bit (Intel EMT64 or AMD64)
</p>
</li>
<li>
<p>
Intel VT/AMD-V capable CPU/motherboard for KVM full virtualization support
</p>
</li>
<li>
<p>
RAM: 1 GB RAM, plus additional RAM needed for guests
</p>
</li>
<li>
<p>
Hard drive
</p>
</li>
<li>
<p>
One network card (NIC)
</p>
</li>
</ul></div>
</div>
<div class="sect3">
<h4 id="install_recommended_requirements">2.1.2. Recommended System Requirements
 <a class="headerlink" href="#install_recommended_requirements" title="Permalink to this heading"></a>
</h4>
<div class="ulist"><ul>
<li>
<p>
Intel EMT64 or AMD64 with Intel VT/AMD-V CPU flag.
</p>
</li>
<li>
<p>
Memory: Minimum 2 GB for the OS and Proxmox VE services, plus designated memory for
  guests. For Ceph and ZFS, additional memory is required; approximately 1GB of
  memory for every TB of used storage.
</p>
</li>
<li>
<p>
Fast and redundant storage, best results are achieved with SSDs.
</p>
</li>
<li>
<p>
OS storage: Use a hardware RAID with battery protected write cache (“BBU”)
  or non-RAID with ZFS (optional SSD for ZIL).
</p>
</li>
<li>
<p>
VM storage:
</p>
<div class="ulist"><ul>
<li>
<p>
For local storage, use either a hardware RAID with battery backed write cache
  (BBU) or non-RAID for ZFS and Ceph. Neither ZFS nor Ceph are compatible with a
  hardware RAID controller.
</p>
</li>
<li>
<p>
Shared and distributed storage is possible.
</p>
</li>
<li>
<p>
SSDs with Power-Loss-Protection (PLP) are recommended for good performance.
  Using consumer SSDs is discouraged.
</p>
</li>
</ul></div>
</li>
<li>
<p>
Redundant (Multi-)Gbit NICs, with additional NICs depending on the preferred
  storage technology and cluster setup.
</p>
</li>
<li>
<p>
For PCI(e) passthrough the CPU needs to support the VT-d/AMD-d flag.
</p>
</li>
</ul></div>
</div>
<div class="sect3">
<h4 id="_simple_performance_overview">2.1.3. Simple Performance Overview
 <a class="headerlink" href="#_simple_performance_overview" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>To get an overview of the CPU and hard disk performance on an installed Proxmox VE
system, run the included <span class="monospaced">pveperf</span> tool.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">This is just a very quick and general benchmark. More detailed tests are
recommended, especially regarding the I/O performance of your system.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="_supported_web_browsers_for_accessing_the_web_interface">2.1.4. Supported Web Browsers for Accessing the Web Interface
 <a class="headerlink" href="#_supported_web_browsers_for_accessing_the_web_interface" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>To access the web-based user interface, we recommend using one of the following
browsers:</p></div>
<div class="ulist"><ul>
<li>
<p>
Firefox, a release from the current year, or the latest Extended Support Release
</p>
</li>
<li>
<p>
Chrome, a release from the current year
</p>
</li>
<li>
<p>
Microsoft’s currently supported version of Edge
</p>
</li>
<li>
<p>
Safari, a release from the current year
</p>
</li>
</ul></div>
<div class="paragraph">
<p>When accessed from a mobile device, Proxmox VE will show a lightweight, touch-based
interface.</p></div>
</div>
</div>
<div class="sect2">
<h3 id="installation_prepare_media">
<span>2.2. Prepare Installation Media</span>
 <a class="headerlink" href="#installation_prepare_media" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Download the installer ISO image from: <a href="https://www.proxmox.com/en/downloads/proxmox-virtual-environment/iso">https://www.proxmox.com/en/downloads/proxmox-virtual-environment/iso</a></p></div>
<div class="paragraph">
<p>The Proxmox VE installation media is a hybrid ISO image. It works in two ways:</p></div>
<div class="ulist"><ul>
<li>
<p>
An ISO image file ready to burn to a CD or DVD.
</p>
</li>
<li>
<p>
A raw sector (IMG) image file ready to copy to a USB flash drive (USB stick).
</p>
</li>
</ul></div>
<div class="paragraph">
<p>Using a USB flash drive to install Proxmox VE is the recommended way because it is
the faster option.</p></div>
<div class="sect3">
<h4 id="_prepare_a_usb_flash_drive_as_installation_medium">2.2.1. Prepare a USB Flash Drive as Installation Medium
 <a class="headerlink" href="#_prepare_a_usb_flash_drive_as_installation_medium" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The flash drive needs to have at least 1 GB of storage available.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Do not use UNetbootin. It does not work with the Proxmox VE installation image.</td>
</tr></tbody></table>
</div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Important" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAALa0lEQVRogdWZa2wc1RXHfzM7O/te
P9e1vXHSmEdjx3YeDkkaF6REKRRCEDSEFNkRjdSWSsgC2iqoRWqLQKiqIBg1NOQDiMeHtkQIQkRR
S9S4aWwgCQoUgl232KkT28J21l6vd3d2dx79sDuTXdtre03Uqlc62tl53Pv/n3PuOffcKxiGwf9z
E//XAL5sk65WR0a6AaDrOjMtKwgCoigiCAKCIAhXa9wvTcAEHhoc5KMjR7jw3nucf+MN67k/GCS4
YQOr77yTpl27kGUZm81mXC0ywlLnQDbwEwcO0P2b3yz4jV5RwZq2Nm5/5BH8fj+SJGGz2b4UiSUR
MAzD0DSNEwcO8MdHHln8d0AcsAUCfOe551h7yy04HA7sdvuSrVEwAV3XjdDgIC/v2cOl06dznqWA
JCAAMvn9MwlEgF1PP8037rsPt9uN3W5fkjUKIqDrujF+4QLP3XQTU0NDOc8UIAa4y8sJtrQQqK/H
7XQSfv99Pn/nHQRRxNB16/0EEAUatm9n76uv4vP5cDgcBZNYNIF84I0MEK20lIZvf5sNe/dSWlqK
1+vFxBEZHubT3/+evqNHmR4dJRWP55BevW0bra+8gt/vx+l0IknSokksikA+8ElgClixYwc3799P
ZWUlTqczbz+x0VFOPvEEQ6dPE/niCwxdt0jUbd1K68svU1xcXBCJBQnMBz5eWkrL/v2s++Y3KSsr
QxQXlxc7f/5zBjo7CQ8NoadSKKQn943t7dyyf79FYjHuNC8BwzAMVVV56Z57+OzNN3PAJ8vK+FZH
B2s2b55X6/nahRMnOP7TnxLOKEXJyO2/+hVf37uXoqIinE6nmfzyksirMjNUvvGjH+WAVwG1vJxb
Dhxg1bp1uFwuMwQWJCu3bWPN3r24SkoAcGbA/OXXv2bws8+IRqOkUqlZGX1RBAzDMHRd51+nTvHe
wYNX7gPTQMO+faxav56SkpIlgTdlU3s7TXv2IGUs6AGioRDH2tuZmJggFouhqirGPCzyEUBVVX7X
1pZzPwI0fP/7bNmzh7KysrzADh8+zP3338++ffs4ePBg3vdEUWTDD3/IsuZmRLsdW4bEpd5eOp99
lnA4jKIoaJqWl8QsAqb2Txw4kDNp40BxXR0bWlspLy/PC+qZZ55hdHQUSZIYHByko6ODxx57bBZw
cy3kq6xk6y9/ib+qCgQBmbQ7nT50iIs9PUSjUZLJJIZhzEliLgKMDQzwp5/9zLqnkQ51W3/xCwKB
AE6nMy+B3t5exsfH6ezspLe3l3g8zuHDh2cBz5bKpibWtrXhLi0FwA3owNsPP8zExATxeBxVVecy
QC4Bc+L+taNjluvU33031dddN6/2BUGgpKSE/v5+JicncweaA3i2tDz8MIHrr0eU0gsQBzDW10ff
yZNEIhESicScrpRDQNd1xgYG+OC3v7XuJQBXIMDWn/yEioqKeUGMjY0hCAKKoszS1MjIyLzfAmx/
4gmKli0DQcBJOmicfOopJicnicfjJoG5LWBq/+Szz+a8oABNe/bgdrvndR1BEKiqqiIejxOLxWYR
CAaDC0alqjVrqGxowFNWhkDalUKff07f3/5mWSFTLFksLAK6rjN+4UKO9lXAUV7O2tbWeaNOthZl
WZ4FHlh0aN3y4IPIHk+6L8AGvNfRQTgctuZCthVEU/u6rnPutddyBk0A9bt3W4us+QY2fbyurg6v
15vTzw033LBoAsHmZoLNzdgcDgBcQKi/n391dTE9PU0ymcyxgkmAVCrF+4cO5WjfFgiwfoGwOTOy
lJWVUVtba1mkUAKCILClvR1veTkA9owVPnzllZy8YFnA1H5/dzfTw8PWgziwcutWfD4fsiwvCNyU
xsZG3G43lZWVVl933XVXQQSq161j+ebNlhXcwBdnzxIOh60lhmkFEUDTNP7++uvWgBrp6mrNHBk3
H/BsAtFoFEdm8GAwyMaNGwsiIAgCjbt346uosKwgAp8cPWolNj1THInmsqHn6FGLQApY1thIxcqV
+Hy+WX6+0ETesWMHLpcLWZZ54IEHCgYvCAKrbrsNyem0JrQEXMzkhKzlBaKu61z+97+ZHhnJmbwr
tm2zJu5CWp8pra2tNDU1sXPnTtrb25dEQBAEvrplC57MXJCB8XPniEQixONxa6Uq6brOQHd3jvvo
QP2ttxZUpMxsL7zwwpK+AyxLXrt9Oxe6uxFEEXvGZc4fO0bVD35AMplM1wuapjGYtbugAp5AgKLq
aquu/W/JTEs37dqFqihWzSADk8PD1jzQNC3tQtmrThUINDbm+H4h0tfXx6OPPsodd9xBW1sbPT09
BQPPluu2b7fWRxIQ6usjFotZBCRN06wZbbZAfT1FRUWWKQtpzz//PMPDw/T09DA1NcXo6CjHjx+f
11Xma95AwHJjEdB0HUVRrhCYuRGrAiXBoFUqFtq6u7sZHx9nJBMUurq65uxnsX0Hm5v56A9/ANIJ
LXrpEolEwgqlUqZQyPlo6OOPlzx5ly1bxnBWQmxpackBW6hSBEC02axrpqasNZGmaek8kN0kYPDj
j1FVdUlz4Mknn6Surs4Cb5aUhYZiUy59+GGOgm3V1WiadiUPCILAV9autV5wAaM9PXQeOWJprBBp
aGjgxIkTpFIpOjs7aWxsXBJwUwZOnUKZmgLS2zl4vUjSlV1XURRFqtavv3IDKAVe/O53efPxx69a
SCxUBs+c4dV77+Vyfz+xy5eBdG3iX7ECm81m9S2JokiwuZmiujrCPT1AuqgOAqcef5zzL77Imt27
uemhhyipqVnYZ5cw8bPbR0eOcOall7g8MEB4aIhEJAJk9qOAwKZNyLJsHpQgRKNRIxQK0XPmDH/+
3vdQQ6G8na/YtIk199zDypYWVmzceFWAT1y8yMWzZzn/9tv0vfsudqeTlKIQHRvL2QSOAqU7d/K1
W2/l2muvpba2Nl3iJhIJIxKJMDw8TO/Zs3zw4x+jTUzkHVCw2fCUleHw+QiuXYu/upqqhgb8VVUY
hkFNczPFweCs7z556y3r+vyxYwiiyD+PHwdBwNB1krEYhqYRC4UwdB2D9LJmmnRtXHz77VTfeCPL
ly+ntraWmpqa9MaaqqqGoiiEQiGGhoYYHBzk00OHiHR2LkqDNlnG7nJhd7nQswqN+ZogiuipFMlY
DDWRgBl5KEF6woqAUFFByc03U756NZWVlSxfvpyamhoCgQAejwdB13VDVVVisRihUIiRkRGGh4cZ
+sc/GO3qIt7VhZGJAle7mVpWuVKDCBng8jXX4N68Gc8111BUVER5eTnV1dVUV1cTCASuFFqGYaDr
upFKpYjH40xMTDA+Ps7Y2Bjj4+OEw2EmenqYPneO1OAgZCWpxTY9S9QMcLhyHGUDRI8HqbISZ309
zoYGnE4nHo8Hv99PaWkpgUCAiooK6/DE4XCkI1HW2a6hqiqKojA9Pc3k5KQl4XCY6elpotFoetuk
v59UKIR66RLToRBJTcMYHUWIRvMSMPO6rbY2/SsIOFatQpYk5NWrkSQJWZZxOBy43W68Xi9+v5+i
oiJKSkooLi7G5/PhdruRZdnads85HzD3hpLJJIqiEIvFiEajRCIRIpEI0WiUWCxGPB631iOKopBK
pazUbi4Oc7Y+snKCzWbDZrMhSRKSJGG3262w6HK5cLlceDwevF4vPp8Pr9eL2+3G5XLhcDjMkxuE
TNibdcBhnv9qmkYqlSKZTJJIJFAUhXg8jqIoKIpCIpGwSCSTSVRVtWSuk3qTiAnebrfngHc4HDid
TpxOJy6Xy7qWZdk6wZzrxCbvCY1JRNd1NE3LAWgSM69ngp+PgCiK1gF3NhFTzHvmO+Y3wCzw8xLI
JpL5tYCZpEyw2f/nWt1ag2UtM0x3Mq1iAp1j+bH0M7KFCGWDnfmbd8AMnuxfUzL/C0rp/wFnFd4n
EQn3XQAAAABJRU5ErkJggg==">
</td>
<td class="content">Make sure that the USB flash drive is not mounted and does not
contain any important data.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="_instructions_for_gnu_linux">2.2.2. Instructions for GNU/Linux
 <a class="headerlink" href="#_instructions_for_gnu_linux" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>On Unix-like operating system use the <span class="monospaced">dd</span> command to copy the ISO image to the
USB flash drive. First find the correct device name of the USB flash drive (see
below). Then run the <span class="monospaced">dd</span> command.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># dd bs=1M conv=fdatasync if=./proxmox-ve_*.iso of=/dev/XYZ</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Be sure to replace /dev/XYZ with the correct device name and adapt the
input filename (<em>if</em>) path.</td>
</tr></tbody></table>
</div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Caution" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKdUlEQVRoge1Ze1AV1x3+zt279wEi
DWCYGzRVktqa1MEmmtbWR22ncXxUrTDWV/5IqG2wUAUfmUwSkk6V+EAJCEQC6figwZBqZtRxqukf
tebRZBpFG1S0hiRErwiaKK977+6eX//YPXt37+UiIJlMZnpmdnb33PP4vt97z2VEhG9yc3zdAO60
/Z/A192+8QScX8Wifr+fWltbzffU1FT4fD72Vew15ASampqovr4eBw8eNPvGjRuHzMxMmj9//tCT
IKIhu958801yuVwEoNdr48aNNJT7EdHQEdi/fz/JshwTvLiKioqGlMSQLFJfX0+MMRvQsWNSqXLz
H2jez78fRWLTpk1DRuKOF6irqyOn02kDOP8XGdTz+VFSP91Hatu79NRvZ0SR2LJly5CQuKPJtbW1
JEmSDdivHp1AwctHSblYSsqFElLOF5PavIvWZU+LIrF169Y7JjHoibt3744CnzXrIQpeOUbKhVJS
mkpIPV9MyrnNpJwtIuVSNa15/MdRJLZt23ZHJBjRwIu5N954gxYvXgzOudm3aO5E1L5SBNZ1Hoxr
INLAyH6HJxVP/WkXtu9+37ZeSUkJVq9ePagQO+BMXFNTEwV+ybxHsPeVIjg6zwOkgSWNhuOuUUBC
KhxJY0AggDSgpwWbn1mO1csftq1ZUFCA0tLSQZXFA0pkVVVVlJOTA6vWsmY9hF07N0AS4OU48LgR
WLO+FPFxboxMS8G5hlPYmPMI4jwM1P0Jtj67DJyrKHvtNADdjPPz8+FwOCgvL29gmuivrVVUVESF
yqxZD1PoyjFSL+7QHbapmJTml+n9EzVRtl676ZekNKwl5VQ+KSfzKHR2E+UutodYxhiVl5cPyCf6
ZUJlZWWUm5trk/wTi6bgtVdf1M2Gq2DEwUgDNBWhUDBqjfSRwwGoIOEP3c3Y/uxirFw0zibMvLw8
VFZW9tucbkugoqKCVq1aZQO/YslUVJU+D9bVBIDroLgK4hygEDweybYGY8AD6d8yftfASL+j+2O8
9NwiPJk51kYiNzcXO3fu7BeJPgkUFxdTbm6ure/JZdNRub0QrOMcGFcBrgLgAHEAGqD2ID7OZZsz
Ji0RcR6HAV4DkaqPJxXouoSywkz8buF9NhIrV65EVVXVbUnEJHDixAlat26drS9n+U+xo/hZMMNh
yZA+uAbAIKOGMCJlmG3eA/clAYaJEamGBlRdC6QBXZewo3AhVswfbSORk5ODDz/8sE8SvRLw+/1U
XV0dBb5MgOe6LUNIEqpOwgCUlOCESw4HuAfG3KX7B2kAGaTJ0Jro77yI8ucX4DfzRtlIHD58GH6/
PyaJXgk0NjZi79695vuUiffrkr91Vt8cHAwc4FZJGiZEKqDcwsi0ZHP+d749POy8MS7iGtBxARWF
8/CDsQnm3BdeeAHWj6N+EYhspSVbgI5zYYkZGmAwzIEbwA0tUM9NfO+7aeb89HuGRQBWLcSNZxj3
zvOYMzk5NpjBEHi99mXD5lXdjqGDJh42h7CEOXhnK3448X5z/shUb1jSpIEM8yFuzOUcjAsNcnR0
KXdG4MEHH8TEiRPN9z0H3sfRE41m+GNcs9i0LjkGTTcrUoGem5g7/V5z/ohElw4O+jxGYQ3qJqmB
jN/+cbINtcfCJvPYY4+hr3qtVwI+n49lZmaa71fbbuH3hX/FycbPQKSCoIIMQEQcehjVo5Gw8/F3
3cSvH03H2HuHY1icQzcvYWKGIMiITMwQwjtnruPpqk9w/ZZq7p2RkYEJEyaAYrCIWY36/X7KyMhA
W1ub2XfP3Qmo3ZKFaQ+P0gmYIdSoOkU4lWSwxGRAdqHL/xm86k27w0blAw3/PncD6yo/xrsfdZj7
paSkoLGxESNGjAgDZsxWK8X0AZ/Px44fP460tLAzXrnWgUX5r+PYuxf1mG/Gf80sofWy2YuWbg/+
uONtPLHhPRw+0WIHL8hbwG95rcUGPiEhAYcOHUJSUhI456YZRWqiz+8BIqLm5mZMmjQJN27cMPsT
E9zYtWE25k4bYzh3mABIBRwOHPgPx+LH/6yDiXPi6pHZcEpk+I5qmtGpCzdQecCPPUevmevLsowj
R45gypQpcDgc5mVowKaJmBoQTEePHo2GhgaMGxcuum52BLHsqUOoP3oWgLB9Sz7QgpgxJoThw/SS
IutnaXBKZEpcgD998Uvs+3ubDbzT6URNTQ0mTZoEVVWhaRo459A0TeCy4etVA+JH8RvnHK2trZgz
Zw5Onz5tjnPJDrxSOAPLZt1nSVIqwDmIVJw6dx1nLt3A3J/cjZThkhE29cvf3o2KA59ja91lcz3G
GDZv3oylS5fC7XbD6XTC6XRCkiRIktS7Jm5HQLDXNA3Xr19HVlYWPvjgA4vEGLav+RHmTxsJX4rb
8AsO4kbOIG6GzTD4HvzlmB/PVH9q2zc/Px/Z2dnwer1wu92QZRkulwuSJEGWZZMEY8wkEGVCVvCC
AOcciqIgPj4e+/btw9SpU83xqkpYteU9HHm7Bf62rnC2RjikisTFSMPV9m4ceucannvVDn7p0qVY
uHAhgsEggsEgQqFQlAlZnVm0PjOxIKFpGogIqqrC5XKhuroa06dPt4wDcl78F+qPNeNKe6cRoTjs
ZYOKK+3dON7wBQrKm2H5pMbMmTOxZMkSBAIB9PT0oKenB4FAAIqiQFEUU4hRRyq3I2AlIZ4553A6
nSgvL8fs2bNtY9eVNaDub58YmhDZWpe+v70bJ5tu4clt/4WihqU4efJkrFixAkRkAg4Gg1AUBaqq
muDF/pGtX7WQNXeIZ0mSUFRUhAULFtjGPl3ZiJdev4QrbZ16zUQqrrZ342JLN5ZvaEJ3ICz68ePH
IycnB5qmmZIW4EOhUJTZWEKo+Rx1KsEYY9Zk4XA4IEmSKXnOuRkVJElCYWEhvF4v6urqzDVK9n2M
zu4Qnl5+L9q/7MHltgBWbr9kA5+eno7c3FzIsoyI5GrTeCzgosU8VmGMmSGLiEwSsixH2WNBQQEA
2EhUH/wc7330BaaOH479/2zHtS/CFabP58OaNWvg9Xp1EE4nZFmGLMtwu93wer1wuVy2ECpJko2I
iTNWJrZGI+HEwi6FisUVCAQQCASwZ88eVFZW9lk9JicnY/369fD5fPB4PCbo+Ph4k4TH44HL5YLX
64XX6zXH9EKE9aUBRkTEGLNJP1Kd4nI4HMjOzkZ8fDyKi4ttJ3eiJSYmYu3atRg1apQJzuv1wuPx
wOPxwOFwmEBFHnA6nXC5XGYSiywlbns2SnoDANN0hMNZnS0UCpkaOnPmDN566y20tLSY2ktLS8OM
GTOQkpICt9uNuLg4M2EJ8CJZSZJkZmFBwkrAWpH2+3BXEBEkRIhTVdVMOCL5qKqKy5cvQ1EUaJoG
VVWRnKx/Jlpt3eVy2QCKu9C68AFr9o0spwd0Oh1ZYggCIlNaiQktiXdhZgKQABcp3ci7LMtM7B0J
fsAErJoAYItG1mdr+hfvVlACtHDIXkploYXbHvQOikDEe6/ZmjFmAhf9ZvJxOk2Qol88W4kYz32S
GNQfHLG+TyMTkAVEr88xQfVnkBg7GAKixSIykDYQsL21/wFkW/B5QqT9lwAAAABJRU5ErkJggg==">
</td>
<td class="content">Be very careful, and do not overwrite the wrong disk!</td>
</tr></tbody></table>
</div>
<div class="sect4">
<h5 id="_find_the_correct_usb_device_name">Find the Correct USB Device Name
 <a class="headerlink" href="#_find_the_correct_usb_device_name" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>There are two ways to find out the name of the USB flash drive. The first one is
to compare the last lines of the <span class="monospaced">dmesg</span> command output before and after
plugging in the flash drive. The second way is to compare the output of the
<span class="monospaced">lsblk</span> command. Open a terminal and run:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># lsblk</pre>
</div></div>
<div class="paragraph">
<p>Then plug in your USB flash drive and run the command again:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># lsblk</pre>
</div></div>
<div class="paragraph">
<p>A new device will appear. This is the one you want to use. To be on the extra
safe side check if the reported size matches your USB flash drive.</p></div>
</div>
</div>
<div class="sect3">
<h4 id="_instructions_for_macos">2.2.3. Instructions for macOS
 <a class="headerlink" href="#_instructions_for_macos" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Open the terminal (query Terminal in Spotlight).</p></div>
<div class="paragraph">
<p>Convert the <span class="monospaced">.iso</span> file to <span class="monospaced">.dmg</span> format using the convert option of <span class="monospaced">hdiutil</span>,
for example:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># hdiutil convert proxmox-ve_*.iso -format UDRW -o proxmox-ve_*.dmg</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">macOS tends to automatically add <em>.dmg</em> to the output file name.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>To get the current list of devices run the command:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># diskutil list</pre>
</div></div>
<div class="paragraph">
<p>Now insert the USB flash drive and run this command again to determine which
device node has been assigned to it. (e.g., /dev/diskX).</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># diskutil list
# diskutil unmountDisk /dev/diskX</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">replace X with the disk number from the last command.</td>
</tr></tbody></table>
</div>
<div class="listingblock">
<div class="content monospaced">
<pre># sudo dd if=proxmox-ve_*.dmg bs=1M of=/dev/rdiskX</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content"><em>rdiskX</em>, instead of <em>diskX</em>, in the last command is intended. It will
increase the write speed.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="_instructions_for_windows">2.2.4. Instructions for Windows
 <a class="headerlink" href="#_instructions_for_windows" title="Permalink to this heading"></a>
</h4>
<div class="sect4">
<h5 id="_using_etcher">Using Etcher
 <a class="headerlink" href="#_using_etcher" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Etcher works out of the box. Download Etcher from <a href="https://etcher.io">https://etcher.io</a>. It will
guide you through the process of selecting the ISO and your USB flash drive.</p></div>
</div>
<div class="sect4">
<h5 id="_using_rufus">Using Rufus
 <a class="headerlink" href="#_using_rufus" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Rufus is a more lightweight alternative, but you need to use the <strong>DD mode</strong> to
make it work. Download Rufus from <a href="https://rufus.ie/">https://rufus.ie/</a>. Either install it or use
the portable version. Select the destination drive and the Proxmox VE ISO file.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Important" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAALa0lEQVRogdWZa2wc1RXHfzM7O/te
P9e1vXHSmEdjx3YeDkkaF6REKRRCEDSEFNkRjdSWSsgC2iqoRWqLQKiqIBg1NOQDiMeHtkQIQkRR
S9S4aWwgCQoUgl232KkT28J21l6vd3d2dx79sDuTXdtre03Uqlc62tl53Pv/n3PuOffcKxiGwf9z
E//XAL5sk65WR0a6AaDrOjMtKwgCoigiCAKCIAhXa9wvTcAEHhoc5KMjR7jw3nucf+MN67k/GCS4
YQOr77yTpl27kGUZm81mXC0ywlLnQDbwEwcO0P2b3yz4jV5RwZq2Nm5/5BH8fj+SJGGz2b4UiSUR
MAzD0DSNEwcO8MdHHln8d0AcsAUCfOe551h7yy04HA7sdvuSrVEwAV3XjdDgIC/v2cOl06dznqWA
JCAAMvn9MwlEgF1PP8037rsPt9uN3W5fkjUKIqDrujF+4QLP3XQTU0NDOc8UIAa4y8sJtrQQqK/H
7XQSfv99Pn/nHQRRxNB16/0EEAUatm9n76uv4vP5cDgcBZNYNIF84I0MEK20lIZvf5sNe/dSWlqK
1+vFxBEZHubT3/+evqNHmR4dJRWP55BevW0bra+8gt/vx+l0IknSokksikA+8ElgClixYwc3799P
ZWUlTqczbz+x0VFOPvEEQ6dPE/niCwxdt0jUbd1K68svU1xcXBCJBQnMBz5eWkrL/v2s++Y3KSsr
QxQXlxc7f/5zBjo7CQ8NoadSKKQn943t7dyyf79FYjHuNC8BwzAMVVV56Z57+OzNN3PAJ8vK+FZH
B2s2b55X6/nahRMnOP7TnxLOKEXJyO2/+hVf37uXoqIinE6nmfzyksirMjNUvvGjH+WAVwG1vJxb
Dhxg1bp1uFwuMwQWJCu3bWPN3r24SkoAcGbA/OXXv2bws8+IRqOkUqlZGX1RBAzDMHRd51+nTvHe
wYNX7gPTQMO+faxav56SkpIlgTdlU3s7TXv2IGUs6AGioRDH2tuZmJggFouhqirGPCzyEUBVVX7X
1pZzPwI0fP/7bNmzh7KysrzADh8+zP3338++ffs4ePBg3vdEUWTDD3/IsuZmRLsdW4bEpd5eOp99
lnA4jKIoaJqWl8QsAqb2Txw4kDNp40BxXR0bWlspLy/PC+qZZ55hdHQUSZIYHByko6ODxx57bBZw
cy3kq6xk6y9/ib+qCgQBmbQ7nT50iIs9PUSjUZLJJIZhzEliLgKMDQzwp5/9zLqnkQ51W3/xCwKB
AE6nMy+B3t5exsfH6ezspLe3l3g8zuHDh2cBz5bKpibWtrXhLi0FwA3owNsPP8zExATxeBxVVecy
QC4Bc+L+taNjluvU33031dddN6/2BUGgpKSE/v5+JicncweaA3i2tDz8MIHrr0eU0gsQBzDW10ff
yZNEIhESicScrpRDQNd1xgYG+OC3v7XuJQBXIMDWn/yEioqKeUGMjY0hCAKKoszS1MjIyLzfAmx/
4gmKli0DQcBJOmicfOopJicnicfjJoG5LWBq/+Szz+a8oABNe/bgdrvndR1BEKiqqiIejxOLxWYR
CAaDC0alqjVrqGxowFNWhkDalUKff07f3/5mWSFTLFksLAK6rjN+4UKO9lXAUV7O2tbWeaNOthZl
WZ4FHlh0aN3y4IPIHk+6L8AGvNfRQTgctuZCthVEU/u6rnPutddyBk0A9bt3W4us+QY2fbyurg6v
15vTzw033LBoAsHmZoLNzdgcDgBcQKi/n391dTE9PU0ymcyxgkmAVCrF+4cO5WjfFgiwfoGwOTOy
lJWVUVtba1mkUAKCILClvR1veTkA9owVPnzllZy8YFnA1H5/dzfTw8PWgziwcutWfD4fsiwvCNyU
xsZG3G43lZWVVl933XVXQQSq161j+ebNlhXcwBdnzxIOh60lhmkFEUDTNP7++uvWgBrp6mrNHBk3
H/BsAtFoFEdm8GAwyMaNGwsiIAgCjbt346uosKwgAp8cPWolNj1THInmsqHn6FGLQApY1thIxcqV
+Hy+WX6+0ETesWMHLpcLWZZ54IEHCgYvCAKrbrsNyem0JrQEXMzkhKzlBaKu61z+97+ZHhnJmbwr
tm2zJu5CWp8pra2tNDU1sXPnTtrb25dEQBAEvrplC57MXJCB8XPniEQixONxa6Uq6brOQHd3jvvo
QP2ttxZUpMxsL7zwwpK+AyxLXrt9Oxe6uxFEEXvGZc4fO0bVD35AMplM1wuapjGYtbugAp5AgKLq
aquu/W/JTEs37dqFqihWzSADk8PD1jzQNC3tQtmrThUINDbm+H4h0tfXx6OPPsodd9xBW1sbPT09
BQPPluu2b7fWRxIQ6usjFotZBCRN06wZbbZAfT1FRUWWKQtpzz//PMPDw/T09DA1NcXo6CjHjx+f
11Xma95AwHJjEdB0HUVRrhCYuRGrAiXBoFUqFtq6u7sZHx9nJBMUurq65uxnsX0Hm5v56A9/ANIJ
LXrpEolEwgqlUqZQyPlo6OOPlzx5ly1bxnBWQmxpackBW6hSBEC02axrpqasNZGmaek8kN0kYPDj
j1FVdUlz4Mknn6Surs4Cb5aUhYZiUy59+GGOgm3V1WiadiUPCILAV9autV5wAaM9PXQeOWJprBBp
aGjgxIkTpFIpOjs7aWxsXBJwUwZOnUKZmgLS2zl4vUjSlV1XURRFqtavv3IDKAVe/O53efPxx69a
SCxUBs+c4dV77+Vyfz+xy5eBdG3iX7ECm81m9S2JokiwuZmiujrCPT1AuqgOAqcef5zzL77Imt27
uemhhyipqVnYZ5cw8bPbR0eOcOall7g8MEB4aIhEJAJk9qOAwKZNyLJsHpQgRKNRIxQK0XPmDH/+
3vdQQ6G8na/YtIk199zDypYWVmzceFWAT1y8yMWzZzn/9tv0vfsudqeTlKIQHRvL2QSOAqU7d/K1
W2/l2muvpba2Nl3iJhIJIxKJMDw8TO/Zs3zw4x+jTUzkHVCw2fCUleHw+QiuXYu/upqqhgb8VVUY
hkFNczPFweCs7z556y3r+vyxYwiiyD+PHwdBwNB1krEYhqYRC4UwdB2D9LJmmnRtXHz77VTfeCPL
ly+ntraWmpqa9MaaqqqGoiiEQiGGhoYYHBzk00OHiHR2LkqDNlnG7nJhd7nQswqN+ZogiuipFMlY
DDWRgBl5KEF6woqAUFFByc03U756NZWVlSxfvpyamhoCgQAejwdB13VDVVVisRihUIiRkRGGh4cZ
+sc/GO3qIt7VhZGJAle7mVpWuVKDCBng8jXX4N68Gc8111BUVER5eTnV1dVUV1cTCASuFFqGYaDr
upFKpYjH40xMTDA+Ps7Y2Bjj4+OEw2EmenqYPneO1OAgZCWpxTY9S9QMcLhyHGUDRI8HqbISZ309
zoYGnE4nHo8Hv99PaWkpgUCAiooK6/DE4XCkI1HW2a6hqiqKojA9Pc3k5KQl4XCY6elpotFoetuk
v59UKIR66RLToRBJTcMYHUWIRvMSMPO6rbY2/SsIOFatQpYk5NWrkSQJWZZxOBy43W68Xi9+v5+i
oiJKSkooLi7G5/PhdruRZdnads85HzD3hpLJJIqiEIvFiEajRCIRIpEI0WiUWCxGPB631iOKopBK
pazUbi4Oc7Y+snKCzWbDZrMhSRKSJGG3262w6HK5cLlceDwevF4vPp8Pr9eL2+3G5XLhcDjMkxuE
TNibdcBhnv9qmkYqlSKZTJJIJFAUhXg8jqIoKIpCIpGwSCSTSVRVtWSuk3qTiAnebrfngHc4HDid
TpxOJy6Xy7qWZdk6wZzrxCbvCY1JRNd1NE3LAWgSM69ngp+PgCiK1gF3NhFTzHvmO+Y3wCzw8xLI
JpL5tYCZpEyw2f/nWt1ag2UtM0x3Mq1iAp1j+bH0M7KFCGWDnfmbd8AMnuxfUzL/C0rp/wFnFd4n
EQn3XQAAAABJRU5ErkJggg==">
</td>
<td class="content">Once you <em>Start</em> you have to click <em>No</em> on the dialog asking to
download a different version of GRUB. In the next dialog select the <em>DD</em> mode.</td>
</tr></tbody></table>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="installation_installer">
<span>2.3. Using the Proxmox VE Installer</span>
 <a class="headerlink" href="#installation_installer" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>The installer ISO image includes the following:</p></div>
<div class="ulist"><ul>
<li>
<p>
Complete operating system (Debian Linux, 64-bit)
</p>
</li>
<li>
<p>
The Proxmox VE installer, which partitions the local disk(s) with ext4, XFS,
  BTRFS (technology preview), or ZFS and installs the operating system
</p>
</li>
<li>
<p>
Proxmox VE Linux kernel with KVM and LXC support
</p>
</li>
<li>
<p>
Complete toolset for administering virtual machines, containers, the host
  system, clusters and all necessary resources
</p>
</li>
<li>
<p>
Web-based management interface
</p>
</li>
</ul></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">All existing data on the selected drives will be removed during the
installation process. The installer does not add boot menu entries for other
operating systems.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>Please insert the <a href="#installation_prepare_media">prepared installation media</a>
(for example, USB flash drive or CD-ROM) and boot from it.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">Make sure that booting from the installation medium (for example, USB) is
enabled in your server’s firmware settings. Secure boot needs to be disabled
when booting an installer prior to Proxmox VE version 8.1.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<a class="image" href="images/screenshot/pve-grub-menu.png">
<img src="images/screenshot/pve-grub-menu.png" alt="screenshot/pve-grub-menu.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>After choosing the correct entry (for example, <em>Boot from USB</em>) the Proxmox VE menu
will be displayed, and one of the following options can be selected:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
Install Proxmox VE (Graphical)
</dt>
<dd>
<p>
Starts the normal installation.
</p>
</dd>
</dl></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">It’s possible to use the installation wizard with a keyboard only. Buttons
can be clicked by pressing the <span class="monospaced">ALT</span> key combined with the underlined character
from the respective button. For example, <span class="monospaced">ALT + N</span> to press a <span class="monospaced">Next</span> button.</td>
</tr></tbody></table>
</div>
<div class="dlist"><dl>
<dt class="hdlist1">
Install Proxmox VE (Terminal UI)
</dt>
<dd>
<p>
Starts the terminal-mode installation wizard. It provides the same overall
installation experience as the graphical installer, but has generally better
compatibility with very old and very new hardware.
</p>
</dd>
<dt class="hdlist1">
Install Proxmox VE (Terminal UI, Serial Console)
</dt>
<dd>
<p>
Starts the terminal-mode installation wizard, additionally setting up the Linux
kernel to use the (first) serial port of the machine for in- and output. This
can be used if the machine is completely headless and only has a serial console
available.
</p>
</dd>
</dl></div>
<div class="paragraph">
<a class="image" href="images/screenshot/pve-tui-installer.png">
<img src="images/screenshot/pve-tui-installer.png" alt="screenshot/pve-tui-installer.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>Both modes use the same code base for the actual installation process to
benefit from more than a decade of bug fixes and ensure feature parity.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">The <em>Terminal UI</em> option can be used in case the graphical installer does
not work correctly, due to e.g. driver issues. See also
<a href="#nomodeset_kernel_param">adding the <span class="monospaced">nomodeset</span> kernel parameter</a>.</td>
</tr></tbody></table>
</div>
<div class="dlist"><dl>
<dt class="hdlist1">
Advanced Options: Install Proxmox VE (Graphical, Debug Mode)
</dt>
<dd>
<p>
Starts the installation in debug mode. A console will be opened at several
installation steps. This helps to debug the situation if something goes wrong.
To exit a debug console, press <span class="monospaced">CTRL-D</span>. This option can be used to boot a live
system with all basic tools available. You can use it, for example, to
<a href="#chapter_zfs">repair a degraded ZFS <em>rpool</em></a> or fix the
<a href="#sysboot">bootloader</a> for an existing Proxmox VE setup.
</p>
</dd>
<dt class="hdlist1">
Advanced Options: Install Proxmox VE (Terminal UI, Debug Mode)
</dt>
<dd>
<p>
Same as the graphical debug mode, but preparing the system to run the
terminal-based installer instead.
</p>
</dd>
<dt class="hdlist1">
Advanced Options: Install Proxmox VE (Serial Console Debug Mode)
</dt>
<dd>
<p>
Same the terminal-based debug mode, but additionally sets up the Linux kernel to
use the (first) serial port of the machine for in- and output.
</p>
</dd>
<dt class="hdlist1">
Advanced Options: Rescue Boot
</dt>
<dd>
<p>
With this option you can boot an existing installation. It searches all attached
hard disks. If it finds an existing installation, it boots directly into that
disk using the Linux kernel from the ISO. This can be useful if there are
problems with the bootloader (GRUB/<span class="monospaced">systemd-boot</span>) or the BIOS/UEFI is unable to
read the boot block from the disk.
</p>
</dd>
<dt class="hdlist1">
Advanced Options: Test Memory (memtest86+)
</dt>
<dd>
<p>
Runs <span class="monospaced">memtest86+</span>. This is useful to check if the memory is functional and free
of errors. Secure Boot must be turned off in the UEFI firmware setup utility to
run this option.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p>You normally select <strong>Install Proxmox VE (Graphical)</strong> to start the installation.</p></div>
<div class="paragraph">
<a class="image" href="images/screenshot/pve-select-target-disk.png">
<img src="images/screenshot/pve-select-target-disk.png" alt="screenshot/pve-select-target-disk.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>The first step is to read our EULA (End User License Agreement). Following this,
you can select the target hard disk(s) for the installation.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Caution" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKdUlEQVRoge1Ze1AV1x3+zt279wEi
DWCYGzRVktqa1MEmmtbWR22ncXxUrTDWV/5IqG2wUAUfmUwSkk6V+EAJCEQC6figwZBqZtRxqukf
tebRZBpFG1S0hiRErwiaKK977+6eX//YPXt37+UiIJlMZnpmdnb33PP4vt97z2VEhG9yc3zdAO60
/Z/A192+8QScX8Wifr+fWltbzffU1FT4fD72Vew15ASampqovr4eBw8eNPvGjRuHzMxMmj9//tCT
IKIhu958801yuVwEoNdr48aNNJT7EdHQEdi/fz/JshwTvLiKioqGlMSQLFJfX0+MMRvQsWNSqXLz
H2jez78fRWLTpk1DRuKOF6irqyOn02kDOP8XGdTz+VFSP91Hatu79NRvZ0SR2LJly5CQuKPJtbW1
JEmSDdivHp1AwctHSblYSsqFElLOF5PavIvWZU+LIrF169Y7JjHoibt3744CnzXrIQpeOUbKhVJS
mkpIPV9MyrnNpJwtIuVSNa15/MdRJLZt23ZHJBjRwIu5N954gxYvXgzOudm3aO5E1L5SBNZ1Hoxr
INLAyH6HJxVP/WkXtu9+37ZeSUkJVq9ePagQO+BMXFNTEwV+ybxHsPeVIjg6zwOkgSWNhuOuUUBC
KhxJY0AggDSgpwWbn1mO1csftq1ZUFCA0tLSQZXFA0pkVVVVlJOTA6vWsmY9hF07N0AS4OU48LgR
WLO+FPFxboxMS8G5hlPYmPMI4jwM1P0Jtj67DJyrKHvtNADdjPPz8+FwOCgvL29gmuivrVVUVESF
yqxZD1PoyjFSL+7QHbapmJTml+n9EzVRtl676ZekNKwl5VQ+KSfzKHR2E+UutodYxhiVl5cPyCf6
ZUJlZWWUm5trk/wTi6bgtVdf1M2Gq2DEwUgDNBWhUDBqjfSRwwGoIOEP3c3Y/uxirFw0zibMvLw8
VFZW9tucbkugoqKCVq1aZQO/YslUVJU+D9bVBIDroLgK4hygEDweybYGY8AD6d8yftfASL+j+2O8
9NwiPJk51kYiNzcXO3fu7BeJPgkUFxdTbm6ure/JZdNRub0QrOMcGFcBrgLgAHEAGqD2ID7OZZsz
Ji0RcR6HAV4DkaqPJxXouoSywkz8buF9NhIrV65EVVXVbUnEJHDixAlat26drS9n+U+xo/hZMMNh
yZA+uAbAIKOGMCJlmG3eA/clAYaJEamGBlRdC6QBXZewo3AhVswfbSORk5ODDz/8sE8SvRLw+/1U
XV0dBb5MgOe6LUNIEqpOwgCUlOCESw4HuAfG3KX7B2kAGaTJ0Jro77yI8ucX4DfzRtlIHD58GH6/
PyaJXgk0NjZi79695vuUiffrkr91Vt8cHAwc4FZJGiZEKqDcwsi0ZHP+d749POy8MS7iGtBxARWF
8/CDsQnm3BdeeAHWj6N+EYhspSVbgI5zYYkZGmAwzIEbwA0tUM9NfO+7aeb89HuGRQBWLcSNZxj3
zvOYMzk5NpjBEHi99mXD5lXdjqGDJh42h7CEOXhnK3448X5z/shUb1jSpIEM8yFuzOUcjAsNcnR0
KXdG4MEHH8TEiRPN9z0H3sfRE41m+GNcs9i0LjkGTTcrUoGem5g7/V5z/ohElw4O+jxGYQ3qJqmB
jN/+cbINtcfCJvPYY4+hr3qtVwI+n49lZmaa71fbbuH3hX/FycbPQKSCoIIMQEQcehjVo5Gw8/F3
3cSvH03H2HuHY1icQzcvYWKGIMiITMwQwjtnruPpqk9w/ZZq7p2RkYEJEyaAYrCIWY36/X7KyMhA
W1ub2XfP3Qmo3ZKFaQ+P0gmYIdSoOkU4lWSwxGRAdqHL/xm86k27w0blAw3/PncD6yo/xrsfdZj7
paSkoLGxESNGjAgDZsxWK8X0AZ/Px44fP460tLAzXrnWgUX5r+PYuxf1mG/Gf80sofWy2YuWbg/+
uONtPLHhPRw+0WIHL8hbwG95rcUGPiEhAYcOHUJSUhI456YZRWqiz+8BIqLm5mZMmjQJN27cMPsT
E9zYtWE25k4bYzh3mABIBRwOHPgPx+LH/6yDiXPi6pHZcEpk+I5qmtGpCzdQecCPPUevmevLsowj
R45gypQpcDgc5mVowKaJmBoQTEePHo2GhgaMGxcuum52BLHsqUOoP3oWgLB9Sz7QgpgxJoThw/SS
IutnaXBKZEpcgD998Uvs+3ubDbzT6URNTQ0mTZoEVVWhaRo459A0TeCy4etVA+JH8RvnHK2trZgz
Zw5Onz5tjnPJDrxSOAPLZt1nSVIqwDmIVJw6dx1nLt3A3J/cjZThkhE29cvf3o2KA59ja91lcz3G
GDZv3oylS5fC7XbD6XTC6XRCkiRIktS7Jm5HQLDXNA3Xr19HVlYWPvjgA4vEGLav+RHmTxsJX4rb
8AsO4kbOIG6GzTD4HvzlmB/PVH9q2zc/Px/Z2dnwer1wu92QZRkulwuSJEGWZZMEY8wkEGVCVvCC
AOcciqIgPj4e+/btw9SpU83xqkpYteU9HHm7Bf62rnC2RjikisTFSMPV9m4ceucannvVDn7p0qVY
uHAhgsEggsEgQqFQlAlZnVm0PjOxIKFpGogIqqrC5XKhuroa06dPt4wDcl78F+qPNeNKe6cRoTjs
ZYOKK+3dON7wBQrKm2H5pMbMmTOxZMkSBAIB9PT0oKenB4FAAIqiQFEUU4hRRyq3I2AlIZ4553A6
nSgvL8fs2bNtY9eVNaDub58YmhDZWpe+v70bJ5tu4clt/4WihqU4efJkrFixAkRkAg4Gg1AUBaqq
muDF/pGtX7WQNXeIZ0mSUFRUhAULFtjGPl3ZiJdev4QrbZ16zUQqrrZ342JLN5ZvaEJ3ICz68ePH
IycnB5qmmZIW4EOhUJTZWEKo+Rx1KsEYY9Zk4XA4IEmSKXnOuRkVJElCYWEhvF4v6urqzDVK9n2M
zu4Qnl5+L9q/7MHltgBWbr9kA5+eno7c3FzIsoyI5GrTeCzgosU8VmGMmSGLiEwSsixH2WNBQQEA
2EhUH/wc7330BaaOH479/2zHtS/CFabP58OaNWvg9Xp1EE4nZFmGLMtwu93wer1wuVy2ECpJko2I
iTNWJrZGI+HEwi6FisUVCAQQCASwZ88eVFZW9lk9JicnY/369fD5fPB4PCbo+Ph4k4TH44HL5YLX
64XX6zXH9EKE9aUBRkTEGLNJP1Kd4nI4HMjOzkZ8fDyKi4ttJ3eiJSYmYu3atRg1apQJzuv1wuPx
wOPxwOFwmEBFHnA6nXC5XGYSiywlbns2SnoDANN0hMNZnS0UCpkaOnPmDN566y20tLSY2ktLS8OM
GTOQkpICt9uNuLg4M2EJ8CJZSZJkZmFBwkrAWpH2+3BXEBEkRIhTVdVMOCL5qKqKy5cvQ1EUaJoG
VVWRnKx/Jlpt3eVy2QCKu9C68AFr9o0spwd0Oh1ZYggCIlNaiQktiXdhZgKQABcp3ci7LMtM7B0J
fsAErJoAYItG1mdr+hfvVlACtHDIXkploYXbHvQOikDEe6/ZmjFmAhf9ZvJxOk2Qol88W4kYz32S
GNQfHLG+TyMTkAVEr88xQfVnkBg7GAKixSIykDYQsL21/wFkW/B5QqT9lwAAAABJRU5ErkJggg==">
</td>
<td class="content">By default, the whole server is used and all existing data is removed.
Make sure there is no important data on the server before proceeding with the
installation.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>The <span class="monospaced">Options</span> button lets you select the target file system, which
defaults to <span class="monospaced">ext4</span>. The installer uses LVM if you select
<span class="monospaced">ext4</span> or <span class="monospaced">xfs</span> as a file system, and offers additional options to
restrict LVM space (see <a href="#advanced_lvm_options">below</a>).</p></div>
<div class="paragraph">
<p>Proxmox VE can also be installed on ZFS. As ZFS offers several software RAID levels,
this is an option for systems that don’t have a hardware RAID controller. The
target disks must be selected in the <span class="monospaced">Options</span> dialog. More ZFS specific
settings can be changed under <a href="#advanced_zfs_options"><span class="monospaced">Advanced Options</span></a>.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">ZFS on top of any hardware RAID is not supported and can result in data
loss.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<a class="image" href="images/screenshot/pve-select-location.png">
<img src="images/screenshot/pve-select-location.png" alt="screenshot/pve-select-location.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>The next page asks for basic configuration options like your location, time
zone, and keyboard layout. The location is used to select a nearby download
server, in order to increase the speed of updates. The installer is usually able
to auto-detect these settings, so you only need to change them in rare
situations when auto-detection fails, or when you want to use a keyboard layout
not commonly used in your country.</p></div>
<div class="paragraph">
<a class="image" href="images/screenshot/pve-set-password.png">
<img src="images/screenshot/pve-set-password.png" alt="screenshot/pve-set-password.png" width="250" style="padding: 0 10px 0 0;float:left;"></a>
<p>Next the password of the superuser (<span class="monospaced">root</span>) and an email address needs to be
specified. The password must consist of at least 5 characters. It’s highly
recommended to use a stronger password. Some guidelines are:</p></div>
<div class="ulist"><ul>
<li>
<p>
Use a minimum password length of at least 12 characters.
</p>
</li>
<li>
<p>
Include lowercase and uppercase alphabetic characters, numbers, and symbols.
</p>
</li>
<li>
<p>
Avoid character repetition, keyboard patterns, common dictionary words,
  letter or number sequences, usernames, relative or pet names, romantic links
  (current or past), and biographical information (for example ID numbers,
  ancestors' names or dates).
</p>
</li>
</ul></div>
<div class="paragraph">
<p>The email address is used to send notifications to the system administrator.
For example:</p></div>
<div class="ulist"><ul>
<li>
<p>
Information about available package updates.
</p>
</li>
<li>
<p>
Error messages from periodic <em>cron</em> jobs.
</p>
</li>
</ul></div>
<div class="paragraph">
<a class="image" href="images/screenshot/pve-setup-network.png">
<img src="images/screenshot/pve-setup-network.png" alt="screenshot/pve-setup-network.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>All those notification mails will be sent to the specified email address.</p></div>
<div class="paragraph">
<p>The last step is the network configuration. Network interfaces that are <em>UP</em>
show a filled circle in front of their name in the drop down menu. Please note
that during installation you can either specify an IPv4 or IPv6 address, but not
both. To configure a dual stack node, add additional IP addresses after the
installation.</p></div>
<div class="paragraph">
<a class="image" href="images/screenshot/pve-installation.png">
<img src="images/screenshot/pve-installation.png" alt="screenshot/pve-installation.png" width="250" style="padding: 0 10px 0 0;float:left;"></a>
<p>The next step shows a summary of the previously selected options. Please
re-check every setting and use the <span class="monospaced">Previous</span> button if a setting needs to be
changed.</p></div>
<div class="paragraph">
<p>After clicking <span class="monospaced">Install</span>, the installer will begin to format the disks and copy
packages to the target disk(s). Please wait until this step has finished; then
remove the installation medium and restart your system.</p></div>
<div class="paragraph">
<a class="image" href="images/screenshot/pve-install-summary.png">
<img src="images/screenshot/pve-install-summary.png" alt="screenshot/pve-install-summary.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>Copying the packages usually takes several minutes, mostly depending on the
speed of the installation medium and the target disk performance.</p></div>
<div class="paragraph">
<p>When copying and setting up the packages has finished, you can reboot the
server. This will be done automatically after a few seconds by default.</p></div>
<div class="paragraph">
<div class="title">Installation Failure</div><p>If the installation failed, check out specific errors on the second TTY
(<em>CTRL + ALT + F2</em>) and ensure that the systems meets the
<a href="#install_minimal_requirements">minimum requirements</a>.</p></div>
<div class="paragraph">
<p>If the installation is still not working, look at the
<a href="#getting_help">how to get help chapter</a>.</p></div>
<div class="sect3">
<h4 id="_accessing_the_management_interface_post_installation">2.3.1. Accessing the Management Interface Post-Installation
 <a class="headerlink" href="#_accessing_the_management_interface_post_installation" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-login-window.png">
<img src="images/screenshot/gui-login-window.png" alt="screenshot/gui-login-window.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>After a successful installation and reboot of the system you can use the Proxmox VE
web interface for further configuration.</p></div>
<div class="olist arabic"><ol class="arabic">
<li>
<p>
Point your browser to the IP address given during the installation and port
  8006, for example: <a href="https://youripaddress:8006">https://youripaddress:8006</a>
</p>
</li>
<li>
<p>
Log in using the <span class="monospaced">root</span> (realm <em>PAM</em>) username and the password chosen during
  installation.
</p>
</li>
<li>
<p>
Upload your subscription key to gain access to the Enterprise repository.
  Otherwise, you will need to set up one of the public, less tested package
  repositories to get updates for security fixes, bug fixes, and new features.
</p>
</li>
<li>
<p>
Check the IP configuration and hostname.
</p>
</li>
<li>
<p>
Check the timezone.
</p>
</li>
<li>
<p>
Check your <a href="#chapter_pve_firewall">Firewall settings</a>.
</p>
</li>
</ol></div>
</div>
<div class="sect3">
<h4 id="advanced_lvm_options">2.3.2. Advanced LVM Configuration Options
 <a class="headerlink" href="#advanced_lvm_options" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The installer creates a Volume Group (VG) called <span class="monospaced">pve</span>, and additional Logical
Volumes (LVs) called <span class="monospaced">root</span>, <span class="monospaced">data</span>, and <span class="monospaced">swap</span>, if <span class="monospaced">ext4</span> or <span class="monospaced">xfs</span> is used. To
control the size of these volumes use:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">hdsize</span>
</dt>
<dd>
<p>
Defines the total hard disk size to be used. This way you can reserve free space
on the hard disk for further partitioning (for example for an additional PV and
VG on the same hard disk that can be used for LVM storage).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">swapsize</span>
</dt>
<dd>
<p>
Defines the size of the <span class="monospaced">swap</span> volume. The default is the size of the installed
memory, minimum 4 GB and maximum 8 GB. The resulting value cannot be greater
than <span class="monospaced">hdsize/8</span>.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">If set to <span class="monospaced">0</span>, no <span class="monospaced">swap</span> volume will be created.</td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">maxroot</span>
</dt>
<dd>
<p>
Defines the maximum size of the <span class="monospaced">root</span> volume, which stores the operation
system. The maximum limit of the <span class="monospaced">root</span> volume size is <span class="monospaced">hdsize/4</span>.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">maxvz</span>
</dt>
<dd>
<p>
Defines the maximum size of the <span class="monospaced">data</span> volume. The actual size of the <span class="monospaced">data</span>
volume is:
</p>
<div class="paragraph">
<p><span class="monospaced">datasize = hdsize - rootsize - swapsize - minfree</span></p></div>
<div class="paragraph">
<p>Where <span class="monospaced">datasize</span> cannot be bigger than <span class="monospaced">maxvz</span>.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">In case of LVM thin, the <span class="monospaced">data</span> pool will only be created if <span class="monospaced">datasize</span> is
bigger than 4GB.</td>
</tr></tbody></table>
</div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">If set to <span class="monospaced">0</span>, no <span class="monospaced">data</span> volume will be created and the storage
configuration will be adapted accordingly.</td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">minfree</span>
</dt>
<dd>
<p>
Defines the amount of free space that should be left in the LVM volume group
<span class="monospaced">pve</span>. With more than 128GB storage available, the default is 16GB, otherwise
<span class="monospaced">hdsize/8</span> will be used.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">LVM requires free space in the VG for snapshot creation (not required for
lvmthin snapshots).</td>
</tr></tbody></table>
</div>
</dd>
</dl></div>
</div>
<div class="sect3">
<h4 id="advanced_zfs_options">2.3.3. Advanced ZFS Configuration Options
 <a class="headerlink" href="#advanced_zfs_options" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The installer creates the ZFS pool <span class="monospaced">rpool</span>, if ZFS is used. No swap space is
created but you can reserve some unpartitioned space on the install disks for
swap. You can also create a swap zvol after the installation, although this can
lead to problems (see <a href="#zfs_swap">ZFS swap notes</a>).</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">ashift</span>
</dt>
<dd>
<p>
Defines the <span class="monospaced">ashift</span> value for the created pool. The <span class="monospaced">ashift</span> needs to be set at
least to the sector-size of the underlying disks (2 to the power of <span class="monospaced">ashift</span> is
the sector-size), or any disk which might be put in the pool (for example the
replacement of a defective disk).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">compress</span>
</dt>
<dd>
<p>
Defines whether compression is enabled for <span class="monospaced">rpool</span>.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">checksum</span>
</dt>
<dd>
<p>
Defines which checksumming algorithm should be used for <span class="monospaced">rpool</span>.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">copies</span>
</dt>
<dd>
<p>
Defines the <span class="monospaced">copies</span> parameter for <span class="monospaced">rpool</span>. Check the <span class="monospaced">zfs(8)</span> manpage for the
semantics, and why this does not replace redundancy on disk-level.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">ARC max size</span>
</dt>
<dd>
<p>
Defines the maximum size the ARC can grow to and thus limits the amount of
memory ZFS will use. See also the section on
<a href="#sysadmin_zfs_limit_memory_usage">how to limit ZFS memory usage</a> for more
details.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">hdsize</span>
</dt>
<dd>
<p>
Defines the total hard disk size to be used. This is useful to save free space
on the hard disk(s) for further partitioning (for example to create a
swap-partition). <span class="monospaced">hdsize</span> is only honored for bootable disks, that is only the
first disk or mirror for RAID0, RAID1 or RAID10, and all disks in RAID-Z[123].
</p>
</dd>
</dl></div>
</div>
<div class="sect3">
<h4 id="_zfs_performance_tips">2.3.4. ZFS Performance Tips
 <a class="headerlink" href="#_zfs_performance_tips" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>ZFS works best with a lot of memory. If you intend to use ZFS make sure to have
enough RAM available for it. A good calculation is 4GB plus 1GB RAM for each TB
RAW disk space.</p></div>
<div class="paragraph">
<p>ZFS can use a dedicated drive as write cache, called the ZFS Intent Log (ZIL).
Use a fast drive (SSD) for it. It can be added after installation with the
following command:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># zpool add &lt;pool-name&gt; log &lt;/dev/path_to_fast_ssd&gt;</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="nomodeset_kernel_param">2.3.5. Adding the <span class="monospaced">nomodeset</span> Kernel Parameter
 <a class="headerlink" href="#nomodeset_kernel_param" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Problems may arise on very old or very new hardware due to graphics drivers. If
the installation hangs during boot, you can try adding the <span class="monospaced">nomodeset</span>
parameter. This prevents the Linux kernel from loading any graphics drivers and
forces it to continue using the BIOS/UEFI-provided framebuffer.</p></div>
<div class="paragraph">
<p>On the Proxmox VE bootloader menu, navigate to <em>Install Proxmox VE (Terminal UI)</em> and
press <span class="monospaced">e</span> to edit the entry. Using the arrow keys, navigate to the line starting
with <span class="monospaced">linux</span>, move the cursor to the end of that line and add the
parameter <span class="monospaced">nomodeset</span>, separated by a space from the pre-existing last
parameter.</p></div>
<div class="paragraph">
<p>Then press <span class="monospaced">Ctrl-X</span> or <span class="monospaced">F10</span> to boot the configuration.</p></div>
</div>
</div>
<div class="sect2">
<h3 id="installation_unattended">
<span>2.4. Unattended Installation</span>
 <a class="headerlink" href="#installation_unattended" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>It is possible to install Proxmox VE automatically in an unattended manner. This
enables you to fully automate the setup process on bare-metal. Once the
installation is complete and the host has booted up, automation tools like
Ansible can be used to further configure the installation.</p></div>
<div class="paragraph">
<p>The necessary options for the installer must be provided in an answer file. This
file allows the use of filter rules to determine which disks and network cards
should be used.</p></div>
<div class="paragraph">
<p>To use the automated installation, it is first necessary to prepare an
installation ISO.
<a href="https://pve.proxmox.com/wiki/Automated_Installation">Visit our wiki</a> for more
details and information on the unattended installation.</p></div>
</div>
<div class="sect2">
<h3 id="_install_proxmox_ve_on_debian">
<span>2.5. Install Proxmox VE on Debian</span>
 <a class="headerlink" href="#_install_proxmox_ve_on_debian" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Proxmox VE ships as a set of Debian packages and can be installed on top of a standard
Debian installation.
<a href="#sysadmin_package_repositories">After configuring the repositories</a> you need
to run the following commands:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># apt-get update
# apt-get install proxmox-ve</pre>
</div></div>
<div class="paragraph">
<p>Installing on top of an existing Debian installation looks easy, but it presumes
that the base system has been installed correctly and that you know how you want
to configure and use the local storage. You also need to configure the network
manually.</p></div>
<div class="paragraph">
<p>In general, this is not trivial, especially when LVM or ZFS is used.</p></div>
<div class="paragraph">
<p>A detailed step by step how-to can be found on the
<a href="https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_12_Bookworm">wiki</a>.</p></div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="chapter_system_administration">
3. Host System Administration
 <a class="headerlink" href="#chapter_system_administration" title="Permalink to this heading"></a>
</h2>
<div class="sectionbody">
<div class="paragraph">
<p>The following sections will focus on common virtualization tasks and explain the
Proxmox VE specifics regarding the administration and management of the host machine.</p></div>
<div class="paragraph">
<p>Proxmox VE is based on <a href="https://www.debian.org/">Debian GNU/Linux</a> with additional
repositories to provide the Proxmox VE related packages. This means that the full
range of Debian packages is available including security updates and bug fixes.
Proxmox VE provides its own Linux kernel based on the Ubuntu kernel. It has all the
necessary virtualization and container features enabled and includes
<a href="https://zfsonlinux.org">ZFS</a> and several extra hardware drivers.</p></div>
<div class="paragraph">
<p>For other topics not included in the following sections, please refer to the
Debian documentation. The
<a href="https://debian-handbook.info/get">Debian
Administrator's Handbook</a> is available online, and provides a comprehensive
introduction to the Debian operating system (see <a href="#Hertzog13">[Hertzog13]</a>).</p></div>
<div class="sect2">
<h3 id="sysadmin_package_repositories">
<span>3.1. Package Repositories</span>
 <a class="headerlink" href="#sysadmin_package_repositories" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Proxmox VE uses <a href="http://en.wikipedia.org/wiki/Advanced_Packaging_Tool">APT</a> as its
package management tool like any other Debian-based system.</p></div>
<div class="paragraph">
<p>Proxmox VE automatically checks for package updates on a daily basis. The <span class="monospaced">root@pam</span>
user is notified via email about available updates. From the GUI, the
<em>Changelog</em> button can be used to see more details about an selected update.</p></div>
<div class="sect3">
<h4 id="_repositories_in_proxmox_ve">3.1.1. Repositories in Proxmox VE
 <a class="headerlink" href="#_repositories_in_proxmox_ve" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Repositories are a collection of software packages, they can be used to install
new software, but are also important to get new updates.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">You need valid Debian and Proxmox repositories to get the latest
security updates, bug fixes and new features.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>APT Repositories are defined in the file <span class="monospaced">/etc/apt/sources.list</span> and in <span class="monospaced">.list</span>
files placed in <span class="monospaced">/etc/apt/sources.list.d/</span>.</p></div>
<div class="sect4">
<h5 id="_repository_management">Repository Management
 <a class="headerlink" href="#_repository_management" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-node-repositories.png">
<img src="images/screenshot/gui-node-repositories.png" alt="screenshot/gui-node-repositories.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>Since Proxmox VE 7, you can check the repository state in the web interface.
The node summary panel shows a high level status overview, while the separate
<em>Repository</em> panel shows in-depth status and list of all configured
repositories.</p></div>
<div class="paragraph">
<p>Basic repository management, for example, activating or deactivating a
repository, is also supported.</p></div>
</div>
<div class="sect4">
<h5 id="_sources_list">Sources.list
 <a class="headerlink" href="#_sources_list" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>In a <span class="monospaced">sources.list</span> file, each line defines a package repository. The preferred
source must come first.  Empty lines are ignored. A <span class="monospaced">#</span> character anywhere on a
line marks the remainder of that line as a comment. The available packages from
a repository are acquired by running <span class="monospaced">apt-get update</span>. Updates can be installed
directly using <span class="monospaced">apt-get</span>, or via the GUI (Node → Updates).</p></div>
<div class="listingblock">
<div class="title">File <span class="monospaced">/etc/apt/sources.list</span></div>
<div class="content monospaced">
<pre>deb http://deb.debian.org/debian bookworm main contrib
deb http://deb.debian.org/debian bookworm-updates main contrib

# security updates
deb http://security.debian.org/debian-security bookworm-security main contrib</pre>
</div></div>
<div class="paragraph">
<p>Proxmox VE provides three different package repositories.</p></div>
</div>
</div>
<div class="sect3">
<h4 id="sysadmin_enterprise_repo">3.1.2. Proxmox VE Enterprise Repository
 <a class="headerlink" href="#sysadmin_enterprise_repo" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>This is the recommended repository and available for all Proxmox VE subscription
users. It contains the most stable packages and is suitable for production use.
The <span class="monospaced">pve-enterprise</span> repository is enabled by default:</p></div>
<div class="listingblock">
<div class="title">File <span class="monospaced">/etc/apt/sources.list.d/pve-enterprise.list</span></div>
<div class="content monospaced">
<pre>deb https://enterprise.proxmox.com/debian/pve bookworm pve-enterprise</pre>
</div></div>
<div class="paragraph">
<p>Please note that you need a valid subscription key to access the
<span class="monospaced">pve-enterprise</span> repository. We offer different support levels, which you can
find further details about at <a href="https://proxmox.com/en/proxmox-virtual-environment/pricing">https://proxmox.com/en/proxmox-virtual-environment/pricing</a>.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">You can disable this repository by commenting out the above line using a
<span class="monospaced">#</span> (at the start of the line). This prevents error messages if your host does
not have a subscription key. Please configure the <span class="monospaced">pve-no-subscription</span>
repository in that case.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="sysadmin_no_subscription_repo">3.1.3. Proxmox VE No-Subscription Repository
 <a class="headerlink" href="#sysadmin_no_subscription_repo" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>As the name suggests, you do not need a subscription key to access
this repository. It can be used for testing and non-production
use. It’s not recommended to use this on production servers, as these
packages are not always as heavily tested and validated.</p></div>
<div class="paragraph">
<p>We recommend to configure this repository in <span class="monospaced">/etc/apt/sources.list</span>.</p></div>
<div class="listingblock">
<div class="title">File <span class="monospaced">/etc/apt/sources.list</span></div>
<div class="content monospaced">
<pre>deb http://ftp.debian.org/debian bookworm main contrib
deb http://ftp.debian.org/debian bookworm-updates main contrib

# Proxmox VE pve-no-subscription repository provided by proxmox.com,
# NOT recommended for production use
deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription

# security updates
deb http://security.debian.org/debian-security bookworm-security main contrib</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="sysadmin_test_repo">3.1.4. Proxmox VE Test Repository
 <a class="headerlink" href="#sysadmin_test_repo" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>This repository contains the latest packages and is primarily used by developers
to test new features. To configure it, add the following line to
<span class="monospaced">/etc/apt/sources.list</span>:</p></div>
<div class="listingblock">
<div class="title">sources.list entry for <span class="monospaced">pvetest</span></div>
<div class="content monospaced">
<pre>deb http://download.proxmox.com/debian/pve bookworm pvetest</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">The <span class="monospaced">pvetest</span> repository should (as the name implies) only be used for
testing new features or bug fixes.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="sysadmin_package_repositories_ceph">3.1.5. Ceph Reef Enterprise Repository
 <a class="headerlink" href="#sysadmin_package_repositories_ceph" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>This repository holds the enterprise Proxmox VE Ceph 18.2 Reef packages. They are
suitable for production. Use this repository if you run the Ceph client or a
full Ceph cluster on Proxmox VE.</p></div>
<div class="listingblock">
<div class="title">File <span class="monospaced">/etc/apt/sources.list.d/ceph.list</span></div>
<div class="content monospaced">
<pre>deb https://enterprise.proxmox.com/debian/ceph-reef bookworm enterprise</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_ceph_reef_no_subscription_repository">3.1.6. Ceph Reef No-Subscription Repository
 <a class="headerlink" href="#_ceph_reef_no_subscription_repository" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>This Ceph repository contains the Ceph 18.2 Reef packages before they are moved
to the enterprise repository and after they where on the test repository.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">It’s recommended to use the enterprise repository for production
machines.</td>
</tr></tbody></table>
</div>
<div class="listingblock">
<div class="title">File <span class="monospaced">/etc/apt/sources.list.d/ceph.list</span></div>
<div class="content monospaced">
<pre>deb http://download.proxmox.com/debian/ceph-reef bookworm no-subscription</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_ceph_reef_test_repository">3.1.7. Ceph Reef Test Repository
 <a class="headerlink" href="#_ceph_reef_test_repository" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>This Ceph repository contains the Ceph 18.2 Reef packages before they are moved
to the main repository. It is used to test new Ceph releases on Proxmox VE.</p></div>
<div class="listingblock">
<div class="title">File <span class="monospaced">/etc/apt/sources.list.d/ceph.list</span></div>
<div class="content monospaced">
<pre>deb http://download.proxmox.com/debian/ceph-reef bookworm test</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_ceph_quincy_enterprise_repository">3.1.8. Ceph Quincy Enterprise Repository
 <a class="headerlink" href="#_ceph_quincy_enterprise_repository" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>This repository holds the enterprise Proxmox VE Ceph Quincy packages. They are
suitable for production. Use this repository if you run the Ceph client or a
full Ceph cluster on Proxmox VE.</p></div>
<div class="listingblock">
<div class="title">File <span class="monospaced">/etc/apt/sources.list.d/ceph.list</span></div>
<div class="content monospaced">
<pre>deb https://enterprise.proxmox.com/debian/ceph-quincy bookworm enterprise</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_ceph_quincy_no_subscription_repository">3.1.9. Ceph Quincy No-Subscription Repository
 <a class="headerlink" href="#_ceph_quincy_no_subscription_repository" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>This Ceph repository contains the Ceph Quincy packages before they are moved
to the enterprise repository and after they where on the test repository.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">It’s recommended to use the enterprise repository for production
machines.</td>
</tr></tbody></table>
</div>
<div class="listingblock">
<div class="title">File <span class="monospaced">/etc/apt/sources.list.d/ceph.list</span></div>
<div class="content monospaced">
<pre>deb http://download.proxmox.com/debian/ceph-quincy bookworm no-subscription</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_ceph_quincy_test_repository">3.1.10. Ceph Quincy Test Repository
 <a class="headerlink" href="#_ceph_quincy_test_repository" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>This Ceph repository contains the Ceph Quincy packages before they are moved
to the main repository. It is used to test new Ceph releases on Proxmox VE.</p></div>
<div class="listingblock">
<div class="title">File <span class="monospaced">/etc/apt/sources.list.d/ceph.list</span></div>
<div class="content monospaced">
<pre>deb http://download.proxmox.com/debian/ceph-quincy bookworm test</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_older_ceph_repositories">3.1.11. Older Ceph Repositories
 <a class="headerlink" href="#_older_ceph_repositories" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Proxmox VE 8 doesn’t support Ceph Pacific, Ceph Octopus, or even older releases for
hyper-converged setups. For those releases, you need to first upgrade Ceph to a
newer release before upgrading to Proxmox VE 8.</p></div>
<div class="paragraph">
<p>See the respective
<a href="https://pve.proxmox.com/wiki/Category:Ceph_Upgrade">upgrade guide</a> for details.</p></div>
</div>
<div class="sect3">
<h4 id="sysadmin_debian_firmware_repo">3.1.12. Debian Firmware Repository
 <a class="headerlink" href="#sysadmin_debian_firmware_repo" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Starting with Debian Bookworm (Proxmox VE 8) non-free firmware (as defined by
<a href="https://www.debian.org/social_contract#guidelines">DFSG</a>) has been moved to the
newly created Debian repository component <span class="monospaced">non-free-firmware</span>.</p></div>
<div class="paragraph">
<p>Enable this repository if you want to set up
<a href="#sysadmin_firmware_cpu">Early OS Microcode Updates</a> or need additional
<a href="#sysadmin_firmware_runtime_files">Runtime Firmware Files</a> not already
included in the pre-installed package <span class="monospaced">pve-firmware</span>.</p></div>
<div class="paragraph">
<p>To be able to install packages from this component, run
<span class="monospaced">editor /etc/apt/sources.list</span>, append <span class="monospaced">non-free-firmware</span> to the end of each
<span class="monospaced">.debian.org</span> repository line and run <span class="monospaced">apt update</span>.</p></div>
</div>
<div class="sect3">
<h4 id="repos_secure_apt">3.1.13. SecureApt
 <a class="headerlink" href="#repos_secure_apt" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The <em>Release</em> files in the repositories are signed with GnuPG. APT is using
these signatures to verify that all packages are from a trusted source.</p></div>
<div class="paragraph">
<p>If you install Proxmox VE from an official ISO image, the key for verification is
already installed.</p></div>
<div class="paragraph">
<p>If you install Proxmox VE on top of Debian, download and install
the key with the following commands:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre> # wget https://enterprise.proxmox.com/debian/proxmox-release-bookworm.gpg -O /etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg</pre>
</div></div>
<div class="paragraph">
<p>Verify the checksum afterwards with the <span class="monospaced">sha512sum</span> CLI tool:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># sha512sum /etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg
7da6fe34168adc6e479327ba517796d4702fa2f8b4f0a9833f5ea6e6b48f6507a6da403a274fe201595edc86a84463d50383d07f64bdde2e3658108db7d6dc87 /etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg</pre>
</div></div>
<div class="paragraph">
<p>or the <span class="monospaced">md5sum</span> CLI tool:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># md5sum /etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg
41558dc019ef90bd0f6067644a51cf5b /etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg</pre>
</div></div>
</div>
</div>
<div class="sect2">
<h3 id="system_software_updates">
<span>3.2. System Software Updates</span>
 <a class="headerlink" href="#system_software_updates" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Proxmox provides updates on a regular basis for all repositories. To install
updates use the web-based GUI or the following CLI commands:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># apt-get update
# apt-get dist-upgrade</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">The APT package management system is very flexible and provides many
features, see <span class="monospaced">man apt-get</span>, or <a href="#Hertzog13">[Hertzog13]</a> for additional information.</td>
</tr></tbody></table>
</div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">Regular updates are essential to get the latest patches and security
related fixes. Major system upgrades are announced in the <a href="https://forum.proxmox.com/">Proxmox VE Community Forum</a>.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect2">
<h3 id="chapter_firmware_updates">
<span>3.3. Firmware Updates</span>
 <a class="headerlink" href="#chapter_firmware_updates" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Firmware updates from this chapter should be applied when running Proxmox VE on a
bare-metal server. Whether configuring firmware updates is appropriate within
guests, e.g. when using device pass-through, depends strongly on your setup and
is therefore out of scope.</p></div>
<div class="paragraph">
<p>In addition to regular software updates, firmware updates are also important
for reliable and secure operation.</p></div>
<div class="paragraph">
<p>When obtaining and applying firmware updates, a combination of available options
is recommended to get them as early as possible or at all.</p></div>
<div class="paragraph">
<p>The term firmware is usually divided linguistically into microcode (for CPUs)
and firmware (for other devices).</p></div>
<div class="sect3">
<h4 id="sysadmin_firmware_persistent">3.3.1. Persistent Firmware
 <a class="headerlink" href="#sysadmin_firmware_persistent" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>This section is suitable for all devices. Updated microcode, which is usually
included in a BIOS/UEFI update, is stored on the motherboard, whereas other
firmware is stored on the respective device. This persistent method is
especially important for the CPU, as it enables the earliest possible regular
loading of the updated microcode at boot time.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Caution" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKdUlEQVRoge1Ze1AV1x3+zt279wEi
DWCYGzRVktqa1MEmmtbWR22ncXxUrTDWV/5IqG2wUAUfmUwSkk6V+EAJCEQC6figwZBqZtRxqukf
tebRZBpFG1S0hiRErwiaKK977+6eX//YPXt37+UiIJlMZnpmdnb33PP4vt97z2VEhG9yc3zdAO60
/Z/A192+8QScX8Wifr+fWltbzffU1FT4fD72Vew15ASampqovr4eBw8eNPvGjRuHzMxMmj9//tCT
IKIhu958801yuVwEoNdr48aNNJT7EdHQEdi/fz/JshwTvLiKioqGlMSQLFJfX0+MMRvQsWNSqXLz
H2jez78fRWLTpk1DRuKOF6irqyOn02kDOP8XGdTz+VFSP91Hatu79NRvZ0SR2LJly5CQuKPJtbW1
JEmSDdivHp1AwctHSblYSsqFElLOF5PavIvWZU+LIrF169Y7JjHoibt3744CnzXrIQpeOUbKhVJS
mkpIPV9MyrnNpJwtIuVSNa15/MdRJLZt23ZHJBjRwIu5N954gxYvXgzOudm3aO5E1L5SBNZ1Hoxr
INLAyH6HJxVP/WkXtu9+37ZeSUkJVq9ePagQO+BMXFNTEwV+ybxHsPeVIjg6zwOkgSWNhuOuUUBC
KhxJY0AggDSgpwWbn1mO1csftq1ZUFCA0tLSQZXFA0pkVVVVlJOTA6vWsmY9hF07N0AS4OU48LgR
WLO+FPFxboxMS8G5hlPYmPMI4jwM1P0Jtj67DJyrKHvtNADdjPPz8+FwOCgvL29gmuivrVVUVESF
yqxZD1PoyjFSL+7QHbapmJTml+n9EzVRtl676ZekNKwl5VQ+KSfzKHR2E+UutodYxhiVl5cPyCf6
ZUJlZWWUm5trk/wTi6bgtVdf1M2Gq2DEwUgDNBWhUDBqjfSRwwGoIOEP3c3Y/uxirFw0zibMvLw8
VFZW9tucbkugoqKCVq1aZQO/YslUVJU+D9bVBIDroLgK4hygEDweybYGY8AD6d8yftfASL+j+2O8
9NwiPJk51kYiNzcXO3fu7BeJPgkUFxdTbm6ure/JZdNRub0QrOMcGFcBrgLgAHEAGqD2ID7OZZsz
Ji0RcR6HAV4DkaqPJxXouoSywkz8buF9NhIrV65EVVXVbUnEJHDixAlat26drS9n+U+xo/hZMMNh
yZA+uAbAIKOGMCJlmG3eA/clAYaJEamGBlRdC6QBXZewo3AhVswfbSORk5ODDz/8sE8SvRLw+/1U
XV0dBb5MgOe6LUNIEqpOwgCUlOCESw4HuAfG3KX7B2kAGaTJ0Jro77yI8ucX4DfzRtlIHD58GH6/
PyaJXgk0NjZi79695vuUiffrkr91Vt8cHAwc4FZJGiZEKqDcwsi0ZHP+d749POy8MS7iGtBxARWF
8/CDsQnm3BdeeAHWj6N+EYhspSVbgI5zYYkZGmAwzIEbwA0tUM9NfO+7aeb89HuGRQBWLcSNZxj3
zvOYMzk5NpjBEHi99mXD5lXdjqGDJh42h7CEOXhnK3448X5z/shUb1jSpIEM8yFuzOUcjAsNcnR0
KXdG4MEHH8TEiRPN9z0H3sfRE41m+GNcs9i0LjkGTTcrUoGem5g7/V5z/ohElw4O+jxGYQ3qJqmB
jN/+cbINtcfCJvPYY4+hr3qtVwI+n49lZmaa71fbbuH3hX/FycbPQKSCoIIMQEQcehjVo5Gw8/F3
3cSvH03H2HuHY1icQzcvYWKGIMiITMwQwjtnruPpqk9w/ZZq7p2RkYEJEyaAYrCIWY36/X7KyMhA
W1ub2XfP3Qmo3ZKFaQ+P0gmYIdSoOkU4lWSwxGRAdqHL/xm86k27w0blAw3/PncD6yo/xrsfdZj7
paSkoLGxESNGjAgDZsxWK8X0AZ/Px44fP460tLAzXrnWgUX5r+PYuxf1mG/Gf80sofWy2YuWbg/+
uONtPLHhPRw+0WIHL8hbwG95rcUGPiEhAYcOHUJSUhI456YZRWqiz+8BIqLm5mZMmjQJN27cMPsT
E9zYtWE25k4bYzh3mABIBRwOHPgPx+LH/6yDiXPi6pHZcEpk+I5qmtGpCzdQecCPPUevmevLsowj
R45gypQpcDgc5mVowKaJmBoQTEePHo2GhgaMGxcuum52BLHsqUOoP3oWgLB9Sz7QgpgxJoThw/SS
IutnaXBKZEpcgD998Uvs+3ubDbzT6URNTQ0mTZoEVVWhaRo459A0TeCy4etVA+JH8RvnHK2trZgz
Zw5Onz5tjnPJDrxSOAPLZt1nSVIqwDmIVJw6dx1nLt3A3J/cjZThkhE29cvf3o2KA59ja91lcz3G
GDZv3oylS5fC7XbD6XTC6XRCkiRIktS7Jm5HQLDXNA3Xr19HVlYWPvjgA4vEGLav+RHmTxsJX4rb
8AsO4kbOIG6GzTD4HvzlmB/PVH9q2zc/Px/Z2dnwer1wu92QZRkulwuSJEGWZZMEY8wkEGVCVvCC
AOcciqIgPj4e+/btw9SpU83xqkpYteU9HHm7Bf62rnC2RjikisTFSMPV9m4ceucannvVDn7p0qVY
uHAhgsEggsEgQqFQlAlZnVm0PjOxIKFpGogIqqrC5XKhuroa06dPt4wDcl78F+qPNeNKe6cRoTjs
ZYOKK+3dON7wBQrKm2H5pMbMmTOxZMkSBAIB9PT0oKenB4FAAIqiQFEUU4hRRyq3I2AlIZ4553A6
nSgvL8fs2bNtY9eVNaDub58YmhDZWpe+v70bJ5tu4clt/4WihqU4efJkrFixAkRkAg4Gg1AUBaqq
muDF/pGtX7WQNXeIZ0mSUFRUhAULFtjGPl3ZiJdev4QrbZ16zUQqrrZ342JLN5ZvaEJ3ICz68ePH
IycnB5qmmZIW4EOhUJTZWEKo+Rx1KsEYY9Zk4XA4IEmSKXnOuRkVJElCYWEhvF4v6urqzDVK9n2M
zu4Qnl5+L9q/7MHltgBWbr9kA5+eno7c3FzIsoyI5GrTeCzgosU8VmGMmSGLiEwSsixH2WNBQQEA
2EhUH/wc7330BaaOH479/2zHtS/CFabP58OaNWvg9Xp1EE4nZFmGLMtwu93wer1wuVy2ECpJko2I
iTNWJrZGI+HEwi6FisUVCAQQCASwZ88eVFZW9lk9JicnY/369fD5fPB4PCbo+Ph4k4TH44HL5YLX
64XX6zXH9EKE9aUBRkTEGLNJP1Kd4nI4HMjOzkZ8fDyKi4ttJ3eiJSYmYu3atRg1apQJzuv1wuPx
wOPxwOFwmEBFHnA6nXC5XGYSiywlbns2SnoDANN0hMNZnS0UCpkaOnPmDN566y20tLSY2ktLS8OM
GTOQkpICt9uNuLg4M2EJ8CJZSZJkZmFBwkrAWpH2+3BXEBEkRIhTVdVMOCL5qKqKy5cvQ1EUaJoG
VVWRnKx/Jlpt3eVy2QCKu9C68AFr9o0spwd0Oh1ZYggCIlNaiQktiXdhZgKQABcp3ci7LMtM7B0J
fsAErJoAYItG1mdr+hfvVlACtHDIXkploYXbHvQOikDEe6/ZmjFmAhf9ZvJxOk2Qol88W4kYz32S
GNQfHLG+TyMTkAVEr88xQfVnkBg7GAKixSIykDYQsL21/wFkW/B5QqT9lwAAAABJRU5ErkJggg==">
</td>
<td class="content">With some updates, such as for BIOS/UEFI or storage controller, the
device configuration could be reset. Please follow the vendor’s instructions
carefully and back up the current configuration.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>Please check with your vendor which update methods are available.</p></div>
<div class="ulist"><ul>
<li>
<p>
Convenient update methods for servers can include Dell’s Lifecycle Manager or
Service Packs from HPE.
</p>
</li>
<li>
<p>
Sometimes there are Linux utilities available as well. Examples are
<a href="https://network.nvidia.com/support/firmware/mlxup-mft/"><em>mlxup</em></a> for NVIDIA
ConnectX or
<a href="https://techdocs.broadcom.com/us/en/storage-and-ethernet-connectivity/ethernet-nic-controllers/bcm957xxx/adapters/software-installation/updating-the-firmware/manually-updating-the-adapter-firmware-on-linuxesx.html"><em>bnxtnvm</em>/<em>niccli</em></a>
for Broadcom network cards.
</p>
</li>
<li>
<p>
<a href="https://fwupd.org">LVFS</a> is also an option if there is a cooperation with
the <a href="https://fwupd.org/lvfs/vendors/">hardware vendor</a> and
<a href="https://fwupd.org/lvfs/devices/">supported hardware</a> in use. The technical
requirement for this is that the system was manufactured after 2014 and is
booted via UEFI.
</p>
</li>
</ul></div>
<div class="paragraph">
<p>Proxmox VE ships its own version of the <span class="monospaced">fwupd</span> package to enable Secure Boot
Support with the Proxmox signing key. This package consciously dropped the
dependency recommendation for the <span class="monospaced">udisks2</span> package, due to observed issues with
its use on hypervisors. That means you must explicitly configure the correct
mount point of the EFI partition in <span class="monospaced">/etc/fwupd/daemon.conf</span>, for example:</p></div>
<div class="listingblock">
<div class="title">File <span class="monospaced">/etc/fwupd/daemon.conf</span></div>
<div class="content monospaced">
<pre># Override the location used for the EFI system partition (ESP) path.
EspLocation=/boot/efi</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">If the update instructions require a host reboot, make sure that it can be
done safely. See also <a href="#ha_manager_node_maintenance">Node Maintenance</a>.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="sysadmin_firmware_runtime_files">3.3.2. Runtime Firmware Files
 <a class="headerlink" href="#sysadmin_firmware_runtime_files" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>This method stores firmware on the Proxmox VE operating system and will pass it to a
device if its <a href="#sysadmin_firmware_persistent">persisted firmware</a> is less
recent. It is supported by devices such as network and graphics cards, but not
by those that rely on persisted firmware such as the motherboard and hard disks.</p></div>
<div class="paragraph">
<p>In Proxmox VE the package <span class="monospaced">pve-firmware</span> is already installed by default. Therefore,
with the normal <a href="#system_software_updates">system updates (APT)</a>, included
firmware of common hardware is automatically kept up to date.</p></div>
<div class="paragraph">
<p>An additional <a href="#sysadmin_debian_firmware_repo">Debian Firmware Repository</a>
exists, but is not configured by default.</p></div>
<div class="paragraph">
<p>If you try to install an additional firmware package but it conflicts, APT will
abort the installation. Perhaps the particular firmware can be obtained in
another way.</p></div>
</div>
<div class="sect3">
<h4 id="sysadmin_firmware_cpu">3.3.3. CPU Microcode Updates
 <a class="headerlink" href="#sysadmin_firmware_cpu" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Microcode updates are intended to fix found security vulnerabilities and other
serious CPU bugs. While the CPU performance can be affected, a patched microcode
is usually still more performant than an unpatched microcode where the kernel
itself has to do mitigations. Depending on the CPU type, it is possible that
performance results of the flawed factory state can no longer be achieved
without knowingly running the CPU in an unsafe state.</p></div>
<div class="paragraph">
<p>To get an overview of present CPU vulnerabilities and their mitigations, run
<span class="monospaced">lscpu</span>. Current real-world known vulnerabilities can only show up if the
Proxmox VE host is <a href="#system_software_updates">up to date</a>, its version not
<a href="#faq-support-table">end of life</a>, and has at least been rebooted since the
last kernel update.</p></div>
<div class="paragraph">
<p>Besides the recommended microcode update via
<a href="#sysadmin_firmware_persistent">persistent</a> BIOS/UEFI updates, there is also
an independent method via <strong>Early OS Microcode Updates</strong>. It is convenient to use
and also quite helpful when the motherboard vendor no longer provides BIOS/UEFI
updates. Regardless of the method in use, a reboot is always needed to apply a
microcode update.</p></div>
<div class="sect4">
<h5 id="_set_up_early_os_microcode_updates">Set up Early OS Microcode Updates
 <a class="headerlink" href="#_set_up_early_os_microcode_updates" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>To set up microcode updates that are applied early on boot by the Linux kernel,
you need to:</p></div>
<div class="olist arabic"><ol class="arabic">
<li>
<p>
Enable the <a href="#sysadmin_debian_firmware_repo">Debian Firmware Repository</a>
</p>
</li>
<li>
<p>
Get the latest available packages <span class="monospaced">apt update</span> (or use the web interface,
  under Node → Updates)
</p>
</li>
<li>
<p>
Install the CPU-vendor specific microcode package:
</p>
<div class="ulist"><ul>
<li>
<p>
For Intel CPUs:  <span class="monospaced">apt install intel-microcode</span>
</p>
</li>
<li>
<p>
For AMD CPUs:  <span class="monospaced">apt install amd64-microcode</span>
</p>
</li>
</ul></div>
</li>
<li>
<p>
Reboot the Proxmox VE host
</p>
</li>
</ol></div>
<div class="paragraph">
<p>Any future microcode update will also require a reboot to be loaded.</p></div>
</div>
<div class="sect4">
<h5 id="_microcode_version">Microcode Version
 <a class="headerlink" href="#_microcode_version" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>To get the current running microcode revision for comparison or debugging
purposes:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># grep microcode /proc/cpuinfo | uniq
microcode       : 0xf0</pre>
</div></div>
<div class="paragraph">
<p>A microcode package has updates for many different CPUs. But updates
specifically for your CPU might not come often. So, just looking at the date on
the package won’t tell you when the company actually released an update for your
specific CPU.</p></div>
<div class="paragraph">
<p>If you’ve installed a new microcode package and rebooted your Proxmox VE host, and
this new microcode is newer than both, the version baked into the CPU and the
one from the motherboard’s firmware, you’ll see a message in the system log
saying "microcode updated early".</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># dmesg | grep microcode
[    0.000000] microcode: microcode updated early to revision 0xf0, date = 2021-11-12
[    0.896580] microcode: Microcode Update Driver: v2.2.</pre>
</div></div>
</div>
<div class="sect4">
<h5 id="sysadmin_firmware_troubleshooting">Troubleshooting
 <a class="headerlink" href="#sysadmin_firmware_troubleshooting" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>For debugging purposes, the set up Early OS Microcode Update applied regularly
at system boot can be temporarily disabled as follows:</p></div>
<div class="olist arabic"><ol class="arabic">
<li>
<p>
make sure that the host can be rebooted <a href="#ha_manager_node_maintenance">safely</a>
</p>
</li>
<li>
<p>
reboot the host to get to the GRUB menu (hold <span class="monospaced">SHIFT</span> if it is hidden)
</p>
</li>
<li>
<p>
at the desired Proxmox VE boot entry press <span class="monospaced">E</span>
</p>
</li>
<li>
<p>
go to the line which starts with <span class="monospaced">linux</span> and append separated by a space
<strong><span class="monospaced">dis_ucode_ldr</span></strong>
</p>
</li>
<li>
<p>
press <span class="monospaced">CTRL-X</span> to boot this time without an Early OS Microcode Update
</p>
</li>
</ol></div>
<div class="paragraph">
<p>If a problem related to a recent microcode update is suspected, a package
downgrade should be considered instead of package removal
(<span class="monospaced">apt purge &lt;intel-microcode|amd64-microcode&gt;</span>). Otherwise, a too old
<a href="#sysadmin_firmware_persistent">persisted</a> microcode might be loaded, even
though a more recent one would run without problems.</p></div>
<div class="paragraph">
<p>A downgrade is possible if an earlier microcode package version is
available in the Debian repository, as shown in this example:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># apt list -a intel-microcode
Listing... Done
intel-microcode/stable-security,now 3.20230808.1~deb12u1 amd64 [installed]
intel-microcode/stable 3.20230512.1 amd64</pre>
</div></div>
<div class="listingblock">
<div class="content monospaced">
<pre># apt install intel-microcode=3.202305*
...
Selected version '3.20230512.1' (Debian:12.1/stable [amd64]) for 'intel-microcode'
...
dpkg: warning: downgrading intel-microcode from 3.20230808.1~deb12u1 to 3.20230512.1
...
intel-microcode: microcode will be updated at next boot
...</pre>
</div></div>
<div class="paragraph">
<p>Make sure (again) that the host can be rebooted
<a href="#ha_manager_node_maintenance">safely</a>. To apply an older microcode
potentially included in the microcode package for your CPU type, reboot now.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">
<div class="paragraph">
<p>It makes sense to hold the downgraded package for a while and try more recent
versions again at a later time. Even if the package version is the same in the
future, system updates may have fixed the experienced problem in the meantime.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># apt-mark hold intel-microcode
intel-microcode set on hold.</pre>
</div></div>
<div class="listingblock">
<div class="content monospaced">
<pre># apt-mark unhold intel-microcode
# apt update
# apt upgrade</pre>
</div></div>
</td>
</tr></tbody></table>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="sysadmin_network_configuration">
<span>3.4. Network Configuration</span>
 <a class="headerlink" href="#sysadmin_network_configuration" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Proxmox VE is using the Linux network stack. This provides a lot of flexibility on
how to set up the network on the Proxmox VE nodes. The configuration can be done
either via the GUI, or by manually editing the file <span class="monospaced">/etc/network/interfaces</span>,
which contains the whole network configuration. The  <span class="monospaced">interfaces(5)</span> manual
page contains the complete format description. All Proxmox VE tools try hard to keep
direct user modifications, but using the GUI is still preferable, because it
protects you from errors.</p></div>
<div class="paragraph">
<p>A Linux bridge interface (commonly called <em>vmbrX</em>) is needed to connect guests
to the underlying physical network. It can be thought of as a virtual switch
which the guests and physical interfaces are connected to. This section provides
some examples on how the network can be set up to accomodate different use cases
like redundancy with a <a href="#sysadmin_network_bond"><em>bond</em></a>,
<a href="#sysadmin_network_vlan"><em>vlans</em></a> or
<a href="#sysadmin_network_routed"><em>routed</em></a> and
<a href="#sysadmin_network_masquerading"><em>NAT</em></a> setups.</p></div>
<div class="paragraph">
<p>The <a href="#chapter_pvesdn">Software Defined Network</a> is an option for more complex
virtual networks in Proxmox VE clusters.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">It’s discouraged to use the traditional Debian tools <span class="monospaced">ifup</span> and <span class="monospaced">ifdown</span>
if unsure, as they have some pitfalls like interupting all guest traffic on
<span class="monospaced">ifdown vmbrX</span> but not reconnecting those guest again when doing <span class="monospaced">ifup</span> on the
same bridge later.</td>
</tr></tbody></table>
</div>
<div class="sect3">
<h4 id="_apply_network_changes">3.4.1. Apply Network Changes
 <a class="headerlink" href="#_apply_network_changes" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Proxmox VE does not write changes directly to <span class="monospaced">/etc/network/interfaces</span>. Instead, we
write into a temporary file called <span class="monospaced">/etc/network/interfaces.new</span>, this way you
can do many related changes at once. This also allows to ensure your changes
are correct before applying, as a wrong network configuration may render a node
inaccessible.</p></div>
<div class="sect4">
<h5 id="_live_reload_network_with_ifupdown2">Live-Reload Network with ifupdown2
 <a class="headerlink" href="#_live_reload_network_with_ifupdown2" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>With the recommended <em>ifupdown2</em> package (default for new installations since
Proxmox VE 7.0), it is possible to apply network configuration changes without a
reboot. If you change the network configuration via the GUI, you can click the
<em>Apply Configuration</em> button. This will move changes from the staging
<span class="monospaced">interfaces.new</span> file to <span class="monospaced">/etc/network/interfaces</span> and apply them live.</p></div>
<div class="paragraph">
<p>If you made manual changes directly to the <span class="monospaced">/etc/network/interfaces</span> file, you
can apply them by running <span class="monospaced">ifreload -a</span></p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">If you installed Proxmox VE on top of Debian, or upgraded to Proxmox VE 7.0 from an
older Proxmox VE installation, make sure <em>ifupdown2</em> is installed: <span class="monospaced">apt install
ifupdown2</span></td>
</tr></tbody></table>
</div>
</div>
<div class="sect4">
<h5 id="_reboot_node_to_apply">Reboot Node to Apply
 <a class="headerlink" href="#_reboot_node_to_apply" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Another way to apply a new network configuration is to reboot the node.
In that case the systemd service <span class="monospaced">pvenetcommit</span> will activate the staging
<span class="monospaced">interfaces.new</span> file before the <span class="monospaced">networking</span> service will apply that
configuration.</p></div>
</div>
</div>
<div class="sect3">
<h4 id="_naming_conventions">3.4.2. Naming Conventions
 <a class="headerlink" href="#_naming_conventions" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>We currently use the following naming conventions for device names:</p></div>
<div class="ulist"><ul>
<li>
<p>
Ethernet devices: <span class="monospaced">en*</span>, systemd network interface names. This naming scheme is
 used for new Proxmox VE installations since version 5.0.
</p>
</li>
<li>
<p>
Ethernet devices: <span class="monospaced">eth[N]</span>, where 0 ≤ N (<span class="monospaced">eth0</span>, <span class="monospaced">eth1</span>, …) This naming
scheme is used for Proxmox VE hosts which were installed before the 5.0
release. When upgrading to 5.0, the names are kept as-is.
</p>
</li>
<li>
<p>
Bridge names: Commonly <span class="monospaced">vmbr[N]</span>, where 0 ≤ N ≤ 4094 (<span class="monospaced">vmbr0</span> - <span class="monospaced">vmbr4094</span>),
but you can use any alphanumeric string that starts with a character and is at
most 10 characters long.
</p>
</li>
<li>
<p>
Bonds: <span class="monospaced">bond[N]</span>, where 0 ≤ N (<span class="monospaced">bond0</span>, <span class="monospaced">bond1</span>, …)
</p>
</li>
<li>
<p>
VLANs: Simply add the VLAN number to the device name,
  separated by a period (<span class="monospaced">eno1.50</span>, <span class="monospaced">bond1.30</span>)
</p>
</li>
</ul></div>
<div class="paragraph">
<p>This makes it easier to debug networks problems, because the device
name implies the device type.</p></div>
<div class="sect4">
<h5 id="systemd_network_interface_names">Systemd Network Interface Names
 <a class="headerlink" href="#systemd_network_interface_names" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Systemd defines a versioned naming scheme for network device names. The
scheme uses the two-character prefix <span class="monospaced">en</span> for Ethernet network devices. The
next characters depends on the device driver, device location and other
attributes. Some possible patterns are:</p></div>
<div class="ulist"><ul>
<li>
<p>
<span class="monospaced">o&lt;index&gt;[n&lt;phys_port_name&gt;|d&lt;dev_port&gt;]</span> — devices on board
</p>
</li>
<li>
<p>
<span class="monospaced">s&lt;slot&gt;[f&lt;function&gt;][n&lt;phys_port_name&gt;|d&lt;dev_port&gt;]</span> — devices by hotplug id
</p>
</li>
<li>
<p>
<span class="monospaced">[P&lt;domain&gt;]p&lt;bus&gt;s&lt;slot&gt;[f&lt;function&gt;][n&lt;phys_port_name&gt;|d&lt;dev_port&gt;]</span> —
devices by bus id
</p>
</li>
<li>
<p>
<span class="monospaced">x&lt;MAC&gt;</span> — devices by MAC address
</p>
</li>
</ul></div>
<div class="paragraph">
<p>Some examples for the most common patterns are:</p></div>
<div class="ulist"><ul>
<li>
<p>
<span class="monospaced">eno1</span> — is the first on-board NIC
</p>
</li>
<li>
<p>
<span class="monospaced">enp3s0f1</span> — is function 1 of the NIC on PCI bus 3, slot 0
</p>
</li>
</ul></div>
<div class="paragraph">
<p>For a full list of possible device name patterns, see the
<a href="https://manpages.debian.org/stable/systemd/systemd.net-naming-scheme.7.en.html">
systemd.net-naming-scheme(7) manpage</a>.</p></div>
<div class="paragraph">
<p>A new version of systemd may define a new version of the network device naming
scheme, which it then uses by default. Consequently, updating to a newer
systemd version, for example during a major Proxmox VE upgrade, can change the names
of network devices and require adjusting the network configuration. To avoid
name changes due to a new version of the naming scheme, you can manually pin a
particular naming scheme version (see
<a href="#network_pin_naming_scheme_version">below</a>).</p></div>
<div class="paragraph">
<p>However, even with a pinned naming scheme version, network device names can
still change due to kernel or driver updates. In order to avoid name changes
for a particular network device altogether, you can manually override its name
using a link file (see <a href="#network_override_device_names">below</a>).</p></div>
<div class="paragraph">
<p>For more information on network interface names, see
<a href="https://systemd.io/PREDICTABLE_INTERFACE_NAMES/">Predictable Network Interface
Names</a>.</p></div>
</div>
<div class="sect4">
<h5 id="network_pin_naming_scheme_version">Pinning a specific naming scheme version
 <a class="headerlink" href="#network_pin_naming_scheme_version" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>You can pin a specific version of the naming scheme for network devices by
adding the <span class="monospaced">net.naming-scheme=&lt;version&gt;</span> parameter to the
<a href="#sysboot_edit_kernel_cmdline">kernel command line</a>. For a list of naming
scheme versions, see the
<a href="https://manpages.debian.org/stable/systemd/systemd.net-naming-scheme.7.en.html">
systemd.net-naming-scheme(7) manpage</a>.</p></div>
<div class="paragraph">
<p>For example, to pin the version <span class="monospaced">v252</span>, which is the latest naming scheme
version for a fresh Proxmox VE 8.0 installation, add the following kernel
command-line parameter:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>net.naming-scheme=v252</pre>
</div></div>
<div class="paragraph">
<p>See also <a href="#sysboot_edit_kernel_cmdline">this section</a> on editing the kernel
command line. You need to reboot for the changes to take effect.</p></div>
</div>
<div class="sect4">
<h5 id="network_override_device_names">Overriding network device names
 <a class="headerlink" href="#network_override_device_names" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>You can manually assign a name to a particular network device using a custom
<a href="https://manpages.debian.org/stable/udev/systemd.link.5.en.html">systemd.link
file</a>. This overrides the name that would be assigned according to the latest
network device naming scheme. This way, you can avoid naming changes due to
kernel updates, driver updates or newer versions of the naming scheme.</p></div>
<div class="paragraph">
<p>Custom link files should be placed in <span class="monospaced">/etc/systemd/network/</span> and named
<span class="monospaced">&lt;n&gt;-&lt;id&gt;.link</span>, where <span class="monospaced">n</span> is a priority smaller than <span class="monospaced">99</span> and <span class="monospaced">id</span> is some
identifier. A link file has two sections: <span class="monospaced">[Match]</span> determines which interfaces
the file will apply to; <span class="monospaced">[Link]</span> determines how these interfaces should be
configured, including their naming.</p></div>
<div class="paragraph">
<p>To assign a name to a particular network device, you need a way to uniquely and
permanently identify that device in the <span class="monospaced">[Match]</span> section. One possibility is
to match the device’s MAC address using the <span class="monospaced">MACAddress</span> option, as it is
unlikely to change.</p></div>
<div class="paragraph">
<p>The <span class="monospaced">[Match]</span> section should also contain a <span class="monospaced">Type</span> option to make sure it only
matches the expected physical interface, and not bridge/bond/VLAN interfaces
with the same MAC address. In most setups, <span class="monospaced">Type</span> should be set to <span class="monospaced">ether</span> to
match only Ethernet devices, but some setups may require other choices. See the
<a href="https://manpages.debian.org/stable/udev/systemd.link.5.en.html">systemd.link(5)
manpage</a> for more details.</p></div>
<div class="paragraph">
<p>Then, you can assign a name using the <span class="monospaced">Name</span> option in the <span class="monospaced">[Link]</span> section.</p></div>
<div class="paragraph">
<p>Link files are copied to the <span class="monospaced">initramfs</span>, so it is recommended to refresh the
<span class="monospaced">initramfs</span> after adding, modifying, or removing a link file:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># update-initramfs -u -k all</pre>
</div></div>
<div class="paragraph">
<p>For example, to assign the name <span class="monospaced">enwan0</span> to the Ethernet device with MAC
address <span class="monospaced">aa:bb:cc:dd:ee:ff</span>, create a file
<span class="monospaced">/etc/systemd/network/10-enwan0.link</span> with the following contents:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>[Match]
MACAddress=aa:bb:cc:dd:ee:ff
Type=ether

[Link]
Name=enwan0</pre>
</div></div>
<div class="paragraph">
<p>Do not forget to adjust <span class="monospaced">/etc/network/interfaces</span> to use the new name, and
refresh your <span class="monospaced">initramfs</span> as described above. You need to reboot the node for
the change to take effect.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">It is recommended to assign a name starting with <span class="monospaced">en</span> or <span class="monospaced">eth</span> so that
Proxmox VE recognizes the interface as a physical network device which can then be
configured via the GUI. Also, you should ensure that the name will not clash
with other interface names in the future. One possibility is to assign a name
that does not match any name pattern that systemd uses for network interfaces
(<a href="#systemd_network_interface_names">see above</a>), such as <span class="monospaced">enwan0</span> in the
example above.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>For more information on link files, see the
<a href="https://manpages.debian.org/stable/udev/systemd.link.5.en.html">systemd.link(5)
manpage</a>.</p></div>
</div>
</div>
<div class="sect3">
<h4 id="_choosing_a_network_configuration">3.4.3. Choosing a network configuration
 <a class="headerlink" href="#_choosing_a_network_configuration" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Depending on your current network organization and your resources you can
choose either a bridged, routed, or masquerading networking setup.</p></div>
<div class="sect4">
<h5 id="_proxmox_ve_server_in_a_private_lan_using_an_external_gateway_to_reach_the_internet">Proxmox VE server in a private LAN, using an external gateway to reach the internet
 <a class="headerlink" href="#_proxmox_ve_server_in_a_private_lan_using_an_external_gateway_to_reach_the_internet" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>The <strong>Bridged</strong> model makes the most sense in this case, and this is also
the default mode on new Proxmox VE installations.
Each of your Guest system will have a virtual interface attached to the
Proxmox VE bridge. This is similar in effect to having the Guest network card
directly connected to a new switch on your LAN, the Proxmox VE host playing the role
of the switch.</p></div>
</div>
<div class="sect4">
<h5 id="_proxmox_ve_server_at_hosting_provider_with_public_ip_ranges_for_guests">Proxmox VE server at hosting provider, with public IP ranges for Guests
 <a class="headerlink" href="#_proxmox_ve_server_at_hosting_provider_with_public_ip_ranges_for_guests" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>For this setup, you can use either a <strong>Bridged</strong> or <strong>Routed</strong> model, depending on
what your provider allows.</p></div>
</div>
<div class="sect4">
<h5 id="_proxmox_ve_server_at_hosting_provider_with_a_single_public_ip_address">Proxmox VE server at hosting provider, with a single public IP address
 <a class="headerlink" href="#_proxmox_ve_server_at_hosting_provider_with_a_single_public_ip_address" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>In that case the only way to get outgoing network accesses for your guest
systems is to use <strong>Masquerading</strong>. For incoming network access to your guests,
you will need to configure <strong>Port Forwarding</strong>.</p></div>
<div class="paragraph">
<p>For further flexibility, you can configure
VLANs (IEEE 802.1q) and network bonding, also known as "link
aggregation". That way it is possible to build complex and flexible
virtual networks.</p></div>
</div>
</div>
<div class="sect3">
<h4 id="_default_configuration_using_a_bridge">3.4.4. Default Configuration using a Bridge
 <a class="headerlink" href="#_default_configuration_using_a_bridge" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<a class="image" href="images/default-network-setup-bridge.svg">
<img src="images/default-network-setup-bridge.svg" alt="default-network-setup-bridge.svg" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>Bridges are like physical network switches implemented in software.
All virtual guests can share a single bridge, or you can create multiple
bridges to separate network domains. Each host can have up to 4094 bridges.</p></div>
<div class="paragraph">
<p>The installation program creates a single bridge named <span class="monospaced">vmbr0</span>, which
is connected to the first Ethernet card. The corresponding
configuration in <span class="monospaced">/etc/network/interfaces</span> might look like this:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>auto lo
iface lo inet loopback

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.10.2/24
        gateway 192.168.10.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0</pre>
</div></div>
<div class="paragraph">
<p>Virtual machines behave as if they were directly connected to the
physical network. The network, in turn, sees each virtual machine as
having its own MAC, even though there is only one network cable
connecting all of these VMs to the network.</p></div>
</div>
<div class="sect3">
<h4 id="sysadmin_network_routed">3.4.5. Routed Configuration
 <a class="headerlink" href="#sysadmin_network_routed" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Most hosting providers do not support the above setup. For security
reasons, they disable networking as soon as they detect multiple MAC
addresses on a single interface.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">Some providers allow you to register additional MACs through their
management interface. This avoids the problem, but can be clumsy to
configure because you need to register a MAC for each of your VMs.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>You can avoid the problem by “routing” all traffic via a single
interface. This makes sure that all network packets use the same MAC
address.</p></div>
<div class="paragraph">
<a class="image" href="images/default-network-setup-routed.svg">
<img src="images/default-network-setup-routed.svg" alt="default-network-setup-routed.svg" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>A common scenario is that you have a public IP (assume <span class="monospaced">198.51.100.5</span>
for this example), and an additional IP block for your VMs
(<span class="monospaced">203.0.113.16/28</span>). We recommend the following setup for such
situations:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>auto lo
iface lo inet loopback

auto eno0
iface eno0 inet static
        address  198.51.100.5/29
        gateway  198.51.100.1
        post-up echo 1 &gt; /proc/sys/net/ipv4/ip_forward
        post-up echo 1 &gt; /proc/sys/net/ipv4/conf/eno0/proxy_arp


auto vmbr0
iface vmbr0 inet static
        address  203.0.113.17/28
        bridge-ports none
        bridge-stp off
        bridge-fd 0</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="sysadmin_network_masquerading">3.4.6. Masquerading (NAT) with <span class="monospaced">iptables</span>
 <a class="headerlink" href="#sysadmin_network_masquerading" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Masquerading allows guests having only a private IP address to access the
network by using the host IP address for outgoing traffic. Each outgoing
packet is rewritten by <span class="monospaced">iptables</span> to appear as originating from the host,
and responses are rewritten accordingly to be routed to the original sender.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>auto lo
iface lo inet loopback

auto eno1
#real IP address
iface eno1 inet static
        address  198.51.100.5/24
        gateway  198.51.100.1

auto vmbr0
#private sub network
iface vmbr0 inet static
        address  10.10.10.1/24
        bridge-ports none
        bridge-stp off
        bridge-fd 0

        post-up   echo 1 &gt; /proc/sys/net/ipv4/ip_forward
        post-up   iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">In some masquerade setups with firewall enabled, conntrack zones might be
needed for outgoing connections. Otherwise the firewall could block outgoing
connections since they will prefer the <span class="monospaced">POSTROUTING</span> of the VM bridge (and not
<span class="monospaced">MASQUERADE</span>).</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>Adding these lines in the <span class="monospaced">/etc/network/interfaces</span> can fix this problem:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>post-up   iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1</pre>
</div></div>
<div class="paragraph">
<p>For more information about this, refer to the following links:</p></div>
<div class="paragraph">
<p><a href="https://commons.wikimedia.org/wiki/File:Netfilter-packet-flow.svg">Netfilter Packet Flow</a></p></div>
<div class="paragraph">
<p><a href="https://lwn.net/Articles/370152/">Patch on netdev-list introducing conntrack zones</a></p></div>
<div class="paragraph">
<p><a href="https://web.archive.org/web/20220610151210/https://blog.lobraun.de/2019/05/19/prox/">Blog post with a good explanation by using TRACE in the raw table</a></p></div>
</div>
<div class="sect3">
<h4 id="sysadmin_network_bond">3.4.7. Linux Bond
 <a class="headerlink" href="#sysadmin_network_bond" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Bonding (also called NIC teaming or Link Aggregation) is a technique
for binding multiple NIC’s to a single network device.  It is possible
to achieve different goals, like make the network fault-tolerant,
increase the performance or both together.</p></div>
<div class="paragraph">
<p>High-speed hardware like Fibre Channel and the associated switching
hardware can be quite expensive. By doing link aggregation, two NICs
can appear as one logical interface, resulting in double speed. This
is a native Linux kernel feature that is supported by most
switches. If your nodes have multiple Ethernet ports, you can
distribute your points of failure by running network cables to
different switches and the bonded connection will failover to one
cable or the other in case of network trouble.</p></div>
<div class="paragraph">
<p>Aggregated links can improve live-migration delays and improve the
speed of replication of data between Proxmox VE Cluster nodes.</p></div>
<div class="paragraph">
<p>There are 7 modes for bonding:</p></div>
<div class="ulist"><ul>
<li>
<p>
<strong>Round-robin (balance-rr):</strong> Transmit network packets in sequential
order from the first available network interface (NIC) slave through
the last. This mode provides load balancing and fault tolerance.
</p>
</li>
<li>
<p>
<strong>Active-backup (active-backup):</strong> Only one NIC slave in the bond is
active. A different slave becomes active if, and only if, the active
slave fails. The single logical bonded interface’s MAC address is
externally visible on only one NIC (port) to avoid distortion in the
network switch. This mode provides fault tolerance.
</p>
</li>
<li>
<p>
<strong>XOR (balance-xor):</strong> Transmit network packets based on [(source MAC
address XOR’d with destination MAC address) modulo NIC slave
count]. This selects the same NIC slave for each destination MAC
address. This mode provides load balancing and fault tolerance.
</p>
</li>
<li>
<p>
<strong>Broadcast (broadcast):</strong> Transmit network packets on all slave
network interfaces. This mode provides fault tolerance.
</p>
</li>
<li>
<p>
<strong>IEEE 802.3ad Dynamic link aggregation (802.3ad)(LACP):</strong> Creates
aggregation groups that share the same speed and duplex
settings. Utilizes all slave network interfaces in the active
aggregator group according to the 802.3ad specification.
</p>
</li>
<li>
<p>
<strong>Adaptive transmit load balancing (balance-tlb):</strong> Linux bonding
driver mode that does not require any special network-switch
support. The outgoing network packet traffic is distributed according
to the current load (computed relative to the speed) on each network
interface slave. Incoming traffic is received by one currently
designated slave network interface. If this receiving slave fails,
another slave takes over the MAC address of the failed receiving
slave.
</p>
</li>
<li>
<p>
<strong>Adaptive load balancing (balance-alb):</strong> Includes balance-tlb plus receive
load balancing (rlb) for IPV4 traffic, and does not require any
special network switch support. The receive load balancing is achieved
by ARP negotiation. The bonding driver intercepts the ARP Replies sent
by the local system on their way out and overwrites the source
hardware address with the unique hardware address of one of the NIC
slaves in the single logical bonded interface such that different
network-peers use different MAC addresses for their network packet
traffic.
</p>
</li>
</ul></div>
<div class="paragraph">
<p>If your switch support the LACP (IEEE 802.3ad) protocol then we recommend using
the corresponding bonding mode (802.3ad). Otherwise you should generally use the
active-backup mode.</p></div>
<div class="paragraph">
<p>For the cluster network (Corosync) we recommend configuring it with multiple
networks. Corosync does not need a bond for network reduncancy as it can switch
between networks by itself, if one becomes unusable.</p></div>
<div class="paragraph">
<p>The following bond configuration can be used as distributed/shared
storage network. The benefit would be that you get more speed and the
network will be fault-tolerant.</p></div>
<div class="listingblock">
<div class="title">Example: Use bond with fixed IP address</div>
<div class="content monospaced">
<pre>auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

iface eno3 inet manual

auto bond0
iface bond0 inet static
      bond-slaves eno1 eno2
      address  192.168.1.2/24
      bond-miimon 100
      bond-mode 802.3ad
      bond-xmit-hash-policy layer2+3

auto vmbr0
iface vmbr0 inet static
        address  10.10.10.2/24
        gateway  10.10.10.1
        bridge-ports eno3
        bridge-stp off
        bridge-fd 0</pre>
</div></div>
<div class="paragraph">
<a class="image" href="images/default-network-setup-bond.svg">
<img src="images/default-network-setup-bond.svg" alt="default-network-setup-bond.svg" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>Another possibility it to use the bond directly as bridge port.
This can be used to make the guest network fault-tolerant.</p></div>
<div class="listingblock">
<div class="title">Example: Use a bond as bridge port</div>
<div class="content monospaced">
<pre>auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

auto bond0
iface bond0 inet manual
      bond-slaves eno1 eno2
      bond-miimon 100
      bond-mode 802.3ad
      bond-xmit-hash-policy layer2+3

auto vmbr0
iface vmbr0 inet static
        address  10.10.10.2/24
        gateway  10.10.10.1
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="sysadmin_network_vlan">3.4.8. VLAN 802.1Q
 <a class="headerlink" href="#sysadmin_network_vlan" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>A virtual LAN (VLAN) is a broadcast domain that is partitioned and
isolated in the network at layer two.  So it is possible to have
multiple networks (4096) in a physical network, each independent of
the other ones.</p></div>
<div class="paragraph">
<p>Each VLAN network is identified by a number often called <em>tag</em>.
Network packages are then <em>tagged</em> to identify which virtual network
they belong to.</p></div>
<div class="sect4">
<h5 id="_vlan_for_guest_networks">VLAN for Guest Networks
 <a class="headerlink" href="#_vlan_for_guest_networks" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Proxmox VE supports this setup out of the box. You can specify the VLAN tag
when you create a VM. The VLAN tag is part of the guest network
configuration. The networking layer supports different modes to
implement VLANs, depending on the bridge configuration:</p></div>
<div class="ulist"><ul>
<li>
<p>
<strong>VLAN awareness on the Linux bridge:</strong>
In this case, each guest’s virtual network card is assigned to a VLAN tag,
which is transparently supported by the Linux bridge.
Trunk mode is also possible, but that makes configuration
in the guest necessary.
</p>
</li>
<li>
<p>
<strong>"traditional" VLAN on the Linux bridge:</strong>
In contrast to the VLAN awareness method, this method is not transparent
and creates a VLAN device with associated bridge for each VLAN.
That is, creating a guest on VLAN 5 for example, would create two
interfaces eno1.5 and vmbr0v5, which would remain until a reboot occurs.
</p>
</li>
<li>
<p>
<strong>Open vSwitch VLAN:</strong>
This mode uses the OVS VLAN feature.
</p>
</li>
<li>
<p>
<strong>Guest configured VLAN:</strong>
VLANs are assigned inside the guest. In this case, the setup is
completely done inside the guest and can not be influenced from the
outside. The benefit is that you can use more than one VLAN on a
single virtual NIC.
</p>
</li>
</ul></div>
</div>
<div class="sect4">
<h5 id="_vlan_on_the_host">VLAN on the Host
 <a class="headerlink" href="#_vlan_on_the_host" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>To allow host communication with an isolated network. It is possible
to apply VLAN tags to any network device (NIC, Bond, Bridge). In
general, you should configure the VLAN on the interface with the least
abstraction layers between itself and the physical NIC.</p></div>
<div class="paragraph">
<p>For example, in a default configuration where you want to place
the host management address on a separate VLAN.</p></div>
<div class="listingblock">
<div class="title">Example: Use VLAN 5 for the Proxmox VE management IP with traditional Linux bridge</div>
<div class="content monospaced">
<pre>auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno1.5 inet manual

auto vmbr0v5
iface vmbr0v5 inet static
        address  10.10.10.2/24
        gateway  10.10.10.1
        bridge-ports eno1.5
        bridge-stp off
        bridge-fd 0

auto vmbr0
iface vmbr0 inet manual
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0</pre>
</div></div>
<div class="listingblock">
<div class="title">Example: Use VLAN 5 for the Proxmox VE management IP with VLAN aware Linux bridge</div>
<div class="content monospaced">
<pre>auto lo
iface lo inet loopback

iface eno1 inet manual


auto vmbr0.5
iface vmbr0.5 inet static
        address  10.10.10.2/24
        gateway  10.10.10.1

auto vmbr0
iface vmbr0 inet manual
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094</pre>
</div></div>
<div class="paragraph">
<p>The next example is the same setup but a bond is used to
make this network fail-safe.</p></div>
<div class="listingblock">
<div class="title">Example: Use VLAN 5 with bond0 for the Proxmox VE management IP with traditional Linux bridge</div>
<div class="content monospaced">
<pre>auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

auto bond0
iface bond0 inet manual
      bond-slaves eno1 eno2
      bond-miimon 100
      bond-mode 802.3ad
      bond-xmit-hash-policy layer2+3

iface bond0.5 inet manual

auto vmbr0v5
iface vmbr0v5 inet static
        address  10.10.10.2/24
        gateway  10.10.10.1
        bridge-ports bond0.5
        bridge-stp off
        bridge-fd 0

auto vmbr0
iface vmbr0 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0</pre>
</div></div>
</div>
</div>
<div class="sect3">
<h4 id="_disabling_ipv6_on_the_node">3.4.9. Disabling IPv6 on the Node
 <a class="headerlink" href="#_disabling_ipv6_on_the_node" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Proxmox VE works correctly in all environments, irrespective of whether IPv6 is
deployed or not. We recommend leaving all settings at the provided defaults.</p></div>
<div class="paragraph">
<p>Should you still need to disable support for IPv6 on your node, do so by
creating an appropriate <span class="monospaced">sysctl.conf (5)</span> snippet file and setting the proper
<a href="https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt">sysctls</a>,
for example adding <span class="monospaced">/etc/sysctl.d/disable-ipv6.conf</span> with content:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1</pre>
</div></div>
<div class="paragraph">
<p>This method is preferred to disabling the loading of the IPv6 module on the
<a href="https://www.kernel.org/doc/Documentation/networking/ipv6.rst">kernel commandline</a>.</p></div>
</div>
<div class="sect3">
<h4 id="_disabling_mac_learning_on_a_bridge">3.4.10. Disabling MAC Learning on a Bridge
 <a class="headerlink" href="#_disabling_mac_learning_on_a_bridge" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>By default, MAC learning is enabled on a bridge to ensure a smooth experience
with virtual guests and their networks.</p></div>
<div class="paragraph">
<p>But in some environments this can be undesired. Since Proxmox VE 7.3 you can disable
MAC learning on the bridge by setting the ‘bridge-disable-mac-learning 1`
configuration on a bridge in `/etc/network/interfaces’, for example:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># ...

auto vmbr0
iface vmbr0 inet static
        address  10.10.10.2/24
        gateway  10.10.10.1
        bridge-ports ens18
        bridge-stp off
        bridge-fd 0
        bridge-disable-mac-learning 1</pre>
</div></div>
<div class="paragraph">
<p>Once enabled, Proxmox VE will manually add the configured MAC address from VMs and
Containers to the bridges forwarding database to ensure that guest can still
use the network - but only when they are using their actual MAC address.</p></div>
</div>
</div>
<div class="sect2">
<h3 id="_time_synchronization">
<span>3.5. Time Synchronization</span>
 <a class="headerlink" href="#_time_synchronization" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>The Proxmox VE cluster stack itself relies heavily on the fact that all
the nodes have precisely synchronized time. Some other components,
like Ceph, also won’t work properly if the local time on all nodes is
not in sync.</p></div>
<div class="paragraph">
<p>Time synchronization between nodes can be achieved using the “Network
Time Protocol” (<span class="monospaced">NTP</span>). As of Proxmox VE 7, <span class="monospaced">chrony</span> is used as the default
NTP daemon, while Proxmox VE 6 uses <span class="monospaced">systemd-timesyncd</span>. Both come preconfigured to
use a set of public servers.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Important" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAALa0lEQVRogdWZa2wc1RXHfzM7O/te
P9e1vXHSmEdjx3YeDkkaF6REKRRCEDSEFNkRjdSWSsgC2iqoRWqLQKiqIBg1NOQDiMeHtkQIQkRR
S9S4aWwgCQoUgl232KkT28J21l6vd3d2dx79sDuTXdtre03Uqlc62tl53Pv/n3PuOffcKxiGwf9z
E//XAL5sk65WR0a6AaDrOjMtKwgCoigiCAKCIAhXa9wvTcAEHhoc5KMjR7jw3nucf+MN67k/GCS4
YQOr77yTpl27kGUZm81mXC0ywlLnQDbwEwcO0P2b3yz4jV5RwZq2Nm5/5BH8fj+SJGGz2b4UiSUR
MAzD0DSNEwcO8MdHHln8d0AcsAUCfOe551h7yy04HA7sdvuSrVEwAV3XjdDgIC/v2cOl06dznqWA
JCAAMvn9MwlEgF1PP8037rsPt9uN3W5fkjUKIqDrujF+4QLP3XQTU0NDOc8UIAa4y8sJtrQQqK/H
7XQSfv99Pn/nHQRRxNB16/0EEAUatm9n76uv4vP5cDgcBZNYNIF84I0MEK20lIZvf5sNe/dSWlqK
1+vFxBEZHubT3/+evqNHmR4dJRWP55BevW0bra+8gt/vx+l0IknSokksikA+8ElgClixYwc3799P
ZWUlTqczbz+x0VFOPvEEQ6dPE/niCwxdt0jUbd1K68svU1xcXBCJBQnMBz5eWkrL/v2s++Y3KSsr
QxQXlxc7f/5zBjo7CQ8NoadSKKQn943t7dyyf79FYjHuNC8BwzAMVVV56Z57+OzNN3PAJ8vK+FZH
B2s2b55X6/nahRMnOP7TnxLOKEXJyO2/+hVf37uXoqIinE6nmfzyksirMjNUvvGjH+WAVwG1vJxb
Dhxg1bp1uFwuMwQWJCu3bWPN3r24SkoAcGbA/OXXv2bws8+IRqOkUqlZGX1RBAzDMHRd51+nTvHe
wYNX7gPTQMO+faxav56SkpIlgTdlU3s7TXv2IGUs6AGioRDH2tuZmJggFouhqirGPCzyEUBVVX7X
1pZzPwI0fP/7bNmzh7KysrzADh8+zP3338++ffs4ePBg3vdEUWTDD3/IsuZmRLsdW4bEpd5eOp99
lnA4jKIoaJqWl8QsAqb2Txw4kDNp40BxXR0bWlspLy/PC+qZZ55hdHQUSZIYHByko6ODxx57bBZw
cy3kq6xk6y9/ib+qCgQBmbQ7nT50iIs9PUSjUZLJJIZhzEliLgKMDQzwp5/9zLqnkQ51W3/xCwKB
AE6nMy+B3t5exsfH6ezspLe3l3g8zuHDh2cBz5bKpibWtrXhLi0FwA3owNsPP8zExATxeBxVVecy
QC4Bc+L+taNjluvU33031dddN6/2BUGgpKSE/v5+JicncweaA3i2tDz8MIHrr0eU0gsQBzDW10ff
yZNEIhESicScrpRDQNd1xgYG+OC3v7XuJQBXIMDWn/yEioqKeUGMjY0hCAKKoszS1MjIyLzfAmx/
4gmKli0DQcBJOmicfOopJicnicfjJoG5LWBq/+Szz+a8oABNe/bgdrvndR1BEKiqqiIejxOLxWYR
CAaDC0alqjVrqGxowFNWhkDalUKff07f3/5mWSFTLFksLAK6rjN+4UKO9lXAUV7O2tbWeaNOthZl
WZ4FHlh0aN3y4IPIHk+6L8AGvNfRQTgctuZCthVEU/u6rnPutddyBk0A9bt3W4us+QY2fbyurg6v
15vTzw033LBoAsHmZoLNzdgcDgBcQKi/n391dTE9PU0ymcyxgkmAVCrF+4cO5WjfFgiwfoGwOTOy
lJWVUVtba1mkUAKCILClvR1veTkA9owVPnzllZy8YFnA1H5/dzfTw8PWgziwcutWfD4fsiwvCNyU
xsZG3G43lZWVVl933XVXQQSq161j+ebNlhXcwBdnzxIOh60lhmkFEUDTNP7++uvWgBrp6mrNHBk3
H/BsAtFoFEdm8GAwyMaNGwsiIAgCjbt346uosKwgAp8cPWolNj1THInmsqHn6FGLQApY1thIxcqV
+Hy+WX6+0ETesWMHLpcLWZZ54IEHCgYvCAKrbrsNyem0JrQEXMzkhKzlBaKu61z+97+ZHhnJmbwr
tm2zJu5CWp8pra2tNDU1sXPnTtrb25dEQBAEvrplC57MXJCB8XPniEQixONxa6Uq6brOQHd3jvvo
QP2ttxZUpMxsL7zwwpK+AyxLXrt9Oxe6uxFEEXvGZc4fO0bVD35AMplM1wuapjGYtbugAp5AgKLq
aquu/W/JTEs37dqFqihWzSADk8PD1jzQNC3tQtmrThUINDbm+H4h0tfXx6OPPsodd9xBW1sbPT09
BQPPluu2b7fWRxIQ6usjFotZBCRN06wZbbZAfT1FRUWWKQtpzz//PMPDw/T09DA1NcXo6CjHjx+f
11Xma95AwHJjEdB0HUVRrhCYuRGrAiXBoFUqFtq6u7sZHx9nJBMUurq65uxnsX0Hm5v56A9/ANIJ
LXrpEolEwgqlUqZQyPlo6OOPlzx5ly1bxnBWQmxpackBW6hSBEC02axrpqasNZGmaek8kN0kYPDj
j1FVdUlz4Mknn6Surs4Cb5aUhYZiUy59+GGOgm3V1WiadiUPCILAV9autV5wAaM9PXQeOWJprBBp
aGjgxIkTpFIpOjs7aWxsXBJwUwZOnUKZmgLS2zl4vUjSlV1XURRFqtavv3IDKAVe/O53efPxx69a
SCxUBs+c4dV77+Vyfz+xy5eBdG3iX7ECm81m9S2JokiwuZmiujrCPT1AuqgOAqcef5zzL77Imt27
uemhhyipqVnYZ5cw8bPbR0eOcOall7g8MEB4aIhEJAJk9qOAwKZNyLJsHpQgRKNRIxQK0XPmDH/+
3vdQQ6G8na/YtIk199zDypYWVmzceFWAT1y8yMWzZzn/9tv0vfsudqeTlKIQHRvL2QSOAqU7d/K1
W2/l2muvpba2Nl3iJhIJIxKJMDw8TO/Zs3zw4x+jTUzkHVCw2fCUleHw+QiuXYu/upqqhgb8VVUY
hkFNczPFweCs7z556y3r+vyxYwiiyD+PHwdBwNB1krEYhqYRC4UwdB2D9LJmmnRtXHz77VTfeCPL
ly+ntraWmpqa9MaaqqqGoiiEQiGGhoYYHBzk00OHiHR2LkqDNlnG7nJhd7nQswqN+ZogiuipFMlY
DDWRgBl5KEF6woqAUFFByc03U756NZWVlSxfvpyamhoCgQAejwdB13VDVVVisRihUIiRkRGGh4cZ
+sc/GO3qIt7VhZGJAle7mVpWuVKDCBng8jXX4N68Gc8111BUVER5eTnV1dVUV1cTCASuFFqGYaDr
upFKpYjH40xMTDA+Ps7Y2Bjj4+OEw2EmenqYPneO1OAgZCWpxTY9S9QMcLhyHGUDRI8HqbISZ309
zoYGnE4nHo8Hv99PaWkpgUCAiooK6/DE4XCkI1HW2a6hqiqKojA9Pc3k5KQl4XCY6elpotFoetuk
v59UKIR66RLToRBJTcMYHUWIRvMSMPO6rbY2/SsIOFatQpYk5NWrkSQJWZZxOBy43W68Xi9+v5+i
oiJKSkooLi7G5/PhdruRZdnads85HzD3hpLJJIqiEIvFiEajRCIRIpEI0WiUWCxGPB631iOKopBK
pazUbi4Oc7Y+snKCzWbDZrMhSRKSJGG3262w6HK5cLlceDwevF4vPp8Pr9eL2+3G5XLhcDjMkxuE
TNibdcBhnv9qmkYqlSKZTJJIJFAUhXg8jqIoKIpCIpGwSCSTSVRVtWSuk3qTiAnebrfngHc4HDid
TpxOJy6Xy7qWZdk6wZzrxCbvCY1JRNd1NE3LAWgSM69ngp+PgCiK1gF3NhFTzHvmO+Y3wCzw8xLI
JpL5tYCZpEyw2f/nWt1ag2UtM0x3Mq1iAp1j+bH0M7KFCGWDnfmbd8AMnuxfUzL/C0rp/wFnFd4n
EQn3XQAAAABJRU5ErkJggg==">
</td>
<td class="content">If you upgrade your system to Proxmox VE 7, it is recommended that you
manually install either <span class="monospaced">chrony</span>, <span class="monospaced">ntp</span> or <span class="monospaced">openntpd</span>.</td>
</tr></tbody></table>
</div>
<div class="sect3">
<h4 id="_using_custom_ntp_servers">3.5.1. Using Custom NTP Servers
 <a class="headerlink" href="#_using_custom_ntp_servers" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>In some cases, it might be desired to use non-default NTP
servers. For example, if your Proxmox VE nodes do not have access to the
public internet due to restrictive firewall rules, you
need to set up local NTP servers and tell the NTP daemon to use
them.</p></div>
<div class="sect4">
<h5 id="_for_systems_using_chrony">For systems using chrony:
 <a class="headerlink" href="#_for_systems_using_chrony" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Specify which servers <span class="monospaced">chrony</span> should use in <span class="monospaced">/etc/chrony/chrony.conf</span>:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>server ntp1.example.com iburst
server ntp2.example.com iburst
server ntp3.example.com iburst</pre>
</div></div>
<div class="paragraph">
<p>Restart <span class="monospaced">chrony</span>:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># systemctl restart chronyd</pre>
</div></div>
<div class="paragraph">
<p>Check the journal to confirm that the newly configured NTP servers are being
used:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># journalctl --since -1h -u chrony</pre>
</div></div>
<div class="listingblock">
<div class="content monospaced">
<pre>...
Aug 26 13:00:09 node1 systemd[1]: Started chrony, an NTP client/server.
Aug 26 13:00:15 node1 chronyd[4873]: Selected source 10.0.0.1 (ntp1.example.com)
Aug 26 13:00:15 node1 chronyd[4873]: System clock TAI offset set to 37 seconds
...</pre>
</div></div>
</div>
<div class="sect4">
<h5 id="_for_systems_using_systemd_timesyncd">For systems using systemd-timesyncd:
 <a class="headerlink" href="#_for_systems_using_systemd_timesyncd" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Specify which servers <span class="monospaced">systemd-timesyncd</span> should use in
<span class="monospaced">/etc/systemd/timesyncd.conf</span>:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>[Time]
NTP=ntp1.example.com ntp2.example.com ntp3.example.com ntp4.example.com</pre>
</div></div>
<div class="paragraph">
<p>Then, restart the synchronization service (<span class="monospaced">systemctl restart
systemd-timesyncd</span>), and verify that your newly configured NTP servers are in
use by checking the journal (<span class="monospaced">journalctl --since -1h -u systemd-timesyncd</span>):</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>...
Oct 07 14:58:36 node1 systemd[1]: Stopping Network Time Synchronization...
Oct 07 14:58:36 node1 systemd[1]: Starting Network Time Synchronization...
Oct 07 14:58:36 node1 systemd[1]: Started Network Time Synchronization.
Oct 07 14:58:36 node1 systemd-timesyncd[13514]: Using NTP server 10.0.0.1:123 (ntp1.example.com).
Oct 07 14:58:36 node1 systemd-timesyncd[13514]: interval/delta/delay/jitter/drift 64s/-0.002s/0.020s/0.000s/-31ppm
...</pre>
</div></div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="external_metric_server">
<span>3.6. External Metric Server</span>
 <a class="headerlink" href="#external_metric_server" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-datacenter-metric-server-list.png">
<img src="images/screenshot/gui-datacenter-metric-server-list.png" alt="screenshot/gui-datacenter-metric-server-list.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>In Proxmox VE, you can define external metric servers, which will periodically
receive various stats about your hosts, virtual guests and storages.</p></div>
<div class="paragraph">
<p>Currently supported are:</p></div>
<div class="ulist"><ul>
<li>
<p>
Graphite (see <a href="https://graphiteapp.org">https://graphiteapp.org</a> )
</p>
</li>
<li>
<p>
InfluxDB (see <a href="https://www.influxdata.com/time-series-platform/influxdb/">https://www.influxdata.com/time-series-platform/influxdb/</a> )
</p>
</li>
</ul></div>
<div class="paragraph">
<p>The external metric server definitions are saved in <em>/etc/pve/status.cfg</em>, and
can be edited through the web interface.</p></div>
<div class="sect3">
<h4 id="metric_server_graphite">3.6.1. Graphite server configuration
 <a class="headerlink" href="#metric_server_graphite" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-datacenter-metric-server-graphite.png">
<img src="images/screenshot/gui-datacenter-metric-server-graphite.png" alt="screenshot/gui-datacenter-metric-server-graphite.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>The default port is set to <strong>2003</strong> and the default graphite path is <strong>proxmox</strong>.</p></div>
<div class="paragraph">
<p>By default, Proxmox VE sends the data over UDP, so the graphite server has to be
configured to accept this. Here the maximum transmission unit (MTU) can be
configured for environments not using the standard <strong>1500</strong> MTU.</p></div>
<div class="paragraph">
<p>You can also configure the plugin to use TCP. In order not to block the
important <span class="monospaced">pvestatd</span> statistic collection daemon, a timeout is required to cope
with network problems.</p></div>
</div>
<div class="sect3">
<h4 id="metric_server_influxdb">3.6.2. Influxdb plugin configuration
 <a class="headerlink" href="#metric_server_influxdb" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-datacenter-metric-server-influxdb.png">
<img src="images/screenshot/gui-datacenter-metric-server-influxdb.png" alt="screenshot/gui-datacenter-metric-server-influxdb.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>Proxmox VE sends the data over UDP, so the influxdb server has to be configured for
this. The MTU can also be configured here, if necessary.</p></div>
<div class="paragraph">
<p>Here is an example configuration for influxdb (on your influxdb server):</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>[[udp]]
   enabled = true
   bind-address = "0.0.0.0:8089"
   database = "proxmox"
   batch-size = 1000
   batch-timeout = "1s"</pre>
</div></div>
<div class="paragraph">
<p>With this configuration, your server listens on all IP addresses on port 8089,
and writes the data in the <strong>proxmox</strong> database</p></div>
<div class="paragraph">
<p>Alternatively, the plugin can be configured to use the http(s) API of InfluxDB 2.x.
InfluxDB 1.8.x does contain a forwards compatible API endpoint for this v2 API.</p></div>
<div class="paragraph">
<p>To use it, set <em>influxdbproto</em> to <em>http</em> or <em>https</em> (depending on your configuration).
By default, Proxmox VE uses the organization <em>proxmox</em> and the bucket/db <em>proxmox</em>
(They can be set with the configuration <em>organization</em> and <em>bucket</em> respectively).</p></div>
<div class="paragraph">
<p>Since InfluxDB’s v2 API is only available with authentication, you have
to generate a token that can write into the correct bucket and set it.</p></div>
<div class="paragraph">
<p>In the v2 compatible API of 1.8.x, you can use <em>user:password</em> as token
(if required), and can omit the <em>organization</em> since that has no meaning in InfluxDB 1.x.</p></div>
<div class="paragraph">
<p>You can also set the HTTP Timeout (default is 1s) with the <em>timeout</em> setting,
as well as the maximum batch size (default 25000000 bytes) with the
<em>max-body-size</em> setting (this corresponds to the InfluxDB setting with the
same name).</p></div>
</div>
</div>
<div class="sect2">
<h3 id="_disk_health_monitoring">
<span>3.7. Disk Health Monitoring</span>
 <a class="headerlink" href="#_disk_health_monitoring" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Although a robust and redundant storage is recommended,
it can be very helpful to monitor the health of your local disks.</p></div>
<div class="paragraph">
<p>Starting with Proxmox VE 4.3, the package smartmontools <span class="footnote" data-note="smartmontools homepage <a href=&quot;https://www.smartmontools.org&quot;>https://www.smartmontools.org</a>">[<a id="_footnoteref_1" href="#_footnote_1" title="View footnote" class="footnote">1</a>]</span>
is installed and required. This is a set of tools to monitor and control
the S.M.A.R.T. system for local hard disks.</p></div>
<div class="paragraph">
<p>You can get the status of a disk by issuing the following command:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># smartctl -a /dev/sdX</pre>
</div></div>
<div class="paragraph">
<p>where /dev/sdX is the path to one of your local disks.</p></div>
<div class="paragraph">
<p>If the output says:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>SMART support is: Disabled</pre>
</div></div>
<div class="paragraph">
<p>you can enable it with the command:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># smartctl -s on /dev/sdX</pre>
</div></div>
<div class="paragraph">
<p>For more information on how to use smartctl, please see <span class="monospaced">man smartctl</span>.</p></div>
<div class="paragraph">
<p>By default, smartmontools daemon smartd is active and enabled, and scans
the disks under /dev/sdX and /dev/hdX every 30 minutes for errors and warnings, and sends an
e-mail to root if it detects a problem.</p></div>
<div class="paragraph">
<p>For more information about how to configure smartd, please see <span class="monospaced">man smartd</span> and
<span class="monospaced">man smartd.conf</span>.</p></div>
<div class="paragraph">
<p>If you use your hard disks with a hardware raid controller, there are most likely tools
to monitor the disks in the raid array and the array itself. For more information about this,
please refer to the vendor of your raid controller.</p></div>
</div>
<div class="sect2">
<h3 id="chapter_lvm">
<span>3.8. Logical Volume Manager (LVM)</span>
 <a class="headerlink" href="#chapter_lvm" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Most people install Proxmox VE directly on a local disk. The Proxmox VE
installation CD offers several options for local disk management, and
the current default setup uses LVM. The installer lets you select a
single disk for such setup, and uses that disk as physical volume for
the <strong>V</strong>olume <strong>G</strong>roup (VG) <span class="monospaced">pve</span>. The following output is from a
test installation using a small 8GB disk:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># pvs
  PV         VG   Fmt  Attr PSize PFree
  /dev/sda3  pve  lvm2 a--  7.87g 876.00m

# vgs
  VG   #PV #LV #SN Attr   VSize VFree
  pve    1   3   0 wz--n- 7.87g 876.00m</pre>
</div></div>
<div class="paragraph">
<p>The installer allocates three <strong>L</strong>ogical <strong>V</strong>olumes (LV) inside this
VG:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># lvs
  LV   VG   Attr       LSize   Pool Origin Data%  Meta%
  data pve  twi-a-tz--   4.38g             0.00   0.63
  root pve  -wi-ao----   1.75g
  swap pve  -wi-ao---- 896.00m</pre>
</div></div>
<div class="dlist"><dl>
<dt class="hdlist1">
root
</dt>
<dd>
<p>
Formatted as <span class="monospaced">ext4</span>, and contains the operating system.
</p>
</dd>
<dt class="hdlist1">
swap
</dt>
<dd>
<p>
Swap partition
</p>
</dd>
<dt class="hdlist1">
data
</dt>
<dd>
<p>
This volume uses LVM-thin, and is used to store VM
images. LVM-thin is preferable for this task, because it offers
efficient support for snapshots and clones.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p>For Proxmox VE versions up to 4.1, the installer creates a standard logical
volume called “data”, which is mounted at <span class="monospaced">/var/lib/vz</span>.</p></div>
<div class="paragraph">
<p>Starting from version 4.2, the logical volume “data” is a LVM-thin pool,
used to store block based guest images, and <span class="monospaced">/var/lib/vz</span> is simply a
directory on the root file system.</p></div>
<div class="sect3">
<h4 id="_hardware">3.8.1. Hardware
 <a class="headerlink" href="#_hardware" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>We highly recommend to use a hardware RAID controller (with BBU) for
such setups. This increases performance, provides redundancy, and make
disk replacements easier (hot-pluggable).</p></div>
<div class="paragraph">
<p>LVM itself does not need any special hardware, and memory requirements
are very low.</p></div>
</div>
<div class="sect3">
<h4 id="_bootloader">3.8.2. Bootloader
 <a class="headerlink" href="#_bootloader" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>We install two boot loaders by default. The first partition contains
the standard GRUB boot loader. The second partition is an <strong>E</strong>FI <strong>S</strong>ystem
<strong>P</strong>artition (ESP), which makes it possible to boot on EFI systems and to
apply <a href="#sysadmin_firmware_persistent">persistent firmware updates</a> from the
user space.</p></div>
</div>
<div class="sect3">
<h4 id="_creating_a_volume_group">3.8.3. Creating a Volume Group
 <a class="headerlink" href="#_creating_a_volume_group" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Let’s assume we have an empty disk <span class="monospaced">/dev/sdb</span>, onto which we want to
create a volume group named “vmdata”.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Caution" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKdUlEQVRoge1Ze1AV1x3+zt279wEi
DWCYGzRVktqa1MEmmtbWR22ncXxUrTDWV/5IqG2wUAUfmUwSkk6V+EAJCEQC6figwZBqZtRxqukf
tebRZBpFG1S0hiRErwiaKK977+6eX//YPXt37+UiIJlMZnpmdnb33PP4vt97z2VEhG9yc3zdAO60
/Z/A192+8QScX8Wifr+fWltbzffU1FT4fD72Vew15ASampqovr4eBw8eNPvGjRuHzMxMmj9//tCT
IKIhu958801yuVwEoNdr48aNNJT7EdHQEdi/fz/JshwTvLiKioqGlMSQLFJfX0+MMRvQsWNSqXLz
H2jez78fRWLTpk1DRuKOF6irqyOn02kDOP8XGdTz+VFSP91Hatu79NRvZ0SR2LJly5CQuKPJtbW1
JEmSDdivHp1AwctHSblYSsqFElLOF5PavIvWZU+LIrF169Y7JjHoibt3744CnzXrIQpeOUbKhVJS
mkpIPV9MyrnNpJwtIuVSNa15/MdRJLZt23ZHJBjRwIu5N954gxYvXgzOudm3aO5E1L5SBNZ1Hoxr
INLAyH6HJxVP/WkXtu9+37ZeSUkJVq9ePagQO+BMXFNTEwV+ybxHsPeVIjg6zwOkgSWNhuOuUUBC
KhxJY0AggDSgpwWbn1mO1csftq1ZUFCA0tLSQZXFA0pkVVVVlJOTA6vWsmY9hF07N0AS4OU48LgR
WLO+FPFxboxMS8G5hlPYmPMI4jwM1P0Jtj67DJyrKHvtNADdjPPz8+FwOCgvL29gmuivrVVUVESF
yqxZD1PoyjFSL+7QHbapmJTml+n9EzVRtl676ZekNKwl5VQ+KSfzKHR2E+UutodYxhiVl5cPyCf6
ZUJlZWWUm5trk/wTi6bgtVdf1M2Gq2DEwUgDNBWhUDBqjfSRwwGoIOEP3c3Y/uxirFw0zibMvLw8
VFZW9tucbkugoqKCVq1aZQO/YslUVJU+D9bVBIDroLgK4hygEDweybYGY8AD6d8yftfASL+j+2O8
9NwiPJk51kYiNzcXO3fu7BeJPgkUFxdTbm6ure/JZdNRub0QrOMcGFcBrgLgAHEAGqD2ID7OZZsz
Ji0RcR6HAV4DkaqPJxXouoSywkz8buF9NhIrV65EVVXVbUnEJHDixAlat26drS9n+U+xo/hZMMNh
yZA+uAbAIKOGMCJlmG3eA/clAYaJEamGBlRdC6QBXZewo3AhVswfbSORk5ODDz/8sE8SvRLw+/1U
XV0dBb5MgOe6LUNIEqpOwgCUlOCESw4HuAfG3KX7B2kAGaTJ0Jro77yI8ucX4DfzRtlIHD58GH6/
PyaJXgk0NjZi79695vuUiffrkr91Vt8cHAwc4FZJGiZEKqDcwsi0ZHP+d749POy8MS7iGtBxARWF
8/CDsQnm3BdeeAHWj6N+EYhspSVbgI5zYYkZGmAwzIEbwA0tUM9NfO+7aeb89HuGRQBWLcSNZxj3
zvOYMzk5NpjBEHi99mXD5lXdjqGDJh42h7CEOXhnK3448X5z/shUb1jSpIEM8yFuzOUcjAsNcnR0
KXdG4MEHH8TEiRPN9z0H3sfRE41m+GNcs9i0LjkGTTcrUoGem5g7/V5z/ohElw4O+jxGYQ3qJqmB
jN/+cbINtcfCJvPYY4+hr3qtVwI+n49lZmaa71fbbuH3hX/FycbPQKSCoIIMQEQcehjVo5Gw8/F3
3cSvH03H2HuHY1icQzcvYWKGIMiITMwQwjtnruPpqk9w/ZZq7p2RkYEJEyaAYrCIWY36/X7KyMhA
W1ub2XfP3Qmo3ZKFaQ+P0gmYIdSoOkU4lWSwxGRAdqHL/xm86k27w0blAw3/PncD6yo/xrsfdZj7
paSkoLGxESNGjAgDZsxWK8X0AZ/Px44fP460tLAzXrnWgUX5r+PYuxf1mG/Gf80sofWy2YuWbg/+
uONtPLHhPRw+0WIHL8hbwG95rcUGPiEhAYcOHUJSUhI456YZRWqiz+8BIqLm5mZMmjQJN27cMPsT
E9zYtWE25k4bYzh3mABIBRwOHPgPx+LH/6yDiXPi6pHZcEpk+I5qmtGpCzdQecCPPUevmevLsowj
R45gypQpcDgc5mVowKaJmBoQTEePHo2GhgaMGxcuum52BLHsqUOoP3oWgLB9Sz7QgpgxJoThw/SS
IutnaXBKZEpcgD998Uvs+3ubDbzT6URNTQ0mTZoEVVWhaRo459A0TeCy4etVA+JH8RvnHK2trZgz
Zw5Onz5tjnPJDrxSOAPLZt1nSVIqwDmIVJw6dx1nLt3A3J/cjZThkhE29cvf3o2KA59ja91lcz3G
GDZv3oylS5fC7XbD6XTC6XRCkiRIktS7Jm5HQLDXNA3Xr19HVlYWPvjgA4vEGLav+RHmTxsJX4rb
8AsO4kbOIG6GzTD4HvzlmB/PVH9q2zc/Px/Z2dnwer1wu92QZRkulwuSJEGWZZMEY8wkEGVCVvCC
AOcciqIgPj4e+/btw9SpU83xqkpYteU9HHm7Bf62rnC2RjikisTFSMPV9m4ceucannvVDn7p0qVY
uHAhgsEggsEgQqFQlAlZnVm0PjOxIKFpGogIqqrC5XKhuroa06dPt4wDcl78F+qPNeNKe6cRoTjs
ZYOKK+3dON7wBQrKm2H5pMbMmTOxZMkSBAIB9PT0oKenB4FAAIqiQFEUU4hRRyq3I2AlIZ4553A6
nSgvL8fs2bNtY9eVNaDub58YmhDZWpe+v70bJ5tu4clt/4WihqU4efJkrFixAkRkAg4Gg1AUBaqq
muDF/pGtX7WQNXeIZ0mSUFRUhAULFtjGPl3ZiJdev4QrbZ16zUQqrrZ342JLN5ZvaEJ3ICz68ePH
IycnB5qmmZIW4EOhUJTZWEKo+Rx1KsEYY9Zk4XA4IEmSKXnOuRkVJElCYWEhvF4v6urqzDVK9n2M
zu4Qnl5+L9q/7MHltgBWbr9kA5+eno7c3FzIsoyI5GrTeCzgosU8VmGMmSGLiEwSsixH2WNBQQEA
2EhUH/wc7330BaaOH479/2zHtS/CFabP58OaNWvg9Xp1EE4nZFmGLMtwu93wer1wuVy2ECpJko2I
iTNWJrZGI+HEwi6FisUVCAQQCASwZ88eVFZW9lk9JicnY/369fD5fPB4PCbo+Ph4k4TH44HL5YLX
64XX6zXH9EKE9aUBRkTEGLNJP1Kd4nI4HMjOzkZ8fDyKi4ttJ3eiJSYmYu3atRg1apQJzuv1wuPx
wOPxwOFwmEBFHnA6nXC5XGYSiywlbns2SnoDANN0hMNZnS0UCpkaOnPmDN566y20tLSY2ktLS8OM
GTOQkpICt9uNuLg4M2EJ8CJZSZJkZmFBwkrAWpH2+3BXEBEkRIhTVdVMOCL5qKqKy5cvQ1EUaJoG
VVWRnKx/Jlpt3eVy2QCKu9C68AFr9o0spwd0Oh1ZYggCIlNaiQktiXdhZgKQABcp3ci7LMtM7B0J
fsAErJoAYItG1mdr+hfvVlACtHDIXkploYXbHvQOikDEe6/ZmjFmAhf9ZvJxOk2Qol88W4kYz32S
GNQfHLG+TyMTkAVEr88xQfVnkBg7GAKixSIykDYQsL21/wFkW/B5QqT9lwAAAABJRU5ErkJggg==">
</td>
<td class="content">Please note that the following commands will destroy all
existing data on <span class="monospaced">/dev/sdb</span>.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>First create a partition.</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># sgdisk -N 1 /dev/sdb</pre>
</div></div>
<div class="paragraph">
<p>Create a <strong>P</strong>hysical <strong>V</strong>olume (PV) without confirmation and 250K
metadatasize.</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># pvcreate --metadatasize 250k -y -ff /dev/sdb1</pre>
</div></div>
<div class="paragraph">
<p>Create a volume group named “vmdata” on <span class="monospaced">/dev/sdb1</span></p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># vgcreate vmdata /dev/sdb1</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_creating_an_extra_lv_for_span_class_monospaced_var_lib_vz_span">3.8.4. Creating an extra LV for <span class="monospaced">/var/lib/vz</span>
 <a class="headerlink" href="#_creating_an_extra_lv_for_span_class_monospaced_var_lib_vz_span" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>This can be easily done by creating a new thin LV.</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># lvcreate -n &lt;Name&gt; -V &lt;Size[M,G,T]&gt; &lt;VG&gt;/&lt;LVThin_pool&gt;</pre>
</div></div>
<div class="paragraph">
<p>A real world example:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># lvcreate -n vz -V 10G pve/data</pre>
</div></div>
<div class="paragraph">
<p>Now a filesystem must be created on the LV.</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># mkfs.ext4 /dev/pve/vz</pre>
</div></div>
<div class="paragraph">
<p>At last this has to be mounted.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">be sure that <span class="monospaced">/var/lib/vz</span> is empty. On a default
installation it’s not.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>To make it always accessible add the following line in <span class="monospaced">/etc/fstab</span>.</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># echo '/dev/pve/vz /var/lib/vz ext4 defaults 0 2' &gt;&gt; /etc/fstab</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_resizing_the_thin_pool">3.8.5. Resizing the thin pool
 <a class="headerlink" href="#_resizing_the_thin_pool" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Resize the LV and the metadata pool with the following command:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># lvresize --size +&lt;size[\M,G,T]&gt; --poolmetadatasize +&lt;size[\M,G]&gt; &lt;VG&gt;/&lt;LVThin_pool&gt;</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">When extending the data pool, the metadata pool must also be
extended.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="_create_a_lvm_thin_pool">3.8.6. Create a LVM-thin pool
 <a class="headerlink" href="#_create_a_lvm_thin_pool" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>A thin pool has to be created on top of a volume group.
How to create a volume group see Section LVM.</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># lvcreate -L 80G -T -n vmstore vmdata</pre>
</div></div>
</div>
</div>
<div class="sect2">
<h3 id="chapter_zfs">
<span>3.9. ZFS on Linux</span>
 <a class="headerlink" href="#chapter_zfs" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>ZFS is a combined file system and logical volume manager designed by
Sun Microsystems. Starting with Proxmox VE 3.4, the native Linux
kernel port of the ZFS file system is introduced as optional
file system and also as an additional selection for the root
file system. There is no need for manually compile ZFS modules - all
packages are included.</p></div>
<div class="paragraph">
<p>By using ZFS, its possible to achieve maximum enterprise features with
low budget hardware, but also high performance systems by leveraging
SSD caching or even SSD only setups. ZFS can replace cost intense
hardware raid cards by moderate CPU and memory load combined with easy
management.</p></div>
<div class="ulist"><div class="title">General ZFS advantages</div><ul>
<li>
<p>
Easy configuration and management with Proxmox VE GUI and CLI.
</p>
</li>
<li>
<p>
Reliable
</p>
</li>
<li>
<p>
Protection against data corruption
</p>
</li>
<li>
<p>
Data compression on file system level
</p>
</li>
<li>
<p>
Snapshots
</p>
</li>
<li>
<p>
Copy-on-write clone
</p>
</li>
<li>
<p>
Various raid levels: RAID0, RAID1, RAID10, RAIDZ-1, RAIDZ-2, RAIDZ-3,
dRAID, dRAID2, dRAID3
</p>
</li>
<li>
<p>
Can use SSD for cache
</p>
</li>
<li>
<p>
Self healing
</p>
</li>
<li>
<p>
Continuous integrity checking
</p>
</li>
<li>
<p>
Designed for high storage capacities
</p>
</li>
<li>
<p>
Asynchronous replication over network
</p>
</li>
<li>
<p>
Open Source
</p>
</li>
<li>
<p>
Encryption
</p>
</li>
<li>
<p>
…
</p>
</li>
</ul></div>
<div class="sect3">
<h4 id="_hardware_2">3.9.1. Hardware
 <a class="headerlink" href="#_hardware_2" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>ZFS depends heavily on memory, so you need at least 8GB to start. In
practice, use as much as you can get for your hardware/budget. To prevent
data corruption, we recommend the use of high quality ECC RAM.</p></div>
<div class="paragraph">
<p>If you use a dedicated cache and/or log disk, you should use an
enterprise class SSD. This can
increase the overall performance significantly.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Important" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAALa0lEQVRogdWZa2wc1RXHfzM7O/te
P9e1vXHSmEdjx3YeDkkaF6REKRRCEDSEFNkRjdSWSsgC2iqoRWqLQKiqIBg1NOQDiMeHtkQIQkRR
S9S4aWwgCQoUgl232KkT28J21l6vd3d2dx79sDuTXdtre03Uqlc62tl53Pv/n3PuOffcKxiGwf9z
E//XAL5sk65WR0a6AaDrOjMtKwgCoigiCAKCIAhXa9wvTcAEHhoc5KMjR7jw3nucf+MN67k/GCS4
YQOr77yTpl27kGUZm81mXC0ywlLnQDbwEwcO0P2b3yz4jV5RwZq2Nm5/5BH8fj+SJGGz2b4UiSUR
MAzD0DSNEwcO8MdHHln8d0AcsAUCfOe551h7yy04HA7sdvuSrVEwAV3XjdDgIC/v2cOl06dznqWA
JCAAMvn9MwlEgF1PP8037rsPt9uN3W5fkjUKIqDrujF+4QLP3XQTU0NDOc8UIAa4y8sJtrQQqK/H
7XQSfv99Pn/nHQRRxNB16/0EEAUatm9n76uv4vP5cDgcBZNYNIF84I0MEK20lIZvf5sNe/dSWlqK
1+vFxBEZHubT3/+evqNHmR4dJRWP55BevW0bra+8gt/vx+l0IknSokksikA+8ElgClixYwc3799P
ZWUlTqczbz+x0VFOPvEEQ6dPE/niCwxdt0jUbd1K68svU1xcXBCJBQnMBz5eWkrL/v2s++Y3KSsr
QxQXlxc7f/5zBjo7CQ8NoadSKKQn943t7dyyf79FYjHuNC8BwzAMVVV56Z57+OzNN3PAJ8vK+FZH
B2s2b55X6/nahRMnOP7TnxLOKEXJyO2/+hVf37uXoqIinE6nmfzyksirMjNUvvGjH+WAVwG1vJxb
Dhxg1bp1uFwuMwQWJCu3bWPN3r24SkoAcGbA/OXXv2bws8+IRqOkUqlZGX1RBAzDMHRd51+nTvHe
wYNX7gPTQMO+faxav56SkpIlgTdlU3s7TXv2IGUs6AGioRDH2tuZmJggFouhqirGPCzyEUBVVX7X
1pZzPwI0fP/7bNmzh7KysrzADh8+zP3338++ffs4ePBg3vdEUWTDD3/IsuZmRLsdW4bEpd5eOp99
lnA4jKIoaJqWl8QsAqb2Txw4kDNp40BxXR0bWlspLy/PC+qZZ55hdHQUSZIYHByko6ODxx57bBZw
cy3kq6xk6y9/ib+qCgQBmbQ7nT50iIs9PUSjUZLJJIZhzEliLgKMDQzwp5/9zLqnkQ51W3/xCwKB
AE6nMy+B3t5exsfH6ezspLe3l3g8zuHDh2cBz5bKpibWtrXhLi0FwA3owNsPP8zExATxeBxVVecy
QC4Bc+L+taNjluvU33031dddN6/2BUGgpKSE/v5+JicncweaA3i2tDz8MIHrr0eU0gsQBzDW10ff
yZNEIhESicScrpRDQNd1xgYG+OC3v7XuJQBXIMDWn/yEioqKeUGMjY0hCAKKoszS1MjIyLzfAmx/
4gmKli0DQcBJOmicfOopJicnicfjJoG5LWBq/+Szz+a8oABNe/bgdrvndR1BEKiqqiIejxOLxWYR
CAaDC0alqjVrqGxowFNWhkDalUKff07f3/5mWSFTLFksLAK6rjN+4UKO9lXAUV7O2tbWeaNOthZl
WZ4FHlh0aN3y4IPIHk+6L8AGvNfRQTgctuZCthVEU/u6rnPutddyBk0A9bt3W4us+QY2fbyurg6v
15vTzw033LBoAsHmZoLNzdgcDgBcQKi/n391dTE9PU0ymcyxgkmAVCrF+4cO5WjfFgiwfoGwOTOy
lJWVUVtba1mkUAKCILClvR1veTkA9owVPnzllZy8YFnA1H5/dzfTw8PWgziwcutWfD4fsiwvCNyU
xsZG3G43lZWVVl933XVXQQSq161j+ebNlhXcwBdnzxIOh60lhmkFEUDTNP7++uvWgBrp6mrNHBk3
H/BsAtFoFEdm8GAwyMaNGwsiIAgCjbt346uosKwgAp8cPWolNj1THInmsqHn6FGLQApY1thIxcqV
+Hy+WX6+0ETesWMHLpcLWZZ54IEHCgYvCAKrbrsNyem0JrQEXMzkhKzlBaKu61z+97+ZHhnJmbwr
tm2zJu5CWp8pra2tNDU1sXPnTtrb25dEQBAEvrplC57MXJCB8XPniEQixONxa6Uq6brOQHd3jvvo
QP2ttxZUpMxsL7zwwpK+AyxLXrt9Oxe6uxFEEXvGZc4fO0bVD35AMplM1wuapjGYtbugAp5AgKLq
aquu/W/JTEs37dqFqihWzSADk8PD1jzQNC3tQtmrThUINDbm+H4h0tfXx6OPPsodd9xBW1sbPT09
BQPPluu2b7fWRxIQ6usjFotZBCRN06wZbbZAfT1FRUWWKQtpzz//PMPDw/T09DA1NcXo6CjHjx+f
11Xma95AwHJjEdB0HUVRrhCYuRGrAiXBoFUqFtq6u7sZHx9nJBMUurq65uxnsX0Hm5v56A9/ANIJ
LXrpEolEwgqlUqZQyPlo6OOPlzx5ly1bxnBWQmxpackBW6hSBEC02axrpqasNZGmaek8kN0kYPDj
j1FVdUlz4Mknn6Surs4Cb5aUhYZiUy59+GGOgm3V1WiadiUPCILAV9autV5wAaM9PXQeOWJprBBp
aGjgxIkTpFIpOjs7aWxsXBJwUwZOnUKZmgLS2zl4vUjSlV1XURRFqtavv3IDKAVe/O53efPxx69a
SCxUBs+c4dV77+Vyfz+xy5eBdG3iX7ECm81m9S2JokiwuZmiujrCPT1AuqgOAqcef5zzL77Imt27
uemhhyipqVnYZ5cw8bPbR0eOcOall7g8MEB4aIhEJAJk9qOAwKZNyLJsHpQgRKNRIxQK0XPmDH/+
3vdQQ6G8na/YtIk199zDypYWVmzceFWAT1y8yMWzZzn/9tv0vfsudqeTlKIQHRvL2QSOAqU7d/K1
W2/l2muvpba2Nl3iJhIJIxKJMDw8TO/Zs3zw4x+jTUzkHVCw2fCUleHw+QiuXYu/upqqhgb8VVUY
hkFNczPFweCs7z556y3r+vyxYwiiyD+PHwdBwNB1krEYhqYRC4UwdB2D9LJmmnRtXHz77VTfeCPL
ly+ntraWmpqa9MaaqqqGoiiEQiGGhoYYHBzk00OHiHR2LkqDNlnG7nJhd7nQswqN+ZogiuipFMlY
DDWRgBl5KEF6woqAUFFByc03U756NZWVlSxfvpyamhoCgQAejwdB13VDVVVisRihUIiRkRGGh4cZ
+sc/GO3qIt7VhZGJAle7mVpWuVKDCBng8jXX4N68Gc8111BUVER5eTnV1dVUV1cTCASuFFqGYaDr
upFKpYjH40xMTDA+Ps7Y2Bjj4+OEw2EmenqYPneO1OAgZCWpxTY9S9QMcLhyHGUDRI8HqbISZ309
zoYGnE4nHo8Hv99PaWkpgUCAiooK6/DE4XCkI1HW2a6hqiqKojA9Pc3k5KQl4XCY6elpotFoetuk
v59UKIR66RLToRBJTcMYHUWIRvMSMPO6rbY2/SsIOFatQpYk5NWrkSQJWZZxOBy43W68Xi9+v5+i
oiJKSkooLi7G5/PhdruRZdnads85HzD3hpLJJIqiEIvFiEajRCIRIpEI0WiUWCxGPB631iOKopBK
pazUbi4Oc7Y+snKCzWbDZrMhSRKSJGG3262w6HK5cLlceDwevF4vPp8Pr9eL2+3G5XLhcDjMkxuE
TNibdcBhnv9qmkYqlSKZTJJIJFAUhXg8jqIoKIpCIpGwSCSTSVRVtWSuk3qTiAnebrfngHc4HDid
TpxOJy6Xy7qWZdk6wZzrxCbvCY1JRNd1NE3LAWgSM69ngp+PgCiK1gF3NhFTzHvmO+Y3wCzw8xLI
JpL5tYCZpEyw2f/nWt1ag2UtM0x3Mq1iAp1j+bH0M7KFCGWDnfmbd8AMnuxfUzL/C0rp/wFnFd4n
EQn3XQAAAABJRU5ErkJggg==">
</td>
<td class="content">Do not use ZFS on top of a hardware RAID controller which has its
own cache management. ZFS needs to communicate directly with the disks. An
HBA adapter or something like an LSI controller flashed in “IT” mode is more
appropriate.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>If you are experimenting with an installation of Proxmox VE inside a VM
(Nested Virtualization), don’t use <span class="monospaced">virtio</span> for disks of that VM,
as they are not supported by ZFS. Use IDE or SCSI instead (also works
with the <span class="monospaced">virtio</span> SCSI controller type).</p></div>
</div>
<div class="sect3">
<h4 id="_installation_as_root_file_system">3.9.2. Installation as Root File System
 <a class="headerlink" href="#_installation_as_root_file_system" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>When you install using the Proxmox VE installer, you can choose ZFS for the
root file system. You need to select the RAID type at installation
time:</p></div>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
RAID0
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Also called “striping”. The capacity of such volume is the sum
of the capacities of all disks. But RAID0 does not add any redundancy,
so the failure of a single drive makes the volume unusable.
</p>
</td>
</tr>
<tr>
<td class="hdlist1">
RAID1
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Also called “mirroring”. Data is written identically to all
disks. This mode requires at least 2 disks with the same size. The
resulting capacity is that of a single disk.
</p>
</td>
</tr>
<tr>
<td class="hdlist1">
RAID10
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
A combination of RAID0 and RAID1. Requires at least 4 disks.
</p>
</td>
</tr>
<tr>
<td class="hdlist1">
RAIDZ-1
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
A variation on RAID-5, single parity. Requires at least 3 disks.
</p>
</td>
</tr>
<tr>
<td class="hdlist1">
RAIDZ-2
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
A variation on RAID-5, double parity. Requires at least 4 disks.
</p>
</td>
</tr>
<tr>
<td class="hdlist1">
RAIDZ-3
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
A variation on RAID-5, triple parity. Requires at least 5 disks.
</p>
</td>
</tr>
</tbody></table></div>
<div class="paragraph">
<p>The installer automatically partitions the disks, creates a ZFS pool
called <span class="monospaced">rpool</span>, and installs the root file system on the ZFS subvolume
<span class="monospaced">rpool/ROOT/pve-1</span>.</p></div>
<div class="paragraph">
<p>Another subvolume called <span class="monospaced">rpool/data</span> is created to store VM
images. In order to use that with the Proxmox VE tools, the installer
creates the following configuration entry in <span class="monospaced">/etc/pve/storage.cfg</span>:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>zfspool: local-zfs
        pool rpool/data
        sparse
        content images,rootdir</pre>
</div></div>
<div class="paragraph">
<p>After installation, you can view your ZFS pool status using the
<span class="monospaced">zpool</span> command:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># zpool status
  pool: rpool
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sda2    ONLINE       0     0     0
            sdb2    ONLINE       0     0     0
          mirror-1  ONLINE       0     0     0
            sdc     ONLINE       0     0     0
            sdd     ONLINE       0     0     0

errors: No known data errors</pre>
</div></div>
<div class="paragraph">
<p>The <span class="monospaced">zfs</span> command is used to configure and manage your ZFS file systems. The
following command lists all file systems after installation:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># zfs list
NAME               USED  AVAIL  REFER  MOUNTPOINT
rpool             4.94G  7.68T    96K  /rpool
rpool/ROOT         702M  7.68T    96K  /rpool/ROOT
rpool/ROOT/pve-1   702M  7.68T   702M  /
rpool/data          96K  7.68T    96K  /rpool/data
rpool/swap        4.25G  7.69T    64K  -</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="sysadmin_zfs_raid_considerations">3.9.3. ZFS RAID Level Considerations
 <a class="headerlink" href="#sysadmin_zfs_raid_considerations" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>There are a few factors to take into consideration when choosing the layout of
a ZFS pool. The basic building block of a ZFS pool is the virtual device, or
<span class="monospaced">vdev</span>. All vdevs in a pool are used equally and the data is striped among them
(RAID0). Check the <span class="monospaced">zpoolconcepts(7)</span> manpage for more details on vdevs.</p></div>
<div class="sect4">
<h5 id="sysadmin_zfs_raid_performance">Performance
 <a class="headerlink" href="#sysadmin_zfs_raid_performance" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Each <span class="monospaced">vdev</span> type has different performance behaviors. The two
parameters of interest are the IOPS (Input/Output Operations per Second) and
the bandwidth with which data can be written or read.</p></div>
<div class="paragraph">
<p>A <em>mirror</em> vdev (RAID1) will approximately behave like a single disk in regard
to both parameters when writing data. When reading data the performance will
scale linearly with the number of disks in the mirror.</p></div>
<div class="paragraph">
<p>A common situation is to have 4 disks. When setting it up as 2 mirror vdevs
(RAID10) the pool will have the write characteristics as two single disks in
regard to IOPS and bandwidth. For read operations it will resemble 4 single
disks.</p></div>
<div class="paragraph">
<p>A <em>RAIDZ</em> of any redundancy level will approximately behave like a single disk
in regard to IOPS with a lot of bandwidth. How much bandwidth depends on the
size of the RAIDZ vdev and the redundancy level.</p></div>
<div class="paragraph">
<p>A <em>dRAID</em> pool should match the performance of an equivalent <em>RAIDZ</em> pool.</p></div>
<div class="paragraph">
<p>For running VMs, IOPS is the more important metric in most situations.</p></div>
</div>
<div class="sect4">
<h5 id="sysadmin_zfs_raid_size_space_usage_redundancy">Size, Space usage and Redundancy
 <a class="headerlink" href="#sysadmin_zfs_raid_size_space_usage_redundancy" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>While a pool made of <em>mirror</em> vdevs will have the best performance
characteristics, the usable space will be 50% of the disks available. Less if a
mirror vdev consists of more than 2 disks, for example in a 3-way mirror. At
least one healthy disk per mirror is needed for the pool to stay functional.</p></div>
<div class="paragraph">
<p>The usable space of a <em>RAIDZ</em> type vdev of N disks is roughly N-P, with P being
the RAIDZ-level. The RAIDZ-level indicates how many arbitrary disks can fail
without losing data. A special case is a 4 disk pool with RAIDZ2. In this
situation it is usually better to use 2 mirror vdevs for the better performance
as the usable space will be the same.</p></div>
<div class="paragraph">
<p>Another important factor when using any RAIDZ level is how ZVOL datasets, which
are used for VM disks, behave. For each data block the pool needs parity data
which is at least the size of the minimum block size defined by the <span class="monospaced">ashift</span>
value of the pool. With an ashift of 12 the block size of the pool is 4k.  The
default block size for a ZVOL is 8k. Therefore, in a RAIDZ2 each 8k block
written will cause two additional 4k parity blocks to be written,
8k + 4k + 4k = 16k.  This is of course a simplified approach and the real
situation will be slightly different with metadata, compression and such not
being accounted for in this example.</p></div>
<div class="paragraph">
<p>This behavior can be observed when checking the following properties of the
ZVOL:</p></div>
<div class="ulist"><ul>
<li>
<p>
<span class="monospaced">volsize</span>
</p>
</li>
<li>
<p>
<span class="monospaced">refreservation</span> (if the pool is not thin provisioned)
</p>
</li>
<li>
<p>
<span class="monospaced">used</span> (if the pool is thin provisioned and without snapshots present)
</p>
</li>
</ul></div>
<div class="listingblock">
<div class="content monospaced">
<pre># zfs get volsize,refreservation,used &lt;pool&gt;/vm-&lt;vmid&gt;-disk-X</pre>
</div></div>
<div class="paragraph">
<p><span class="monospaced">volsize</span> is the size of the disk as it is presented to the VM, while
<span class="monospaced">refreservation</span> shows the reserved space on the pool which includes the
expected space needed for the parity data. If the pool is thin provisioned, the
<span class="monospaced">refreservation</span> will be set to 0. Another way to observe the behavior is to
compare the used disk space within the VM and the <span class="monospaced">used</span> property. Be aware
that snapshots will skew the value.</p></div>
<div class="paragraph">
<p>There are a few options to counter the increased use of space:</p></div>
<div class="ulist"><ul>
<li>
<p>
Increase the <span class="monospaced">volblocksize</span> to improve the data to parity ratio
</p>
</li>
<li>
<p>
Use <em>mirror</em> vdevs instead of <em>RAIDZ</em>
</p>
</li>
<li>
<p>
Use <span class="monospaced">ashift=9</span> (block size of 512 bytes)
</p>
</li>
</ul></div>
<div class="paragraph">
<p>The <span class="monospaced">volblocksize</span> property can only be set when creating a ZVOL. The default
value can be changed in the storage configuration. When doing this, the guest
needs to be tuned accordingly and depending on the use case, the problem of
write amplification is just moved from the ZFS layer up to the guest.</p></div>
<div class="paragraph">
<p>Using <span class="monospaced">ashift=9</span> when creating the pool can lead to bad
performance, depending on the disks underneath, and cannot be changed later on.</p></div>
<div class="paragraph">
<p>Mirror vdevs (RAID1, RAID10) have favorable behavior for VM workloads. Use
them, unless your environment has specific needs and characteristics where
RAIDZ performance characteristics are acceptable.</p></div>
</div>
</div>
<div class="sect3">
<h4 id="_zfs_draid">3.9.4. ZFS dRAID
 <a class="headerlink" href="#_zfs_draid" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>In a ZFS dRAID (declustered RAID) the hot spare drive(s) participate in the RAID.
Their spare capacity is reserved and used for rebuilding when one drive fails.
This provides, depending on the configuration, faster rebuilding compared to a
RAIDZ in case of drive failure. More information can be found in the official
OpenZFS documentation. <span class="footnote" data-note="OpenZFS dRAID
<a href=&quot;https://openzfs.github.io/openzfs-docs/Basic%20Concepts/dRAID%20Howto.html&quot;>https://openzfs.github.io/openzfs-docs/Basic%20Concepts/dRAID%20Howto.html</a>">[<a id="_footnoteref_2" href="#_footnote_2" title="View footnote" class="footnote">2</a>]</span></p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">dRAID is intended for more than 10-15 disks in a dRAID. A RAIDZ
setup should be better for a lower amount of disks in most use cases.</td>
</tr></tbody></table>
</div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">The GUI requires one more disk than the minimum (i.e. dRAID1 needs 3). It
expects that a spare disk is added as well.</td>
</tr></tbody></table>
</div>
<div class="ulist"><ul>
<li>
<p>
<span class="monospaced">dRAID1</span> or <span class="monospaced">dRAID</span>: requires at least 2 disks, one can fail before data is
lost
</p>
</li>
<li>
<p>
<span class="monospaced">dRAID2</span>: requires at least 3 disks, two can fail before data is lost
</p>
</li>
<li>
<p>
<span class="monospaced">dRAID3</span>: requires at least 4 disks, three can fail before data is lost
</p>
</li>
</ul></div>
<div class="paragraph">
<p>Additional information can be found on the manual page:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># man zpoolconcepts</pre>
</div></div>
<div class="sect4">
<h5 id="_spares_and_data">Spares and Data
 <a class="headerlink" href="#_spares_and_data" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>The number of <span class="monospaced">spares</span> tells the system how many disks it should keep ready in
case of a disk failure. The default value is 0 <span class="monospaced">spares</span>. Without spares,
rebuilding won’t get any speed benefits.</p></div>
<div class="paragraph">
<p><span class="monospaced">data</span> defines the number of devices in a redundancy group. The default value is
8. Except when <span class="monospaced">disks - parity - spares</span> equal something less than 8, the lower
number is used. In general, a smaller number of <span class="monospaced">data</span> devices leads to higher
IOPS, better compression ratios and faster resilvering, but defining fewer data
devices reduces the available storage capacity of the pool.</p></div>
</div>
</div>
<div class="sect3">
<h4 id="_bootloader_2">3.9.5. Bootloader
 <a class="headerlink" href="#_bootloader_2" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Proxmox VE uses <a href="#sysboot_proxmox_boot_tool"><span class="monospaced">proxmox-boot-tool</span></a> to manage the
bootloader configuration.
See the chapter on <a href="#sysboot">Proxmox VE host bootloaders</a> for details.</p></div>
</div>
<div class="sect3">
<h4 id="_zfs_administration">3.9.6. ZFS Administration
 <a class="headerlink" href="#_zfs_administration" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>This section gives you some usage examples for common tasks. ZFS
itself is really powerful and provides many options. The main commands
to manage ZFS are <span class="monospaced">zfs</span> and <span class="monospaced">zpool</span>. Both commands come with great
manual pages, which can be read with:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># man zpool
# man zfs</pre>
</div></div>
<div class="sect4">
<h5 id="sysadmin_zfs_create_new_zpool">Create a new zpool
 <a class="headerlink" href="#sysadmin_zfs_create_new_zpool" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>To create a new pool, at least one disk is needed. The <span class="monospaced">ashift</span> should have the
same sector-size (2 power of <span class="monospaced">ashift</span>) or larger as the underlying disk.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># zpool create -f -o ashift=12 &lt;pool&gt; &lt;device&gt;</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">
<div class="paragraph">
<p>Pool names must adhere to the following rules:</p></div>
<div class="ulist"><ul>
<li>
<p>
begin with a letter (a-z or A-Z)
</p>
</li>
<li>
<p>
contain only alphanumeric, <span class="monospaced">-</span>, <span class="monospaced">_</span>, <span class="monospaced">.</span>, <span class="monospaced">:</span> or ` ` (space) characters
</p>
</li>
<li>
<p>
must <strong>not begin</strong> with one of <span class="monospaced">mirror</span>, <span class="monospaced">raidz</span>, <span class="monospaced">draid</span> or <span class="monospaced">spare</span>
</p>
</li>
<li>
<p>
must not be <span class="monospaced">log</span>
</p>
</li>
</ul></div>
</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>To activate compression (see section <a href="#zfs_compression">Compression in ZFS</a>):</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># zfs set compression=lz4 &lt;pool&gt;</pre>
</div></div>
</div>
<div class="sect4">
<h5 id="sysadmin_zfs_create_new_zpool_raid0">Create a new pool with RAID-0
 <a class="headerlink" href="#sysadmin_zfs_create_new_zpool_raid0" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Minimum 1 disk</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># zpool create -f -o ashift=12 &lt;pool&gt; &lt;device1&gt; &lt;device2&gt;</pre>
</div></div>
</div>
<div class="sect4">
<h5 id="sysadmin_zfs_create_new_zpool_raid1">Create a new pool with RAID-1
 <a class="headerlink" href="#sysadmin_zfs_create_new_zpool_raid1" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Minimum 2 disks</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># zpool create -f -o ashift=12 &lt;pool&gt; mirror &lt;device1&gt; &lt;device2&gt;</pre>
</div></div>
</div>
<div class="sect4">
<h5 id="sysadmin_zfs_create_new_zpool_raid10">Create a new pool with RAID-10
 <a class="headerlink" href="#sysadmin_zfs_create_new_zpool_raid10" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Minimum 4 disks</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># zpool create -f -o ashift=12 &lt;pool&gt; mirror &lt;device1&gt; &lt;device2&gt; mirror &lt;device3&gt; &lt;device4&gt;</pre>
</div></div>
</div>
<div class="sect4">
<h5 id="sysadmin_zfs_create_new_zpool_raidz1">Create a new pool with RAIDZ-1
 <a class="headerlink" href="#sysadmin_zfs_create_new_zpool_raidz1" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Minimum 3 disks</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># zpool create -f -o ashift=12 &lt;pool&gt; raidz1 &lt;device1&gt; &lt;device2&gt; &lt;device3&gt;</pre>
</div></div>
</div>
<div class="sect4">
<h5 id="_create_a_new_pool_with_raidz_2">Create a new pool with RAIDZ-2
 <a class="headerlink" href="#_create_a_new_pool_with_raidz_2" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Minimum 4 disks</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># zpool create -f -o ashift=12 &lt;pool&gt; raidz2 &lt;device1&gt; &lt;device2&gt; &lt;device3&gt; &lt;device4&gt;</pre>
</div></div>
<div class="paragraph">
<p>Please read the section for
<a href="#sysadmin_zfs_raid_considerations">ZFS RAID Level Considerations</a>
to get a rough estimate on how IOPS and bandwidth expectations before setting up
a pool, especially when wanting to use a RAID-Z mode.</p></div>
</div>
<div class="sect4">
<h5 id="sysadmin_zfs_create_new_zpool_with_cache">Create a new pool with cache (L2ARC)
 <a class="headerlink" href="#sysadmin_zfs_create_new_zpool_with_cache" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>It is possible to use a dedicated device, or partition, as second-level cache to
increase the performance. Such a cache device will especially help with
random-read workloads of data that is mostly static. As it acts as additional
caching layer between the actual storage, and the in-memory ARC, it can also
help if the ARC must be reduced due to memory constraints.</p></div>
<div class="listingblock">
<div class="title">Create ZFS pool with a on-disk cache</div>
<div class="content monospaced">
<pre># zpool create -f -o ashift=12 &lt;pool&gt; &lt;device&gt; cache &lt;cache-device&gt;</pre>
</div></div>
<div class="paragraph">
<p>Here only a single <span class="monospaced">&lt;device&gt;</span> and a single <span class="monospaced">&lt;cache-device&gt;</span> was used, but it is
possible to use more devices, like it’s shown in
<a href="#sysadmin_zfs_create_new_zpool_raid0">Create a new pool with RAID</a>.</p></div>
<div class="paragraph">
<p>Note that for cache devices no mirror or raid modi exist, they are all simply
accumulated.</p></div>
<div class="paragraph">
<p>If any cache device produces errors on read, ZFS will transparently divert that
request to the underlying storage layer.</p></div>
</div>
<div class="sect4">
<h5 id="sysadmin_zfs_create_new_zpool_with_log">Create a new pool with log (ZIL)
 <a class="headerlink" href="#sysadmin_zfs_create_new_zpool_with_log" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>It is possible to use a dedicated drive, or partition, for the ZFS Intent Log
(ZIL), it is mainly used to provide safe synchronous transactions, so often in
performance critical paths like databases, or other programs that issue <span class="monospaced">fsync</span>
operations more frequently.</p></div>
<div class="paragraph">
<p>The pool is used as default ZIL location, diverting the ZIL IO load to a
separate device can, help to reduce transaction latencies while relieving the
main pool at the same time, increasing overall performance.</p></div>
<div class="paragraph">
<p>For disks to be used as log devices, directly or through a partition, it’s
recommend to:</p></div>
<div class="ulist"><ul>
<li>
<p>
use fast SSDs with power-loss protection, as those have much smaller commit
  latencies.
</p>
</li>
<li>
<p>
Use at least a few GB for the partition (or whole device), but using more than
  half of your installed memory won’t provide you with any real advantage.
</p>
</li>
</ul></div>
<div class="listingblock">
<div class="title">Create ZFS pool with separate log device</div>
<div class="content monospaced">
<pre># zpool create -f -o ashift=12 &lt;pool&gt; &lt;device&gt; log &lt;log-device&gt;</pre>
</div></div>
<div class="paragraph">
<p>In above example a single <span class="monospaced">&lt;device&gt;</span> and a single <span class="monospaced">&lt;log-device&gt;</span> is used, but you
can also combine this with other RAID variants, as described in the
<a href="#sysadmin_zfs_create_new_zpool_raid0">Create a new pool with RAID</a> section.</p></div>
<div class="paragraph">
<p>You can also mirror the log device to multiple devices, this is mainly useful to
ensure that performance doesn’t immediately degrades if a single log device
fails.</p></div>
<div class="paragraph">
<p>If all log devices fail the ZFS main pool itself will be used again, until the
log device(s) get replaced.</p></div>
</div>
<div class="sect4">
<h5 id="sysadmin_zfs_add_cache_and_log_dev">Add cache and log to an existing pool
 <a class="headerlink" href="#sysadmin_zfs_add_cache_and_log_dev" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>If you have a pool without cache and log you can still add both, or just one of
them, at any time.</p></div>
<div class="paragraph">
<p>For example, let’s assume you got a good enterprise SSD with power-loss
protection that you want to use for improving the overall performance of your
pool.</p></div>
<div class="paragraph">
<p>As the maximum size of a log device should be about half the size of the
installed physical memory, it means that the ZIL will mostly likely only take up
a relatively small part of the SSD, the remaining space can be used as cache.</p></div>
<div class="paragraph">
<p>First you have to create two GPT partitions on the SSD with <span class="monospaced">parted</span> or <span class="monospaced">gdisk</span>.</p></div>
<div class="paragraph">
<p>Then you’re ready to add them to an pool:</p></div>
<div class="listingblock">
<div class="title">Add both, a separate log device and a second-level cache, to an existing pool</div>
<div class="content monospaced">
<pre># zpool add -f &lt;pool&gt; log &lt;device-part1&gt; cache &lt;device-part2&gt;</pre>
</div></div>
<div class="paragraph">
<p>Just replay <span class="monospaced">&lt;pool&gt;</span>, <span class="monospaced">&lt;device-part1&gt;</span> and <span class="monospaced">&lt;device-part2&gt;</span> with the pool name
and the two <span class="monospaced">/dev/disk/by-id/</span> paths to the partitions.</p></div>
<div class="paragraph">
<p>You can also add ZIL and cache separately.</p></div>
<div class="listingblock">
<div class="title">Add a log device to an existing ZFS pool</div>
<div class="content monospaced">
<pre># zpool add &lt;pool&gt; log &lt;log-device&gt;</pre>
</div></div>
</div>
<div class="sect4">
<h5 id="sysadmin_zfs_change_failed_dev">Changing a failed device
 <a class="headerlink" href="#sysadmin_zfs_change_failed_dev" title="Permalink to this heading"></a>
</h5>
<div class="listingblock">
<div class="content monospaced">
<pre># zpool replace -f &lt;pool&gt; &lt;old-device&gt; &lt;new-device&gt;</pre>
</div></div>
<div class="paragraph">
<div class="title">Changing a failed bootable device</div><p>Depending on how Proxmox VE was installed it is either using <span class="monospaced">systemd-boot</span> or GRUB
through <span class="monospaced">proxmox-boot-tool</span> <span class="footnote" data-note="Systems installed with Proxmox VE 6.4 or later,
EFI systems installed with Proxmox VE 5.4 or later">[<a id="_footnoteref_3" href="#_footnote_3" title="View footnote" class="footnote">3</a>]</span> or plain GRUB as bootloader (see
<a href="#sysboot">Host Bootloader</a>). You can check by running:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># proxmox-boot-tool status</pre>
</div></div>
<div class="paragraph">
<p>The first steps of copying the partition table, reissuing GUIDs and replacing
the ZFS partition are the same. To make the system bootable from the new disk,
different steps are needed which depend on the bootloader in use.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># sgdisk &lt;healthy bootable device&gt; -R &lt;new device&gt;
# sgdisk -G &lt;new device&gt;
# zpool replace -f &lt;pool&gt; &lt;old zfs partition&gt; &lt;new zfs partition&gt;</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Use the <span class="monospaced">zpool status -v</span> command to monitor how far the resilvering
process of the new disk has progressed.</td>
</tr></tbody></table>
</div>
<div class="listingblock">
<div class="title">With <span class="monospaced">proxmox-boot-tool</span>:</div>
<div class="content monospaced">
<pre># proxmox-boot-tool format &lt;new disk's ESP&gt;
# proxmox-boot-tool init &lt;new disk's ESP&gt; [grub]</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content"><span class="monospaced">ESP</span> stands for EFI System Partition, which is setup as partition #2 on
bootable disks setup by the Proxmox VE installer since version 5.4. For details, see
<a href="#sysboot_proxmox_boot_setup">Setting up a new partition for use as synced ESP</a>.</td>
</tr></tbody></table>
</div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Make sure to pass <em>grub</em> as mode to <span class="monospaced">proxmox-boot-tool init</span> if
<span class="monospaced">proxmox-boot-tool status</span> indicates your current disks are using GRUB,
especially if Secure Boot is enabled!</td>
</tr></tbody></table>
</div>
<div class="listingblock">
<div class="title">With plain GRUB:</div>
<div class="content monospaced">
<pre># grub-install &lt;new disk&gt;</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Plain GRUB is only used on systems installed with Proxmox VE 6.3 or earlier,
which have not been manually migrated to using <span class="monospaced">proxmox-boot-tool</span> yet.</td>
</tr></tbody></table>
</div>
</div>
</div>
<div class="sect3">
<h4 id="_configure_e_mail_notification">3.9.7. Configure E-Mail Notification
 <a class="headerlink" href="#_configure_e_mail_notification" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>ZFS comes with an event daemon <span class="monospaced">ZED</span>, which monitors events generated by the ZFS
kernel module. The daemon can also send emails on ZFS events like pool errors.
Newer ZFS packages ship the daemon in a separate <span class="monospaced">zfs-zed</span> package, which should
already be installed by default in Proxmox VE.</p></div>
<div class="paragraph">
<p>You can configure the daemon via the file <span class="monospaced">/etc/zfs/zed.d/zed.rc</span> with your
favorite editor. The required setting for email notification is
<span class="monospaced">ZED_EMAIL_ADDR</span>, which is set to <span class="monospaced">root</span> by default.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>ZED_EMAIL_ADDR="root"</pre>
</div></div>
<div class="paragraph">
<p>Please note Proxmox VE forwards mails to <span class="monospaced">root</span> to the email address
configured for the root user.</p></div>
</div>
<div class="sect3">
<h4 id="sysadmin_zfs_limit_memory_usage">3.9.8. Limit ZFS Memory Usage
 <a class="headerlink" href="#sysadmin_zfs_limit_memory_usage" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>ZFS uses <em>50 %</em> of the host memory for the <strong>A</strong>daptive <strong>R</strong>eplacement
<strong>C</strong>ache (ARC) by default. For new installations starting with Proxmox VE 8.1, the
ARC usage limit will be set to <em>10 %</em> of the installed physical memory, clamped
to a maximum of <span class="monospaced">16 GiB</span>. This value is written to <span class="monospaced">/etc/modprobe.d/zfs.conf</span>.</p></div>
<div class="paragraph">
<p>Allocating enough memory for the ARC is crucial for IO performance, so reduce it
with caution. As a general rule of thumb, allocate at least <span class="monospaced">2 GiB Base + 1
GiB/TiB-Storage</span>. For example, if you have a pool with <span class="monospaced">8 TiB</span> of available
storage space then you should use <span class="monospaced">10 GiB</span> of memory for the ARC.</p></div>
<div class="paragraph">
<p>ZFS also enforces a minimum value of <span class="monospaced">64 MiB</span>.</p></div>
<div class="paragraph">
<p>You can change the ARC usage limit for the current boot (a reboot resets this
change again) by writing to the <span class="monospaced">zfs_arc_max</span> module parameter directly:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre> echo "$[10 * 1024*1024*1024]" &gt;/sys/module/zfs/parameters/zfs_arc_max</pre>
</div></div>
<div class="paragraph">
<p>To <strong>permanently change</strong> the ARC limits, add (or change if already present) the
following line to <span class="monospaced">/etc/modprobe.d/zfs.conf</span>:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>options zfs zfs_arc_max=8589934592</pre>
</div></div>
<div class="paragraph">
<p>This example setting limits the usage to 8 GiB (<em>8 * 2<sup>30</sup></em>).</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Important" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAALa0lEQVRogdWZa2wc1RXHfzM7O/te
P9e1vXHSmEdjx3YeDkkaF6REKRRCEDSEFNkRjdSWSsgC2iqoRWqLQKiqIBg1NOQDiMeHtkQIQkRR
S9S4aWwgCQoUgl232KkT28J21l6vd3d2dx79sDuTXdtre03Uqlc62tl53Pv/n3PuOffcKxiGwf9z
E//XAL5sk65WR0a6AaDrOjMtKwgCoigiCAKCIAhXa9wvTcAEHhoc5KMjR7jw3nucf+MN67k/GCS4
YQOr77yTpl27kGUZm81mXC0ywlLnQDbwEwcO0P2b3yz4jV5RwZq2Nm5/5BH8fj+SJGGz2b4UiSUR
MAzD0DSNEwcO8MdHHln8d0AcsAUCfOe551h7yy04HA7sdvuSrVEwAV3XjdDgIC/v2cOl06dznqWA
JCAAMvn9MwlEgF1PP8037rsPt9uN3W5fkjUKIqDrujF+4QLP3XQTU0NDOc8UIAa4y8sJtrQQqK/H
7XQSfv99Pn/nHQRRxNB16/0EEAUatm9n76uv4vP5cDgcBZNYNIF84I0MEK20lIZvf5sNe/dSWlqK
1+vFxBEZHubT3/+evqNHmR4dJRWP55BevW0bra+8gt/vx+l0IknSokksikA+8ElgClixYwc3799P
ZWUlTqczbz+x0VFOPvEEQ6dPE/niCwxdt0jUbd1K68svU1xcXBCJBQnMBz5eWkrL/v2s++Y3KSsr
QxQXlxc7f/5zBjo7CQ8NoadSKKQn943t7dyyf79FYjHuNC8BwzAMVVV56Z57+OzNN3PAJ8vK+FZH
B2s2b55X6/nahRMnOP7TnxLOKEXJyO2/+hVf37uXoqIinE6nmfzyksirMjNUvvGjH+WAVwG1vJxb
Dhxg1bp1uFwuMwQWJCu3bWPN3r24SkoAcGbA/OXXv2bws8+IRqOkUqlZGX1RBAzDMHRd51+nTvHe
wYNX7gPTQMO+faxav56SkpIlgTdlU3s7TXv2IGUs6AGioRDH2tuZmJggFouhqirGPCzyEUBVVX7X
1pZzPwI0fP/7bNmzh7KysrzADh8+zP3338++ffs4ePBg3vdEUWTDD3/IsuZmRLsdW4bEpd5eOp99
lnA4jKIoaJqWl8QsAqb2Txw4kDNp40BxXR0bWlspLy/PC+qZZ55hdHQUSZIYHByko6ODxx57bBZw
cy3kq6xk6y9/ib+qCgQBmbQ7nT50iIs9PUSjUZLJJIZhzEliLgKMDQzwp5/9zLqnkQ51W3/xCwKB
AE6nMy+B3t5exsfH6ezspLe3l3g8zuHDh2cBz5bKpibWtrXhLi0FwA3owNsPP8zExATxeBxVVecy
QC4Bc+L+taNjluvU33031dddN6/2BUGgpKSE/v5+JicncweaA3i2tDz8MIHrr0eU0gsQBzDW10ff
yZNEIhESicScrpRDQNd1xgYG+OC3v7XuJQBXIMDWn/yEioqKeUGMjY0hCAKKoszS1MjIyLzfAmx/
4gmKli0DQcBJOmicfOopJicnicfjJoG5LWBq/+Szz+a8oABNe/bgdrvndR1BEKiqqiIejxOLxWYR
CAaDC0alqjVrqGxowFNWhkDalUKff07f3/5mWSFTLFksLAK6rjN+4UKO9lXAUV7O2tbWeaNOthZl
WZ4FHlh0aN3y4IPIHk+6L8AGvNfRQTgctuZCthVEU/u6rnPutddyBk0A9bt3W4us+QY2fbyurg6v
15vTzw033LBoAsHmZoLNzdgcDgBcQKi/n391dTE9PU0ymcyxgkmAVCrF+4cO5WjfFgiwfoGwOTOy
lJWVUVtba1mkUAKCILClvR1veTkA9owVPnzllZy8YFnA1H5/dzfTw8PWgziwcutWfD4fsiwvCNyU
xsZG3G43lZWVVl933XVXQQSq161j+ebNlhXcwBdnzxIOh60lhmkFEUDTNP7++uvWgBrp6mrNHBk3
H/BsAtFoFEdm8GAwyMaNGwsiIAgCjbt346uosKwgAp8cPWolNj1THInmsqHn6FGLQApY1thIxcqV
+Hy+WX6+0ETesWMHLpcLWZZ54IEHCgYvCAKrbrsNyem0JrQEXMzkhKzlBaKu61z+97+ZHhnJmbwr
tm2zJu5CWp8pra2tNDU1sXPnTtrb25dEQBAEvrplC57MXJCB8XPniEQixONxa6Uq6brOQHd3jvvo
QP2ttxZUpMxsL7zwwpK+AyxLXrt9Oxe6uxFEEXvGZc4fO0bVD35AMplM1wuapjGYtbugAp5AgKLq
aquu/W/JTEs37dqFqihWzSADk8PD1jzQNC3tQtmrThUINDbm+H4h0tfXx6OPPsodd9xBW1sbPT09
BQPPluu2b7fWRxIQ6usjFotZBCRN06wZbbZAfT1FRUWWKQtpzz//PMPDw/T09DA1NcXo6CjHjx+f
11Xma95AwHJjEdB0HUVRrhCYuRGrAiXBoFUqFtq6u7sZHx9nJBMUurq65uxnsX0Hm5v56A9/ANIJ
LXrpEolEwgqlUqZQyPlo6OOPlzx5ly1bxnBWQmxpackBW6hSBEC02axrpqasNZGmaek8kN0kYPDj
j1FVdUlz4Mknn6Surs4Cb5aUhYZiUy59+GGOgm3V1WiadiUPCILAV9autV5wAaM9PXQeOWJprBBp
aGjgxIkTpFIpOjs7aWxsXBJwUwZOnUKZmgLS2zl4vUjSlV1XURRFqtavv3IDKAVe/O53efPxx69a
SCxUBs+c4dV77+Vyfz+xy5eBdG3iX7ECm81m9S2JokiwuZmiujrCPT1AuqgOAqcef5zzL77Imt27
uemhhyipqVnYZ5cw8bPbR0eOcOall7g8MEB4aIhEJAJk9qOAwKZNyLJsHpQgRKNRIxQK0XPmDH/+
3vdQQ6G8na/YtIk199zDypYWVmzceFWAT1y8yMWzZzn/9tv0vfsudqeTlKIQHRvL2QSOAqU7d/K1
W2/l2muvpba2Nl3iJhIJIxKJMDw8TO/Zs3zw4x+jTUzkHVCw2fCUleHw+QiuXYu/upqqhgb8VVUY
hkFNczPFweCs7z556y3r+vyxYwiiyD+PHwdBwNB1krEYhqYRC4UwdB2D9LJmmnRtXHz77VTfeCPL
ly+ntraWmpqa9MaaqqqGoiiEQiGGhoYYHBzk00OHiHR2LkqDNlnG7nJhd7nQswqN+ZogiuipFMlY
DDWRgBl5KEF6woqAUFFByc03U756NZWVlSxfvpyamhoCgQAejwdB13VDVVVisRihUIiRkRGGh4cZ
+sc/GO3qIt7VhZGJAle7mVpWuVKDCBng8jXX4N68Gc8111BUVER5eTnV1dVUV1cTCASuFFqGYaDr
upFKpYjH40xMTDA+Ps7Y2Bjj4+OEw2EmenqYPneO1OAgZCWpxTY9S9QMcLhyHGUDRI8HqbISZ309
zoYGnE4nHo8Hv99PaWkpgUCAiooK6/DE4XCkI1HW2a6hqiqKojA9Pc3k5KQl4XCY6elpotFoetuk
v59UKIR66RLToRBJTcMYHUWIRvMSMPO6rbY2/SsIOFatQpYk5NWrkSQJWZZxOBy43W68Xi9+v5+i
oiJKSkooLi7G5/PhdruRZdnads85HzD3hpLJJIqiEIvFiEajRCIRIpEI0WiUWCxGPB631iOKopBK
pazUbi4Oc7Y+snKCzWbDZrMhSRKSJGG3262w6HK5cLlceDwevF4vPp8Pr9eL2+3G5XLhcDjMkxuE
TNibdcBhnv9qmkYqlSKZTJJIJFAUhXg8jqIoKIpCIpGwSCSTSVRVtWSuk3qTiAnebrfngHc4HDid
TpxOJy6Xy7qWZdk6wZzrxCbvCY1JRNd1NE3LAWgSM69ngp+PgCiK1gF3NhFTzHvmO+Y3wCzw8xLI
JpL5tYCZpEyw2f/nWt1ag2UtM0x3Mq1iAp1j+bH0M7KFCGWDnfmbd8AMnuxfUzL/C0rp/wFnFd4n
EQn3XQAAAABJRU5ErkJggg==">
</td>
<td class="content">In case your desired <span class="monospaced">zfs_arc_max</span> value is lower than or equal to
<span class="monospaced">zfs_arc_min</span> (which defaults to 1/32 of the system memory), <span class="monospaced">zfs_arc_max</span> will
be ignored unless you also set <span class="monospaced">zfs_arc_min</span> to at most <span class="monospaced">zfs_arc_max - 1</span>.</td>
</tr></tbody></table>
</div>
<div class="listingblock">
<div class="content monospaced">
<pre>echo "$[8 * 1024*1024*1024 - 1]" &gt;/sys/module/zfs/parameters/zfs_arc_min
echo "$[8 * 1024*1024*1024]" &gt;/sys/module/zfs/parameters/zfs_arc_max</pre>
</div></div>
<div class="paragraph">
<p>This example setting (temporarily) limits the usage to 8 GiB (<em>8 * 2<sup>30</sup></em>) on
systems with more than 256 GiB of total memory, where simply setting
<span class="monospaced">zfs_arc_max</span> alone would not work.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Important" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAALa0lEQVRogdWZa2wc1RXHfzM7O/te
P9e1vXHSmEdjx3YeDkkaF6REKRRCEDSEFNkRjdSWSsgC2iqoRWqLQKiqIBg1NOQDiMeHtkQIQkRR
S9S4aWwgCQoUgl232KkT28J21l6vd3d2dx79sDuTXdtre03Uqlc62tl53Pv/n3PuOffcKxiGwf9z
E//XAL5sk65WR0a6AaDrOjMtKwgCoigiCAKCIAhXa9wvTcAEHhoc5KMjR7jw3nucf+MN67k/GCS4
YQOr77yTpl27kGUZm81mXC0ywlLnQDbwEwcO0P2b3yz4jV5RwZq2Nm5/5BH8fj+SJGGz2b4UiSUR
MAzD0DSNEwcO8MdHHln8d0AcsAUCfOe551h7yy04HA7sdvuSrVEwAV3XjdDgIC/v2cOl06dznqWA
JCAAMvn9MwlEgF1PP8037rsPt9uN3W5fkjUKIqDrujF+4QLP3XQTU0NDOc8UIAa4y8sJtrQQqK/H
7XQSfv99Pn/nHQRRxNB16/0EEAUatm9n76uv4vP5cDgcBZNYNIF84I0MEK20lIZvf5sNe/dSWlqK
1+vFxBEZHubT3/+evqNHmR4dJRWP55BevW0bra+8gt/vx+l0IknSokksikA+8ElgClixYwc3799P
ZWUlTqczbz+x0VFOPvEEQ6dPE/niCwxdt0jUbd1K68svU1xcXBCJBQnMBz5eWkrL/v2s++Y3KSsr
QxQXlxc7f/5zBjo7CQ8NoadSKKQn943t7dyyf79FYjHuNC8BwzAMVVV56Z57+OzNN3PAJ8vK+FZH
B2s2b55X6/nahRMnOP7TnxLOKEXJyO2/+hVf37uXoqIinE6nmfzyksirMjNUvvGjH+WAVwG1vJxb
Dhxg1bp1uFwuMwQWJCu3bWPN3r24SkoAcGbA/OXXv2bws8+IRqOkUqlZGX1RBAzDMHRd51+nTvHe
wYNX7gPTQMO+faxav56SkpIlgTdlU3s7TXv2IGUs6AGioRDH2tuZmJggFouhqirGPCzyEUBVVX7X
1pZzPwI0fP/7bNmzh7KysrzADh8+zP3338++ffs4ePBg3vdEUWTDD3/IsuZmRLsdW4bEpd5eOp99
lnA4jKIoaJqWl8QsAqb2Txw4kDNp40BxXR0bWlspLy/PC+qZZ55hdHQUSZIYHByko6ODxx57bBZw
cy3kq6xk6y9/ib+qCgQBmbQ7nT50iIs9PUSjUZLJJIZhzEliLgKMDQzwp5/9zLqnkQ51W3/xCwKB
AE6nMy+B3t5exsfH6ezspLe3l3g8zuHDh2cBz5bKpibWtrXhLi0FwA3owNsPP8zExATxeBxVVecy
QC4Bc+L+taNjluvU33031dddN6/2BUGgpKSE/v5+JicncweaA3i2tDz8MIHrr0eU0gsQBzDW10ff
yZNEIhESicScrpRDQNd1xgYG+OC3v7XuJQBXIMDWn/yEioqKeUGMjY0hCAKKoszS1MjIyLzfAmx/
4gmKli0DQcBJOmicfOopJicnicfjJoG5LWBq/+Szz+a8oABNe/bgdrvndR1BEKiqqiIejxOLxWYR
CAaDC0alqjVrqGxowFNWhkDalUKff07f3/5mWSFTLFksLAK6rjN+4UKO9lXAUV7O2tbWeaNOthZl
WZ4FHlh0aN3y4IPIHk+6L8AGvNfRQTgctuZCthVEU/u6rnPutddyBk0A9bt3W4us+QY2fbyurg6v
15vTzw033LBoAsHmZoLNzdgcDgBcQKi/n391dTE9PU0ymcyxgkmAVCrF+4cO5WjfFgiwfoGwOTOy
lJWVUVtba1mkUAKCILClvR1veTkA9owVPnzllZy8YFnA1H5/dzfTw8PWgziwcutWfD4fsiwvCNyU
xsZG3G43lZWVVl933XVXQQSq161j+ebNlhXcwBdnzxIOh60lhmkFEUDTNP7++uvWgBrp6mrNHBk3
H/BsAtFoFEdm8GAwyMaNGwsiIAgCjbt346uosKwgAp8cPWolNj1THInmsqHn6FGLQApY1thIxcqV
+Hy+WX6+0ETesWMHLpcLWZZ54IEHCgYvCAKrbrsNyem0JrQEXMzkhKzlBaKu61z+97+ZHhnJmbwr
tm2zJu5CWp8pra2tNDU1sXPnTtrb25dEQBAEvrplC57MXJCB8XPniEQixONxa6Uq6brOQHd3jvvo
QP2ttxZUpMxsL7zwwpK+AyxLXrt9Oxe6uxFEEXvGZc4fO0bVD35AMplM1wuapjGYtbugAp5AgKLq
aquu/W/JTEs37dqFqihWzSADk8PD1jzQNC3tQtmrThUINDbm+H4h0tfXx6OPPsodd9xBW1sbPT09
BQPPluu2b7fWRxIQ6usjFotZBCRN06wZbbZAfT1FRUWWKQtpzz//PMPDw/T09DA1NcXo6CjHjx+f
11Xma95AwHJjEdB0HUVRrhCYuRGrAiXBoFUqFtq6u7sZHx9nJBMUurq65uxnsX0Hm5v56A9/ANIJ
LXrpEolEwgqlUqZQyPlo6OOPlzx5ly1bxnBWQmxpackBW6hSBEC02axrpqasNZGmaek8kN0kYPDj
j1FVdUlz4Mknn6Surs4Cb5aUhYZiUy59+GGOgm3V1WiadiUPCILAV9autV5wAaM9PXQeOWJprBBp
aGjgxIkTpFIpOjs7aWxsXBJwUwZOnUKZmgLS2zl4vUjSlV1XURRFqtavv3IDKAVe/O53efPxx69a
SCxUBs+c4dV77+Vyfz+xy5eBdG3iX7ECm81m9S2JokiwuZmiujrCPT1AuqgOAqcef5zzL77Imt27
uemhhyipqVnYZ5cw8bPbR0eOcOall7g8MEB4aIhEJAJk9qOAwKZNyLJsHpQgRKNRIxQK0XPmDH/+
3vdQQ6G8na/YtIk199zDypYWVmzceFWAT1y8yMWzZzn/9tv0vfsudqeTlKIQHRvL2QSOAqU7d/K1
W2/l2muvpba2Nl3iJhIJIxKJMDw8TO/Zs3zw4x+jTUzkHVCw2fCUleHw+QiuXYu/upqqhgb8VVUY
hkFNczPFweCs7z556y3r+vyxYwiiyD+PHwdBwNB1krEYhqYRC4UwdB2D9LJmmnRtXHz77VTfeCPL
ly+ntraWmpqa9MaaqqqGoiiEQiGGhoYYHBzk00OHiHR2LkqDNlnG7nJhd7nQswqN+ZogiuipFMlY
DDWRgBl5KEF6woqAUFFByc03U756NZWVlSxfvpyamhoCgQAejwdB13VDVVVisRihUIiRkRGGh4cZ
+sc/GO3qIt7VhZGJAle7mVpWuVKDCBng8jXX4N68Gc8111BUVER5eTnV1dVUV1cTCASuFFqGYaDr
upFKpYjH40xMTDA+Ps7Y2Bjj4+OEw2EmenqYPneO1OAgZCWpxTY9S9QMcLhyHGUDRI8HqbISZ309
zoYGnE4nHo8Hv99PaWkpgUCAiooK6/DE4XCkI1HW2a6hqiqKojA9Pc3k5KQl4XCY6elpotFoetuk
v59UKIR66RLToRBJTcMYHUWIRvMSMPO6rbY2/SsIOFatQpYk5NWrkSQJWZZxOBy43W68Xi9+v5+i
oiJKSkooLi7G5/PhdruRZdnads85HzD3hpLJJIqiEIvFiEajRCIRIpEI0WiUWCxGPB631iOKopBK
pazUbi4Oc7Y+snKCzWbDZrMhSRKSJGG3262w6HK5cLlceDwevF4vPp8Pr9eL2+3G5XLhcDjMkxuE
TNibdcBhnv9qmkYqlSKZTJJIJFAUhXg8jqIoKIpCIpGwSCSTSVRVtWSuk3qTiAnebrfngHc4HDid
TpxOJy6Xy7qWZdk6wZzrxCbvCY1JRNd1NE3LAWgSM69ngp+PgCiK1gF3NhFTzHvmO+Y3wCzw8xLI
JpL5tYCZpEyw2f/nWt1ag2UtM0x3Mq1iAp1j+bH0M7KFCGWDnfmbd8AMnuxfUzL/C0rp/wFnFd4n
EQn3XQAAAABJRU5ErkJggg==">
</td>
<td class="content">
<div class="paragraph">
<p>If your root file system is ZFS, you must update your initramfs every
time this value changes:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># update-initramfs -u -k all</pre>
</div></div>
<div class="paragraph">
<p>You <strong>must reboot</strong> to activate these changes.</p></div>
</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="zfs_swap">3.9.9. SWAP on ZFS
 <a class="headerlink" href="#zfs_swap" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Swap-space created on a zvol may generate some troubles, like blocking the
server or generating a high IO load, often seen when starting a Backup
to an external Storage.</p></div>
<div class="paragraph">
<p>We strongly recommend to use enough memory, so that you normally do not
run into low memory situations. Should you need or want to add swap, it is
preferred to create a partition on a physical disk and use it as a swap device.
You can leave some space free for this purpose in the advanced options of the
installer. Additionally, you can lower the
“swappiness” value. A good value for servers is 10:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># sysctl -w vm.swappiness=10</pre>
</div></div>
<div class="paragraph">
<p>To make the swappiness persistent, open <span class="monospaced">/etc/sysctl.conf</span> with
an editor of your choice and add the following line:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>vm.swappiness = 10</pre>
</div></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<caption class="title">Table 1. Linux kernel <span class="monospaced">swappiness</span> parameter values</caption>
<colgroup><col style="width:33%;">
<col style="width:66%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top"> Value               </th>
<th class="tableblock halign-left valign-top"> Strategy</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">vm.swappiness = 0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">The kernel will swap only to avoid
an <em>out of memory</em> condition</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">vm.swappiness = 1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Minimum amount of swapping without
disabling it entirely.</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">vm.swappiness = 10</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">This value is sometimes recommended to
improve performance when sufficient memory exists in a system.</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">vm.swappiness = 60</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">The default value.</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">vm.swappiness = 100</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">The kernel will swap aggressively.</p></td>
</tr>
</tbody>
</table>
</div>
<div class="sect3">
<h4 id="zfs_encryption">3.9.10. Encrypted ZFS Datasets
 <a class="headerlink" href="#zfs_encryption" title="Permalink to this heading"></a>
</h4>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">Native ZFS encryption in Proxmox VE is experimental. Known limitations and
issues include Replication with encrypted datasets
<span class="footnote" data-note="<a href=&quot;https://bugzilla.proxmox.com/show_bug.cgi?id=2350&quot;>https://bugzilla.proxmox.com/show_bug.cgi?id=2350</a>">[<a id="_footnoteref_4" href="#_footnote_4" title="View footnote" class="footnote">4</a>]</span>,
as well as checksum errors when using Snapshots or ZVOLs.
<span class="footnote" data-note="<a href=&quot;https://github.com/openzfs/zfs/issues/11688&quot;>https://github.com/openzfs/zfs/issues/11688</a>">[<a id="_footnoteref_5" href="#_footnote_5" title="View footnote" class="footnote">5</a>]</span></td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>ZFS on Linux version 0.8.0 introduced support for native encryption of
datasets. After an upgrade from previous ZFS on Linux versions, the encryption
feature can be enabled per pool:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># zpool get feature@encryption tank
NAME  PROPERTY            VALUE            SOURCE
tank  feature@encryption  disabled         local

# zpool set feature@encryption=enabled

# zpool get feature@encryption tank
NAME  PROPERTY            VALUE            SOURCE
tank  feature@encryption  enabled         local</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">There is currently no support for booting from pools with encrypted
datasets using GRUB, and only limited support for automatically unlocking
encrypted datasets on boot. Older versions of ZFS without encryption support
will not be able to decrypt stored data.</td>
</tr></tbody></table>
</div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">It is recommended to either unlock storage datasets manually after
booting, or to write a custom unit to pass the key material needed for
unlocking on boot to <span class="monospaced">zfs load-key</span>.</td>
</tr></tbody></table>
</div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">Establish and test a backup procedure before enabling encryption of
production data. If the associated key material/passphrase/keyfile has been
lost, accessing the encrypted data is no longer possible.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>Encryption needs to be setup when creating datasets/zvols, and is inherited by
default to child datasets. For example, to create an encrypted dataset
<span class="monospaced">tank/encrypted_data</span> and configure it as storage in Proxmox VE, run the following
commands:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># zfs create -o encryption=on -o keyformat=passphrase tank/encrypted_data
Enter passphrase:
Re-enter passphrase:

# pvesm add zfspool encrypted_zfs -pool tank/encrypted_data</pre>
</div></div>
<div class="paragraph">
<p>All guest volumes/disks create on this storage will be encrypted with the
shared key material of the parent dataset.</p></div>
<div class="paragraph">
<p>To actually use the storage, the associated key material needs to be loaded
and the dataset needs to be mounted. This can be done in one step with:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># zfs mount -l tank/encrypted_data
Enter passphrase for 'tank/encrypted_data':</pre>
</div></div>
<div class="paragraph">
<p>It is also possible to use a (random) keyfile instead of prompting for a
passphrase by setting the <span class="monospaced">keylocation</span> and <span class="monospaced">keyformat</span> properties, either at
creation time or with <span class="monospaced">zfs change-key</span> on existing datasets:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># dd if=/dev/urandom of=/path/to/keyfile bs=32 count=1

# zfs change-key -o keyformat=raw -o keylocation=file:///path/to/keyfile tank/encrypted_data</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">When using a keyfile, special care needs to be taken to secure the
keyfile against unauthorized access or accidental loss. Without the keyfile, it
is not possible to access the plaintext data!</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>A guest volume created underneath an encrypted dataset will have its
<span class="monospaced">encryptionroot</span> property set accordingly. The key material only needs to be
loaded once per encryptionroot to be available to all encrypted datasets
underneath it.</p></div>
<div class="paragraph">
<p>See the <span class="monospaced">encryptionroot</span>, <span class="monospaced">encryption</span>, <span class="monospaced">keylocation</span>, <span class="monospaced">keyformat</span> and
<span class="monospaced">keystatus</span> properties, the <span class="monospaced">zfs load-key</span>, <span class="monospaced">zfs unload-key</span> and <span class="monospaced">zfs
change-key</span> commands and the <span class="monospaced">Encryption</span> section from <span class="monospaced">man zfs</span> for more
details and advanced usage.</p></div>
</div>
<div class="sect3">
<h4 id="zfs_compression">3.9.11. Compression in ZFS
 <a class="headerlink" href="#zfs_compression" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>When compression is enabled on a dataset, ZFS tries to compress all <strong>new</strong>
blocks before writing them and decompresses them on reading. Already
existing data will not be compressed retroactively.</p></div>
<div class="paragraph">
<p>You can enable compression with:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># zfs set compression=&lt;algorithm&gt; &lt;dataset&gt;</pre>
</div></div>
<div class="paragraph">
<p>We recommend using the <span class="monospaced">lz4</span> algorithm, because it adds very little CPU
overhead. Other algorithms like <span class="monospaced">lzjb</span> and <span class="monospaced">gzip-N</span>, where <span class="monospaced">N</span> is an
integer from <span class="monospaced">1</span> (fastest) to <span class="monospaced">9</span> (best compression ratio), are also
available. Depending on the algorithm and how compressible the data is,
having compression enabled can even increase I/O performance.</p></div>
<div class="paragraph">
<p>You can disable compression at any time with:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># zfs set compression=off &lt;dataset&gt;</pre>
</div></div>
<div class="paragraph">
<p>Again, only new blocks will be affected by this change.</p></div>
</div>
<div class="sect3">
<h4 id="sysadmin_zfs_special_device">3.9.12. ZFS Special Device
 <a class="headerlink" href="#sysadmin_zfs_special_device" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Since version 0.8.0 ZFS supports <span class="monospaced">special</span> devices. A <span class="monospaced">special</span> device in a
pool is used to store metadata, deduplication tables, and optionally small
file blocks.</p></div>
<div class="paragraph">
<p>A <span class="monospaced">special</span> device can improve the speed of a pool consisting of slow spinning
hard disks with a lot of metadata changes. For example workloads that involve
creating, updating or deleting a large number of files will benefit from the
presence of a <span class="monospaced">special</span> device. ZFS datasets can also be configured to store
whole small files on the <span class="monospaced">special</span> device which can further improve the
performance. Use fast SSDs for the <span class="monospaced">special</span> device.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Important" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAALa0lEQVRogdWZa2wc1RXHfzM7O/te
P9e1vXHSmEdjx3YeDkkaF6REKRRCEDSEFNkRjdSWSsgC2iqoRWqLQKiqIBg1NOQDiMeHtkQIQkRR
S9S4aWwgCQoUgl232KkT28J21l6vd3d2dx79sDuTXdtre03Uqlc62tl53Pv/n3PuOffcKxiGwf9z
E//XAL5sk65WR0a6AaDrOjMtKwgCoigiCAKCIAhXa9wvTcAEHhoc5KMjR7jw3nucf+MN67k/GCS4
YQOr77yTpl27kGUZm81mXC0ywlLnQDbwEwcO0P2b3yz4jV5RwZq2Nm5/5BH8fj+SJGGz2b4UiSUR
MAzD0DSNEwcO8MdHHln8d0AcsAUCfOe551h7yy04HA7sdvuSrVEwAV3XjdDgIC/v2cOl06dznqWA
JCAAMvn9MwlEgF1PP8037rsPt9uN3W5fkjUKIqDrujF+4QLP3XQTU0NDOc8UIAa4y8sJtrQQqK/H
7XQSfv99Pn/nHQRRxNB16/0EEAUatm9n76uv4vP5cDgcBZNYNIF84I0MEK20lIZvf5sNe/dSWlqK
1+vFxBEZHubT3/+evqNHmR4dJRWP55BevW0bra+8gt/vx+l0IknSokksikA+8ElgClixYwc3799P
ZWUlTqczbz+x0VFOPvEEQ6dPE/niCwxdt0jUbd1K68svU1xcXBCJBQnMBz5eWkrL/v2s++Y3KSsr
QxQXlxc7f/5zBjo7CQ8NoadSKKQn943t7dyyf79FYjHuNC8BwzAMVVV56Z57+OzNN3PAJ8vK+FZH
B2s2b55X6/nahRMnOP7TnxLOKEXJyO2/+hVf37uXoqIinE6nmfzyksirMjNUvvGjH+WAVwG1vJxb
Dhxg1bp1uFwuMwQWJCu3bWPN3r24SkoAcGbA/OXXv2bws8+IRqOkUqlZGX1RBAzDMHRd51+nTvHe
wYNX7gPTQMO+faxav56SkpIlgTdlU3s7TXv2IGUs6AGioRDH2tuZmJggFouhqirGPCzyEUBVVX7X
1pZzPwI0fP/7bNmzh7KysrzADh8+zP3338++ffs4ePBg3vdEUWTDD3/IsuZmRLsdW4bEpd5eOp99
lnA4jKIoaJqWl8QsAqb2Txw4kDNp40BxXR0bWlspLy/PC+qZZ55hdHQUSZIYHByko6ODxx57bBZw
cy3kq6xk6y9/ib+qCgQBmbQ7nT50iIs9PUSjUZLJJIZhzEliLgKMDQzwp5/9zLqnkQ51W3/xCwKB
AE6nMy+B3t5exsfH6ezspLe3l3g8zuHDh2cBz5bKpibWtrXhLi0FwA3owNsPP8zExATxeBxVVecy
QC4Bc+L+taNjluvU33031dddN6/2BUGgpKSE/v5+JicncweaA3i2tDz8MIHrr0eU0gsQBzDW10ff
yZNEIhESicScrpRDQNd1xgYG+OC3v7XuJQBXIMDWn/yEioqKeUGMjY0hCAKKoszS1MjIyLzfAmx/
4gmKli0DQcBJOmicfOopJicnicfjJoG5LWBq/+Szz+a8oABNe/bgdrvndR1BEKiqqiIejxOLxWYR
CAaDC0alqjVrqGxowFNWhkDalUKff07f3/5mWSFTLFksLAK6rjN+4UKO9lXAUV7O2tbWeaNOthZl
WZ4FHlh0aN3y4IPIHk+6L8AGvNfRQTgctuZCthVEU/u6rnPutddyBk0A9bt3W4us+QY2fbyurg6v
15vTzw033LBoAsHmZoLNzdgcDgBcQKi/n391dTE9PU0ymcyxgkmAVCrF+4cO5WjfFgiwfoGwOTOy
lJWVUVtba1mkUAKCILClvR1veTkA9owVPnzllZy8YFnA1H5/dzfTw8PWgziwcutWfD4fsiwvCNyU
xsZG3G43lZWVVl933XVXQQSq161j+ebNlhXcwBdnzxIOh60lhmkFEUDTNP7++uvWgBrp6mrNHBk3
H/BsAtFoFEdm8GAwyMaNGwsiIAgCjbt346uosKwgAp8cPWolNj1THInmsqHn6FGLQApY1thIxcqV
+Hy+WX6+0ETesWMHLpcLWZZ54IEHCgYvCAKrbrsNyem0JrQEXMzkhKzlBaKu61z+97+ZHhnJmbwr
tm2zJu5CWp8pra2tNDU1sXPnTtrb25dEQBAEvrplC57MXJCB8XPniEQixONxa6Uq6brOQHd3jvvo
QP2ttxZUpMxsL7zwwpK+AyxLXrt9Oxe6uxFEEXvGZc4fO0bVD35AMplM1wuapjGYtbugAp5AgKLq
aquu/W/JTEs37dqFqihWzSADk8PD1jzQNC3tQtmrThUINDbm+H4h0tfXx6OPPsodd9xBW1sbPT09
BQPPluu2b7fWRxIQ6usjFotZBCRN06wZbbZAfT1FRUWWKQtpzz//PMPDw/T09DA1NcXo6CjHjx+f
11Xma95AwHJjEdB0HUVRrhCYuRGrAiXBoFUqFtq6u7sZHx9nJBMUurq65uxnsX0Hm5v56A9/ANIJ
LXrpEolEwgqlUqZQyPlo6OOPlzx5ly1bxnBWQmxpackBW6hSBEC02axrpqasNZGmaek8kN0kYPDj
j1FVdUlz4Mknn6Surs4Cb5aUhYZiUy59+GGOgm3V1WiadiUPCILAV9autV5wAaM9PXQeOWJprBBp
aGjgxIkTpFIpOjs7aWxsXBJwUwZOnUKZmgLS2zl4vUjSlV1XURRFqtavv3IDKAVe/O53efPxx69a
SCxUBs+c4dV77+Vyfz+xy5eBdG3iX7ECm81m9S2JokiwuZmiujrCPT1AuqgOAqcef5zzL77Imt27
uemhhyipqVnYZ5cw8bPbR0eOcOall7g8MEB4aIhEJAJk9qOAwKZNyLJsHpQgRKNRIxQK0XPmDH/+
3vdQQ6G8na/YtIk199zDypYWVmzceFWAT1y8yMWzZzn/9tv0vfsudqeTlKIQHRvL2QSOAqU7d/K1
W2/l2muvpba2Nl3iJhIJIxKJMDw8TO/Zs3zw4x+jTUzkHVCw2fCUleHw+QiuXYu/upqqhgb8VVUY
hkFNczPFweCs7z556y3r+vyxYwiiyD+PHwdBwNB1krEYhqYRC4UwdB2D9LJmmnRtXHz77VTfeCPL
ly+ntraWmpqa9MaaqqqGoiiEQiGGhoYYHBzk00OHiHR2LkqDNlnG7nJhd7nQswqN+ZogiuipFMlY
DDWRgBl5KEF6woqAUFFByc03U756NZWVlSxfvpyamhoCgQAejwdB13VDVVVisRihUIiRkRGGh4cZ
+sc/GO3qIt7VhZGJAle7mVpWuVKDCBng8jXX4N68Gc8111BUVER5eTnV1dVUV1cTCASuFFqGYaDr
upFKpYjH40xMTDA+Ps7Y2Bjj4+OEw2EmenqYPneO1OAgZCWpxTY9S9QMcLhyHGUDRI8HqbISZ309
zoYGnE4nHo8Hv99PaWkpgUCAiooK6/DE4XCkI1HW2a6hqiqKojA9Pc3k5KQl4XCY6elpotFoetuk
v59UKIR66RLToRBJTcMYHUWIRvMSMPO6rbY2/SsIOFatQpYk5NWrkSQJWZZxOBy43W68Xi9+v5+i
oiJKSkooLi7G5/PhdruRZdnads85HzD3hpLJJIqiEIvFiEajRCIRIpEI0WiUWCxGPB631iOKopBK
pazUbi4Oc7Y+snKCzWbDZrMhSRKSJGG3262w6HK5cLlceDwevF4vPp8Pr9eL2+3G5XLhcDjMkxuE
TNibdcBhnv9qmkYqlSKZTJJIJFAUhXg8jqIoKIpCIpGwSCSTSVRVtWSuk3qTiAnebrfngHc4HDid
TpxOJy6Xy7qWZdk6wZzrxCbvCY1JRNd1NE3LAWgSM69ngp+PgCiK1gF3NhFTzHvmO+Y3wCzw8xLI
JpL5tYCZpEyw2f/nWt1ag2UtM0x3Mq1iAp1j+bH0M7KFCGWDnfmbd8AMnuxfUzL/C0rp/wFnFd4n
EQn3XQAAAABJRU5ErkJggg==">
</td>
<td class="content">The redundancy of the <span class="monospaced">special</span> device should match the one of the
pool, since the <span class="monospaced">special</span> device is a point of failure for the whole pool.</td>
</tr></tbody></table>
</div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">Adding a <span class="monospaced">special</span> device to a pool cannot be undone!</td>
</tr></tbody></table>
</div>
<div class="listingblock">
<div class="title">Create a pool with <span class="monospaced">special</span> device and RAID-1:</div>
<div class="content monospaced">
<pre># zpool create -f -o ashift=12 &lt;pool&gt; mirror &lt;device1&gt; &lt;device2&gt; special mirror &lt;device3&gt; &lt;device4&gt;</pre>
</div></div>
<div class="listingblock">
<div class="title">Add a <span class="monospaced">special</span> device to an existing pool with RAID-1:</div>
<div class="content monospaced">
<pre># zpool add &lt;pool&gt; special mirror &lt;device1&gt; &lt;device2&gt;</pre>
</div></div>
<div class="paragraph">
<p>ZFS datasets expose the <span class="monospaced">special_small_blocks=&lt;size&gt;</span> property. <span class="monospaced">size</span> can be
<span class="monospaced">0</span> to disable storing small file blocks on the <span class="monospaced">special</span> device or a power of
two in the range between <span class="monospaced">512B</span> to <span class="monospaced">1M</span>. After setting the property new file
blocks smaller than <span class="monospaced">size</span> will be allocated on the <span class="monospaced">special</span> device.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Important" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAALa0lEQVRogdWZa2wc1RXHfzM7O/te
P9e1vXHSmEdjx3YeDkkaF6REKRRCEDSEFNkRjdSWSsgC2iqoRWqLQKiqIBg1NOQDiMeHtkQIQkRR
S9S4aWwgCQoUgl232KkT28J21l6vd3d2dx79sDuTXdtre03Uqlc62tl53Pv/n3PuOffcKxiGwf9z
E//XAL5sk65WR0a6AaDrOjMtKwgCoigiCAKCIAhXa9wvTcAEHhoc5KMjR7jw3nucf+MN67k/GCS4
YQOr77yTpl27kGUZm81mXC0ywlLnQDbwEwcO0P2b3yz4jV5RwZq2Nm5/5BH8fj+SJGGz2b4UiSUR
MAzD0DSNEwcO8MdHHln8d0AcsAUCfOe551h7yy04HA7sdvuSrVEwAV3XjdDgIC/v2cOl06dznqWA
JCAAMvn9MwlEgF1PP8037rsPt9uN3W5fkjUKIqDrujF+4QLP3XQTU0NDOc8UIAa4y8sJtrQQqK/H
7XQSfv99Pn/nHQRRxNB16/0EEAUatm9n76uv4vP5cDgcBZNYNIF84I0MEK20lIZvf5sNe/dSWlqK
1+vFxBEZHubT3/+evqNHmR4dJRWP55BevW0bra+8gt/vx+l0IknSokksikA+8ElgClixYwc3799P
ZWUlTqczbz+x0VFOPvEEQ6dPE/niCwxdt0jUbd1K68svU1xcXBCJBQnMBz5eWkrL/v2s++Y3KSsr
QxQXlxc7f/5zBjo7CQ8NoadSKKQn943t7dyyf79FYjHuNC8BwzAMVVV56Z57+OzNN3PAJ8vK+FZH
B2s2b55X6/nahRMnOP7TnxLOKEXJyO2/+hVf37uXoqIinE6nmfzyksirMjNUvvGjH+WAVwG1vJxb
Dhxg1bp1uFwuMwQWJCu3bWPN3r24SkoAcGbA/OXXv2bws8+IRqOkUqlZGX1RBAzDMHRd51+nTvHe
wYNX7gPTQMO+faxav56SkpIlgTdlU3s7TXv2IGUs6AGioRDH2tuZmJggFouhqirGPCzyEUBVVX7X
1pZzPwI0fP/7bNmzh7KysrzADh8+zP3338++ffs4ePBg3vdEUWTDD3/IsuZmRLsdW4bEpd5eOp99
lnA4jKIoaJqWl8QsAqb2Txw4kDNp40BxXR0bWlspLy/PC+qZZ55hdHQUSZIYHByko6ODxx57bBZw
cy3kq6xk6y9/ib+qCgQBmbQ7nT50iIs9PUSjUZLJJIZhzEliLgKMDQzwp5/9zLqnkQ51W3/xCwKB
AE6nMy+B3t5exsfH6ezspLe3l3g8zuHDh2cBz5bKpibWtrXhLi0FwA3owNsPP8zExATxeBxVVecy
QC4Bc+L+taNjluvU33031dddN6/2BUGgpKSE/v5+JicncweaA3i2tDz8MIHrr0eU0gsQBzDW10ff
yZNEIhESicScrpRDQNd1xgYG+OC3v7XuJQBXIMDWn/yEioqKeUGMjY0hCAKKoszS1MjIyLzfAmx/
4gmKli0DQcBJOmicfOopJicnicfjJoG5LWBq/+Szz+a8oABNe/bgdrvndR1BEKiqqiIejxOLxWYR
CAaDC0alqjVrqGxowFNWhkDalUKff07f3/5mWSFTLFksLAK6rjN+4UKO9lXAUV7O2tbWeaNOthZl
WZ4FHlh0aN3y4IPIHk+6L8AGvNfRQTgctuZCthVEU/u6rnPutddyBk0A9bt3W4us+QY2fbyurg6v
15vTzw033LBoAsHmZoLNzdgcDgBcQKi/n391dTE9PU0ymcyxgkmAVCrF+4cO5WjfFgiwfoGwOTOy
lJWVUVtba1mkUAKCILClvR1veTkA9owVPnzllZy8YFnA1H5/dzfTw8PWgziwcutWfD4fsiwvCNyU
xsZG3G43lZWVVl933XVXQQSq161j+ebNlhXcwBdnzxIOh60lhmkFEUDTNP7++uvWgBrp6mrNHBk3
H/BsAtFoFEdm8GAwyMaNGwsiIAgCjbt346uosKwgAp8cPWolNj1THInmsqHn6FGLQApY1thIxcqV
+Hy+WX6+0ETesWMHLpcLWZZ54IEHCgYvCAKrbrsNyem0JrQEXMzkhKzlBaKu61z+97+ZHhnJmbwr
tm2zJu5CWp8pra2tNDU1sXPnTtrb25dEQBAEvrplC57MXJCB8XPniEQixONxa6Uq6brOQHd3jvvo
QP2ttxZUpMxsL7zwwpK+AyxLXrt9Oxe6uxFEEXvGZc4fO0bVD35AMplM1wuapjGYtbugAp5AgKLq
aquu/W/JTEs37dqFqihWzSADk8PD1jzQNC3tQtmrThUINDbm+H4h0tfXx6OPPsodd9xBW1sbPT09
BQPPluu2b7fWRxIQ6usjFotZBCRN06wZbbZAfT1FRUWWKQtpzz//PMPDw/T09DA1NcXo6CjHjx+f
11Xma95AwHJjEdB0HUVRrhCYuRGrAiXBoFUqFtq6u7sZHx9nJBMUurq65uxnsX0Hm5v56A9/ANIJ
LXrpEolEwgqlUqZQyPlo6OOPlzx5ly1bxnBWQmxpackBW6hSBEC02axrpqasNZGmaek8kN0kYPDj
j1FVdUlz4Mknn6Surs4Cb5aUhYZiUy59+GGOgm3V1WiadiUPCILAV9autV5wAaM9PXQeOWJprBBp
aGjgxIkTpFIpOjs7aWxsXBJwUwZOnUKZmgLS2zl4vUjSlV1XURRFqtavv3IDKAVe/O53efPxx69a
SCxUBs+c4dV77+Vyfz+xy5eBdG3iX7ECm81m9S2JokiwuZmiujrCPT1AuqgOAqcef5zzL77Imt27
uemhhyipqVnYZ5cw8bPbR0eOcOall7g8MEB4aIhEJAJk9qOAwKZNyLJsHpQgRKNRIxQK0XPmDH/+
3vdQQ6G8na/YtIk199zDypYWVmzceFWAT1y8yMWzZzn/9tv0vfsudqeTlKIQHRvL2QSOAqU7d/K1
W2/l2muvpba2Nl3iJhIJIxKJMDw8TO/Zs3zw4x+jTUzkHVCw2fCUleHw+QiuXYu/upqqhgb8VVUY
hkFNczPFweCs7z556y3r+vyxYwiiyD+PHwdBwNB1krEYhqYRC4UwdB2D9LJmmnRtXHz77VTfeCPL
ly+ntraWmpqa9MaaqqqGoiiEQiGGhoYYHBzk00OHiHR2LkqDNlnG7nJhd7nQswqN+ZogiuipFMlY
DDWRgBl5KEF6woqAUFFByc03U756NZWVlSxfvpyamhoCgQAejwdB13VDVVVisRihUIiRkRGGh4cZ
+sc/GO3qIt7VhZGJAle7mVpWuVKDCBng8jXX4N68Gc8111BUVER5eTnV1dVUV1cTCASuFFqGYaDr
upFKpYjH40xMTDA+Ps7Y2Bjj4+OEw2EmenqYPneO1OAgZCWpxTY9S9QMcLhyHGUDRI8HqbISZ309
zoYGnE4nHo8Hv99PaWkpgUCAiooK6/DE4XCkI1HW2a6hqiqKojA9Pc3k5KQl4XCY6elpotFoetuk
v59UKIR66RLToRBJTcMYHUWIRvMSMPO6rbY2/SsIOFatQpYk5NWrkSQJWZZxOBy43W68Xi9+v5+i
oiJKSkooLi7G5/PhdruRZdnads85HzD3hpLJJIqiEIvFiEajRCIRIpEI0WiUWCxGPB631iOKopBK
pazUbi4Oc7Y+snKCzWbDZrMhSRKSJGG3262w6HK5cLlceDwevF4vPp8Pr9eL2+3G5XLhcDjMkxuE
TNibdcBhnv9qmkYqlSKZTJJIJFAUhXg8jqIoKIpCIpGwSCSTSVRVtWSuk3qTiAnebrfngHc4HDid
TpxOJy6Xy7qWZdk6wZzrxCbvCY1JRNd1NE3LAWgSM69ngp+PgCiK1gF3NhFTzHvmO+Y3wCzw8xLI
JpL5tYCZpEyw2f/nWt1ag2UtM0x3Mq1iAp1j+bH0M7KFCGWDnfmbd8AMnuxfUzL/C0rp/wFnFd4n
EQn3XQAAAABJRU5ErkJggg==">
</td>
<td class="content">If the value for <span class="monospaced">special_small_blocks</span> is greater than or equal to
the <span class="monospaced">recordsize</span> (default <span class="monospaced">128K</span>) of the dataset, <strong>all</strong> data will be written to
the <span class="monospaced">special</span> device, so be careful!</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>Setting the <span class="monospaced">special_small_blocks</span> property on a pool will change the default
value of that property for all child ZFS datasets (for example all containers
in the pool will opt in for small file blocks).</p></div>
<div class="listingblock">
<div class="title">Opt in for all file smaller than 4K-blocks pool-wide:</div>
<div class="content monospaced">
<pre># zfs set special_small_blocks=4K &lt;pool&gt;</pre>
</div></div>
<div class="listingblock">
<div class="title">Opt in for small file blocks for a single dataset:</div>
<div class="content monospaced">
<pre># zfs set special_small_blocks=4K &lt;pool&gt;/&lt;filesystem&gt;</pre>
</div></div>
<div class="listingblock">
<div class="title">Opt out from small file blocks for a single dataset:</div>
<div class="content monospaced">
<pre># zfs set special_small_blocks=0 &lt;pool&gt;/&lt;filesystem&gt;</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="sysadmin_zfs_features">3.9.13. ZFS Pool Features
 <a class="headerlink" href="#sysadmin_zfs_features" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Changes to the on-disk format in ZFS are only made between major version changes
and are specified through <strong>features</strong>. All features, as well as the general
mechanism are well documented in the <span class="monospaced">zpool-features(5)</span> manpage.</p></div>
<div class="paragraph">
<p>Since enabling new features can render a pool not importable by an older version
of ZFS, this needs to be done actively by the administrator, by running
<span class="monospaced">zpool upgrade</span> on the pool (see the <span class="monospaced">zpool-upgrade(8)</span> manpage).</p></div>
<div class="paragraph">
<p>Unless you need to use one of the new features, there is no upside to enabling
them.</p></div>
<div class="paragraph">
<p>In fact, there are some downsides to enabling new features:</p></div>
<div class="ulist"><ul>
<li>
<p>
A system with root on ZFS, that still boots using GRUB will become
  unbootable if a new feature is active on the rpool, due to the incompatible
  implementation of ZFS in GRUB.
</p>
</li>
<li>
<p>
The system will not be able to import any upgraded pool when booted with an
  older kernel, which still ships with the old ZFS modules.
</p>
</li>
<li>
<p>
Booting an older Proxmox VE ISO to repair a non-booting system will likewise not
  work.
</p>
</li>
</ul></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Important" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAALa0lEQVRogdWZa2wc1RXHfzM7O/te
P9e1vXHSmEdjx3YeDkkaF6REKRRCEDSEFNkRjdSWSsgC2iqoRWqLQKiqIBg1NOQDiMeHtkQIQkRR
S9S4aWwgCQoUgl232KkT28J21l6vd3d2dx79sDuTXdtre03Uqlc62tl53Pv/n3PuOffcKxiGwf9z
E//XAL5sk65WR0a6AaDrOjMtKwgCoigiCAKCIAhXa9wvTcAEHhoc5KMjR7jw3nucf+MN67k/GCS4
YQOr77yTpl27kGUZm81mXC0ywlLnQDbwEwcO0P2b3yz4jV5RwZq2Nm5/5BH8fj+SJGGz2b4UiSUR
MAzD0DSNEwcO8MdHHln8d0AcsAUCfOe551h7yy04HA7sdvuSrVEwAV3XjdDgIC/v2cOl06dznqWA
JCAAMvn9MwlEgF1PP8037rsPt9uN3W5fkjUKIqDrujF+4QLP3XQTU0NDOc8UIAa4y8sJtrQQqK/H
7XQSfv99Pn/nHQRRxNB16/0EEAUatm9n76uv4vP5cDgcBZNYNIF84I0MEK20lIZvf5sNe/dSWlqK
1+vFxBEZHubT3/+evqNHmR4dJRWP55BevW0bra+8gt/vx+l0IknSokksikA+8ElgClixYwc3799P
ZWUlTqczbz+x0VFOPvEEQ6dPE/niCwxdt0jUbd1K68svU1xcXBCJBQnMBz5eWkrL/v2s++Y3KSsr
QxQXlxc7f/5zBjo7CQ8NoadSKKQn943t7dyyf79FYjHuNC8BwzAMVVV56Z57+OzNN3PAJ8vK+FZH
B2s2b55X6/nahRMnOP7TnxLOKEXJyO2/+hVf37uXoqIinE6nmfzyksirMjNUvvGjH+WAVwG1vJxb
Dhxg1bp1uFwuMwQWJCu3bWPN3r24SkoAcGbA/OXXv2bws8+IRqOkUqlZGX1RBAzDMHRd51+nTvHe
wYNX7gPTQMO+faxav56SkpIlgTdlU3s7TXv2IGUs6AGioRDH2tuZmJggFouhqirGPCzyEUBVVX7X
1pZzPwI0fP/7bNmzh7KysrzADh8+zP3338++ffs4ePBg3vdEUWTDD3/IsuZmRLsdW4bEpd5eOp99
lnA4jKIoaJqWl8QsAqb2Txw4kDNp40BxXR0bWlspLy/PC+qZZ55hdHQUSZIYHByko6ODxx57bBZw
cy3kq6xk6y9/ib+qCgQBmbQ7nT50iIs9PUSjUZLJJIZhzEliLgKMDQzwp5/9zLqnkQ51W3/xCwKB
AE6nMy+B3t5exsfH6ezspLe3l3g8zuHDh2cBz5bKpibWtrXhLi0FwA3owNsPP8zExATxeBxVVecy
QC4Bc+L+taNjluvU33031dddN6/2BUGgpKSE/v5+JicncweaA3i2tDz8MIHrr0eU0gsQBzDW10ff
yZNEIhESicScrpRDQNd1xgYG+OC3v7XuJQBXIMDWn/yEioqKeUGMjY0hCAKKoszS1MjIyLzfAmx/
4gmKli0DQcBJOmicfOopJicnicfjJoG5LWBq/+Szz+a8oABNe/bgdrvndR1BEKiqqiIejxOLxWYR
CAaDC0alqjVrqGxowFNWhkDalUKff07f3/5mWSFTLFksLAK6rjN+4UKO9lXAUV7O2tbWeaNOthZl
WZ4FHlh0aN3y4IPIHk+6L8AGvNfRQTgctuZCthVEU/u6rnPutddyBk0A9bt3W4us+QY2fbyurg6v
15vTzw033LBoAsHmZoLNzdgcDgBcQKi/n391dTE9PU0ymcyxgkmAVCrF+4cO5WjfFgiwfoGwOTOy
lJWVUVtba1mkUAKCILClvR1veTkA9owVPnzllZy8YFnA1H5/dzfTw8PWgziwcutWfD4fsiwvCNyU
xsZG3G43lZWVVl933XVXQQSq161j+ebNlhXcwBdnzxIOh60lhmkFEUDTNP7++uvWgBrp6mrNHBk3
H/BsAtFoFEdm8GAwyMaNGwsiIAgCjbt346uosKwgAp8cPWolNj1THInmsqHn6FGLQApY1thIxcqV
+Hy+WX6+0ETesWMHLpcLWZZ54IEHCgYvCAKrbrsNyem0JrQEXMzkhKzlBaKu61z+97+ZHhnJmbwr
tm2zJu5CWp8pra2tNDU1sXPnTtrb25dEQBAEvrplC57MXJCB8XPniEQixONxa6Uq6brOQHd3jvvo
QP2ttxZUpMxsL7zwwpK+AyxLXrt9Oxe6uxFEEXvGZc4fO0bVD35AMplM1wuapjGYtbugAp5AgKLq
aquu/W/JTEs37dqFqihWzSADk8PD1jzQNC3tQtmrThUINDbm+H4h0tfXx6OPPsodd9xBW1sbPT09
BQPPluu2b7fWRxIQ6usjFotZBCRN06wZbbZAfT1FRUWWKQtpzz//PMPDw/T09DA1NcXo6CjHjx+f
11Xma95AwHJjEdB0HUVRrhCYuRGrAiXBoFUqFtq6u7sZHx9nJBMUurq65uxnsX0Hm5v56A9/ANIJ
LXrpEolEwgqlUqZQyPlo6OOPlzx5ly1bxnBWQmxpackBW6hSBEC02axrpqasNZGmaek8kN0kYPDj
j1FVdUlz4Mknn6Surs4Cb5aUhYZiUy59+GGOgm3V1WiadiUPCILAV9autV5wAaM9PXQeOWJprBBp
aGjgxIkTpFIpOjs7aWxsXBJwUwZOnUKZmgLS2zl4vUjSlV1XURRFqtavv3IDKAVe/O53efPxx69a
SCxUBs+c4dV77+Vyfz+xy5eBdG3iX7ECm81m9S2JokiwuZmiujrCPT1AuqgOAqcef5zzL77Imt27
uemhhyipqVnYZ5cw8bPbR0eOcOall7g8MEB4aIhEJAJk9qOAwKZNyLJsHpQgRKNRIxQK0XPmDH/+
3vdQQ6G8na/YtIk199zDypYWVmzceFWAT1y8yMWzZzn/9tv0vfsudqeTlKIQHRvL2QSOAqU7d/K1
W2/l2muvpba2Nl3iJhIJIxKJMDw8TO/Zs3zw4x+jTUzkHVCw2fCUleHw+QiuXYu/upqqhgb8VVUY
hkFNczPFweCs7z556y3r+vyxYwiiyD+PHwdBwNB1krEYhqYRC4UwdB2D9LJmmnRtXHz77VTfeCPL
ly+ntraWmpqa9MaaqqqGoiiEQiGGhoYYHBzk00OHiHR2LkqDNlnG7nJhd7nQswqN+ZogiuipFMlY
DDWRgBl5KEF6woqAUFFByc03U756NZWVlSxfvpyamhoCgQAejwdB13VDVVVisRihUIiRkRGGh4cZ
+sc/GO3qIt7VhZGJAle7mVpWuVKDCBng8jXX4N68Gc8111BUVER5eTnV1dVUV1cTCASuFFqGYaDr
upFKpYjH40xMTDA+Ps7Y2Bjj4+OEw2EmenqYPneO1OAgZCWpxTY9S9QMcLhyHGUDRI8HqbISZ309
zoYGnE4nHo8Hv99PaWkpgUCAiooK6/DE4XCkI1HW2a6hqiqKojA9Pc3k5KQl4XCY6elpotFoetuk
v59UKIR66RLToRBJTcMYHUWIRvMSMPO6rbY2/SsIOFatQpYk5NWrkSQJWZZxOBy43W68Xi9+v5+i
oiJKSkooLi7G5/PhdruRZdnads85HzD3hpLJJIqiEIvFiEajRCIRIpEI0WiUWCxGPB631iOKopBK
pazUbi4Oc7Y+snKCzWbDZrMhSRKSJGG3262w6HK5cLlceDwevF4vPp8Pr9eL2+3G5XLhcDjMkxuE
TNibdcBhnv9qmkYqlSKZTJJIJFAUhXg8jqIoKIpCIpGwSCSTSVRVtWSuk3qTiAnebrfngHc4HDid
TpxOJy6Xy7qWZdk6wZzrxCbvCY1JRNd1NE3LAWgSM69ngp+PgCiK1gF3NhFTzHvmO+Y3wCzw8xLI
JpL5tYCZpEyw2f/nWt1ag2UtM0x3Mq1iAp1j+bH0M7KFCGWDnfmbd8AMnuxfUzL/C0rp/wFnFd4n
EQn3XQAAAABJRU5ErkJggg==">
</td>
<td class="content">Do <strong>not</strong> upgrade your rpool if your system is still booted with
GRUB, as this will render your system unbootable. This includes systems
installed before Proxmox VE 5.4, and systems booting with legacy BIOS boot (see
<a href="#sysboot_determine_bootloader_used">how to determine the bootloader</a>).</td>
</tr></tbody></table>
</div>
<div class="listingblock">
<div class="title">Enable new features for a ZFS pool:</div>
<div class="content monospaced">
<pre># zpool upgrade &lt;pool&gt;</pre>
</div></div>
</div>
</div>
<div class="sect2">
<h3 id="chapter_btrfs">
<span>3.10. BTRFS</span>
 <a class="headerlink" href="#chapter_btrfs" title="Permalink to this heading"></a>
</h3>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">BTRFS integration is currently a <strong>technology preview</strong> in Proxmox VE.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>BTRFS is a modern copy on write file system natively supported by the Linux
kernel, implementing features such as snapshots, built-in RAID and self healing
via checksums for data and metadata. Starting with Proxmox VE 7.0, BTRFS is
introduced as optional selection for the root file system.</p></div>
<div class="ulist"><div class="title">General BTRFS advantages</div><ul>
<li>
<p>
Main system setup almost identical to the traditional ext4 based setup
</p>
</li>
<li>
<p>
Snapshots
</p>
</li>
<li>
<p>
Data compression on file system level
</p>
</li>
<li>
<p>
Copy-on-write clone
</p>
</li>
<li>
<p>
RAID0, RAID1 and RAID10
</p>
</li>
<li>
<p>
Protection against data corruption
</p>
</li>
<li>
<p>
Self healing
</p>
</li>
<li>
<p>
natively supported by the Linux kernel
</p>
</li>
<li>
<p>
…
</p>
</li>
</ul></div>
<div class="ulist"><div class="title">Caveats</div><ul>
<li>
<p>
RAID levels 5/6 are experimental and dangerous
</p>
</li>
</ul></div>
<div class="sect3">
<h4 id="_installation_as_root_file_system_2">3.10.1. Installation as Root File System
 <a class="headerlink" href="#_installation_as_root_file_system_2" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>When you install using the Proxmox VE installer, you can choose BTRFS for the root
file system. You need to select the RAID type at installation time:</p></div>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
RAID0
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Also called “striping”. The capacity of such volume is the sum
of the capacities of all disks. But RAID0 does not add any redundancy,
so the failure of a single drive makes the volume unusable.
</p>
</td>
</tr>
<tr>
<td class="hdlist1">
RAID1
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Also called “mirroring”. Data is written identically to all
disks. This mode requires at least 2 disks with the same size. The
resulting capacity is that of a single disk.
</p>
</td>
</tr>
<tr>
<td class="hdlist1">
RAID10
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
A combination of RAID0 and RAID1. Requires at least 4 disks.
</p>
</td>
</tr>
</tbody></table></div>
<div class="paragraph">
<p>The installer automatically partitions the disks and creates an additional
subvolume at <span class="monospaced">/var/lib/pve/local-btrfs</span>.  In order to use that with the Proxmox VE
tools, the installer creates the following configuration entry in
<span class="monospaced">/etc/pve/storage.cfg</span>:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>dir: local
        path /var/lib/vz
        content iso,vztmpl,backup
        disable

btrfs: local-btrfs
        path /var/lib/pve/local-btrfs
        content iso,vztmpl,backup,images,rootdir</pre>
</div></div>
<div class="paragraph">
<p>This explicitly disables the default <span class="monospaced">local</span> storage in favor of a BTRFS
specific storage entry on the additional subvolume.</p></div>
<div class="paragraph">
<p>The <span class="monospaced">btrfs</span> command is used to configure and manage the BTRFS file system,
After the installation, the following command lists all additional subvolumes:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># btrfs subvolume list /
ID 256 gen 6 top level 5 path var/lib/pve/local-btrfs</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_btrfs_administration">3.10.2. BTRFS Administration
 <a class="headerlink" href="#_btrfs_administration" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>This section gives you some usage examples for common tasks.</p></div>
<div class="sect4">
<h5 id="_creating_a_btrfs_file_system">Creating a BTRFS file system
 <a class="headerlink" href="#_creating_a_btrfs_file_system" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>To create BTRFS file systems, <span class="monospaced">mkfs.btrfs</span> is used. The <span class="monospaced">-d</span> and <span class="monospaced">-m</span> parameters
are used to set the profile for metadata and data respectively. With the
optional <span class="monospaced">-L</span> parameter, a label can be set.</p></div>
<div class="paragraph">
<p>Generally, the following modes are supported: <span class="monospaced">single</span>, <span class="monospaced">raid0</span>, <span class="monospaced">raid1</span>,
<span class="monospaced">raid10</span>.</p></div>
<div class="paragraph">
<p>Create a BTRFS file system on a single disk <span class="monospaced">/dev/sdb</span> with the label
<span class="monospaced">My-Storage</span>:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre> # mkfs.btrfs -m single -d single -L My-Storage /dev/sdb</pre>
</div></div>
<div class="paragraph">
<p>Or create a RAID1 on the two partitions <span class="monospaced">/dev/sdb1</span> and <span class="monospaced">/dev/sdc1</span>:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre> # mkfs.btrfs -m raid1 -d raid1 -L My-Storage /dev/sdb1 /dev/sdc1</pre>
</div></div>
</div>
<div class="sect4">
<h5 id="_mounting_a_btrfs_file_system">Mounting a BTRFS file system
 <a class="headerlink" href="#_mounting_a_btrfs_file_system" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>The new file-system can then be mounted either manually, for example:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre> # mkdir /my-storage
 # mount /dev/sdb /my-storage</pre>
</div></div>
<div class="paragraph">
<p>A BTRFS can also be added to <span class="monospaced">/etc/fstab</span> like any other mount point,
automatically mounting it on boot. It’s recommended to avoid  using
block-device paths but use the <span class="monospaced">UUID</span> value the <span class="monospaced">mkfs.btrfs</span> command printed,
especially there is more than one disk in a BTRFS setup.</p></div>
<div class="paragraph">
<p>For example:</p></div>
<div class="listingblock">
<div class="title">File <span class="monospaced">/etc/fstab</span></div>
<div class="content monospaced">
<pre># ... other mount points left out for brevity

# using the UUID from the mkfs.btrfs output is highly recommended
UUID=e2c0c3ff-2114-4f54-b767-3a203e49f6f3 /my-storage btrfs defaults 0 0</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">If you do not have the UUID available anymore you can use the <span class="monospaced">blkid</span> tool
 to list all properties of block-devices.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>Afterwards you can trigger the first mount by executing:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>mount /my-storage</pre>
</div></div>
<div class="paragraph">
<p>After the next reboot this will be automatically done by the system at boot.</p></div>
</div>
<div class="sect4">
<h5 id="_adding_a_btrfs_file_system_to_proxmox_ve">Adding a BTRFS file system to Proxmox VE
 <a class="headerlink" href="#_adding_a_btrfs_file_system_to_proxmox_ve" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>You can add an existing BTRFS file system to Proxmox VE via the web interface, or
using the CLI, for example:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>pvesm add btrfs my-storage --path /my-storage</pre>
</div></div>
</div>
<div class="sect4">
<h5 id="_creating_a_subvolume">Creating a subvolume
 <a class="headerlink" href="#_creating_a_subvolume" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Creating a subvolume links it to a path in the BTRFS file system, where it will
appear as a regular directory.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># btrfs subvolume create /some/path</pre>
</div></div>
<div class="paragraph">
<p>Afterwards <span class="monospaced">/some/path</span> will act like a regular directory.</p></div>
</div>
<div class="sect4">
<h5 id="_deleting_a_subvolume">Deleting a subvolume
 <a class="headerlink" href="#_deleting_a_subvolume" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Contrary to directories removed via <span class="monospaced">rmdir</span>, subvolumes do not need to be empty
in order to be deleted via the <span class="monospaced">btrfs</span> command.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># btrfs subvolume delete /some/path</pre>
</div></div>
</div>
<div class="sect4">
<h5 id="_creating_a_snapshot_of_a_subvolume">Creating a snapshot of a subvolume
 <a class="headerlink" href="#_creating_a_snapshot_of_a_subvolume" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>BTRFS does not actually distinguish between snapshots and normal subvolumes, so
taking a snapshot can also be seen as creating an arbitrary copy of a subvolume.
By convention, Proxmox VE will use the read-only flag when creating snapshots of
guest disks or subvolumes, but this flag can also be changed later on.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># btrfs subvolume snapshot -r /some/path /a/new/path</pre>
</div></div>
<div class="paragraph">
<p>This will create a read-only "clone" of the subvolume on <span class="monospaced">/some/path</span> at
<span class="monospaced">/a/new/path</span>. Any future modifications to <span class="monospaced">/some/path</span> cause the modified data
to be copied before modification.</p></div>
<div class="paragraph">
<p>If the read-only (<span class="monospaced">-r</span>) option is left out, both subvolumes will be writable.</p></div>
</div>
<div class="sect4">
<h5 id="_enabling_compression">Enabling compression
 <a class="headerlink" href="#_enabling_compression" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>By default, BTRFS does not compress data. To enable compression, the <span class="monospaced">compress</span>
mount option can be added. Note that data already written will not be compressed
after the fact.</p></div>
<div class="paragraph">
<p>By default, the rootfs will be listed in <span class="monospaced">/etc/fstab</span> as follows:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>UUID=&lt;uuid of your root file system&gt; / btrfs defaults 0 1</pre>
</div></div>
<div class="paragraph">
<p>You can simply append <span class="monospaced">compress=zstd</span>, <span class="monospaced">compress=lzo</span>, or <span class="monospaced">compress=zlib</span> to the
<span class="monospaced">defaults</span> above like so:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>UUID=&lt;uuid of your root file system&gt; / btrfs defaults,compress=zstd 0 1</pre>
</div></div>
<div class="paragraph">
<p>This change will take effect after rebooting.</p></div>
</div>
<div class="sect4">
<h5 id="_checking_space_usage">Checking Space Usage
 <a class="headerlink" href="#_checking_space_usage" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>The classic <span class="monospaced">df</span> tool may output confusing values for some BTRFS setups.
For a better estimate use the <span class="monospaced">btrfs filesystem usage /PATH</span> command, for example:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># btrfs fi usage /my-storage</pre>
</div></div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="proxmox_node_management">
<span>3.11. Proxmox Node Management</span>
 <a class="headerlink" href="#proxmox_node_management" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>The Proxmox VE node management tool (<span class="monospaced">pvenode</span>) allows you to control node specific
settings and resources.</p></div>
<div class="paragraph">
<p>Currently <span class="monospaced">pvenode</span> allows you to set a node’s description, run various
bulk operations on the node’s guests, view the node’s task history, and
manage the node’s SSL certificates, which are used for the API and the web GUI
through <span class="monospaced">pveproxy</span>.</p></div>
<div class="sect3">
<h4 id="_wake_on_lan">3.11.1. Wake-on-LAN
 <a class="headerlink" href="#_wake_on_lan" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Wake-on-LAN (WoL) allows you to switch on a sleeping computer in the network, by
sending a magic packet. At least one NIC must support this feature, and the
respective option needs to be enabled in the computer’s firmware (BIOS/UEFI)
configuration. The option name can vary from <em>Enable Wake-on-Lan</em> to
<em>Power On By PCIE Device</em>; check your motherboard’s vendor manual, if you’re
unsure. <span class="monospaced">ethtool</span> can be used to check the WoL configuration of <span class="monospaced">&lt;interface&gt;</span>
by running:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>ethtool &lt;interface&gt; | grep Wake-on</pre>
</div></div>
<div class="paragraph">
<p><span class="monospaced">pvenode</span> allows you to wake sleeping members of a cluster via WoL, using the
command:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>pvenode wakeonlan &lt;node&gt;</pre>
</div></div>
<div class="paragraph">
<p>This broadcasts the WoL magic packet on UDP port 9, containing the MAC address
of <span class="monospaced">&lt;node&gt;</span> obtained from the <span class="monospaced">wakeonlan</span> property. The node-specific
<span class="monospaced">wakeonlan</span> property can be set using the following command:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>pvenode config set -wakeonlan XX:XX:XX:XX:XX:XX</pre>
</div></div>
<div class="paragraph">
<p>The interface via which to send the WoL packet is determined from the default
route. It can be overwritten by setting the <span class="monospaced">bind-interface</span> via the following
command:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>pvenode config set -wakeonlan XX:XX:XX:XX:XX:XX,bind-interface=&lt;iface-name&gt;</pre>
</div></div>
<div class="paragraph">
<p>The broadcast address (default <span class="monospaced">255.255.255.255</span>) used when sending the WoL
packet can further be changed by setting the <span class="monospaced">broadcast-address</span> explicitly
using the following command:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>pvenode config set -wakeonlan XX:XX:XX:XX:XX:XX,broadcast-address=&lt;broadcast-address&gt;</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_task_history">3.11.2. Task History
 <a class="headerlink" href="#_task_history" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>When troubleshooting server issues, for example, failed backup jobs, it can
often be helpful to have a log of the previously run tasks. With Proxmox VE, you can
access the nodes’s task history through the <span class="monospaced">pvenode task</span> command.</p></div>
<div class="paragraph">
<p>You can get a filtered list of a node’s finished tasks with the <span class="monospaced">list</span>
subcommand. For example, to get a list of tasks related to VM <em>100</em>
that ended with an error, the command would be:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>pvenode task list --errors --vmid 100</pre>
</div></div>
<div class="paragraph">
<p>The log of a task can then be printed using its UPID:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>pvenode task log UPID:pve1:00010D94:001CA6EA:6124E1B9:vzdump:100:root@pam:</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_bulk_guest_power_management">3.11.3. Bulk Guest Power Management
 <a class="headerlink" href="#_bulk_guest_power_management" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>In case you have many VMs/containers, starting and stopping guests can be
carried out in bulk operations with the <span class="monospaced">startall</span> and <span class="monospaced">stopall</span> subcommands of
<span class="monospaced">pvenode</span>.  By default, <span class="monospaced">pvenode startall</span> will only start VMs/containers which
have been set to automatically start on boot (see
<a href="#qm_startup_and_shutdown">Automatic Start and Shutdown of Virtual Machines</a>),
however, you can override this behavior with the <span class="monospaced">--force</span> flag. Both commands
also have a <span class="monospaced">--vms</span> option, which limits the stopped/started guests to the
specified VMIDs.</p></div>
<div class="paragraph">
<p>For example, to start VMs <em>100</em>, <em>101</em>, and <em>102</em>, regardless of whether they
have <span class="monospaced">onboot</span> set, you can use:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>pvenode startall --vms 100,101,102 --force</pre>
</div></div>
<div class="paragraph">
<p>To stop these guests (and any other guests that may be running), use the
command:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>pvenode stopall</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">The stopall command first attempts to perform a clean shutdown and then
waits until either all guests have successfully shut down or an overridable
timeout (3 minutes by default) has expired. Once that happens and the
force-stop parameter is not explicitly set to 0 (false), all virtual guests
that are still running are hard stopped.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="first_guest_boot_delay">3.11.4. First Guest Boot Delay
 <a class="headerlink" href="#first_guest_boot_delay" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>In case your VMs/containers rely on slow-to-start external resources, for
example an NFS server, you can also set a per-node delay between the time Proxmox VE
boots and the time the first VM/container that is configured to autostart boots
(see <a href="#qm_startup_and_shutdown">Automatic Start and Shutdown of Virtual Machines</a>).</p></div>
<div class="paragraph">
<p>You can achieve this by setting the following (where <span class="monospaced">10</span> represents the delay
in seconds):</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>pvenode config set --startall-onboot-delay 10</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_bulk_guest_migration">3.11.5. Bulk Guest Migration
 <a class="headerlink" href="#_bulk_guest_migration" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>In case an upgrade situation requires you to migrate all of your guests from one
node to another, <span class="monospaced">pvenode</span> also offers the <span class="monospaced">migrateall</span> subcommand for bulk
migration. By default, this command will migrate every guest on the system to
the target node. It can however be set to only migrate a set of guests.</p></div>
<div class="paragraph">
<p>For example, to migrate VMs <em>100</em>, <em>101</em>, and <em>102</em>, to the node <em>pve2</em>, with
live-migration for local disks enabled, you can run:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>pvenode migrateall pve2 --vms 100,101,102 --with-local-disks</pre>
</div></div>
</div>
</div>
<div class="sect2">
<h3 id="sysadmin_certificate_management">
<span>3.12. Certificate Management</span>
 <a class="headerlink" href="#sysadmin_certificate_management" title="Permalink to this heading"></a>
</h3>
<div class="sect3">
<h4 id="_certificates_for_intra_cluster_communication">3.12.1. Certificates for Intra-Cluster Communication
 <a class="headerlink" href="#_certificates_for_intra_cluster_communication" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Each Proxmox VE cluster creates by default its own (self-signed) Certificate
Authority (CA) and generates a certificate for each node which gets signed by
the aforementioned CA. These certificates are used for encrypted communication
with the cluster’s <span class="monospaced">pveproxy</span> service and the Shell/Console feature if SPICE is
used.</p></div>
<div class="paragraph">
<p>The CA certificate and key are stored in the <a href="#chapter_pmxcfs">Proxmox Cluster File System (pmxcfs)</a>.</p></div>
</div>
<div class="sect3">
<h4 id="sysadmin_certs_api_gui">3.12.2. Certificates for API and Web GUI
 <a class="headerlink" href="#sysadmin_certs_api_gui" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The REST API and web GUI are provided by the <span class="monospaced">pveproxy</span> service, which runs on
each node.</p></div>
<div class="paragraph">
<p>You have the following options for the certificate used by <span class="monospaced">pveproxy</span>:</p></div>
<div class="olist arabic"><ol class="arabic">
<li>
<p>
By default the node-specific certificate in
<span class="monospaced">/etc/pve/nodes/NODENAME/pve-ssl.pem</span> is used. This certificate is signed by
the cluster CA and therefore not automatically trusted by browsers and
operating systems.
</p>
</li>
<li>
<p>
use an externally provided certificate (e.g. signed by a commercial CA).
</p>
</li>
<li>
<p>
use ACME (Let’s Encrypt) to get a trusted certificate with automatic
renewal, this is also integrated in the Proxmox VE API and web interface.
</p>
</li>
</ol></div>
<div class="paragraph">
<p>For options 2 and 3 the file <span class="monospaced">/etc/pve/local/pveproxy-ssl.pem</span> (and
<span class="monospaced">/etc/pve/local/pveproxy-ssl.key</span>, which needs to be without password) is used.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Keep in mind that <span class="monospaced">/etc/pve/local</span> is a node specific symlink to
<span class="monospaced">/etc/pve/nodes/NODENAME</span>.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>Certificates are managed with the Proxmox VE Node management command
(see the <span class="monospaced">pvenode(1)</span> manpage).</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">Do not replace or manually modify the automatically generated node
certificate files in <span class="monospaced">/etc/pve/local/pve-ssl.pem</span> and
<span class="monospaced">/etc/pve/local/pve-ssl.key</span> or the cluster CA files in
<span class="monospaced">/etc/pve/pve-root-ca.pem</span> and <span class="monospaced">/etc/pve/priv/pve-root-ca.key</span>.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="sysadmin_certs_upload_custom">3.12.3. Upload Custom Certificate
 <a class="headerlink" href="#sysadmin_certs_upload_custom" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>If you already have a certificate which you want to use for a Proxmox VE node you
can upload that certificate simply over the web interface.</p></div>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-node-certs-upload-custom.png">
<img src="images/screenshot/gui-node-certs-upload-custom.png" alt="screenshot/gui-node-certs-upload-custom.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>Note that the certificates key file, if provided, mustn’t be password
protected.</p></div>
</div>
<div class="sect3">
<h4 id="sysadmin_certs_get_trusted_acme_cert">3.12.4. Trusted certificates via Let’s Encrypt (ACME)
 <a class="headerlink" href="#sysadmin_certs_get_trusted_acme_cert" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Proxmox VE includes an implementation of the <strong>A</strong>utomatic <strong>C</strong>ertificate
<strong>M</strong>anagement <strong>E</strong>nvironment <strong>ACME</strong> protocol, allowing Proxmox VE admins to
use an ACME provider like Let’s Encrypt for easy setup of TLS certificates
which are accepted and trusted on modern operating systems and web browsers
out of the box.</p></div>
<div class="paragraph">
<p>Currently, the two ACME endpoints implemented are the
<a href="https://letsencrypt.org">Let’s Encrypt (LE)</a> production and its staging
environment. Our ACME client supports validation of <span class="monospaced">http-01</span> challenges using
a built-in web server and validation of <span class="monospaced">dns-01</span> challenges using a DNS plugin
supporting all the DNS API endpoints <a href="https://acme.sh">acme.sh</a> does.</p></div>
<div class="sect4">
<h5 id="sysadmin_certs_acme_account">ACME Account
 <a class="headerlink" href="#sysadmin_certs_acme_account" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-datacenter-acme-register-account.png">
<img src="images/screenshot/gui-datacenter-acme-register-account.png" alt="screenshot/gui-datacenter-acme-register-account.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>You need to register an ACME account per cluster with the endpoint you want to
use. The email address used for that account will serve as contact point for
renewal-due or similar notifications from the ACME endpoint.</p></div>
<div class="paragraph">
<p>You can register and deactivate ACME accounts over the web interface
<span class="monospaced">Datacenter -&gt; ACME</span> or using the <span class="monospaced">pvenode</span> command-line tool.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre> pvenode acme account register account-name [email protected]</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">Because of <a href="https://letsencrypt.org/docs/rate-limits/">rate-limits</a> you
should use LE <span class="monospaced">staging</span> for experiments or if you use ACME for the first time.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect4">
<h5 id="sysadmin_certs_acme_plugins">ACME Plugins
 <a class="headerlink" href="#sysadmin_certs_acme_plugins" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>The ACME plugins task is to provide automatic verification that you, and thus
the Proxmox VE cluster under your operation, are the real owner of a domain. This is
the basis building block for automatic certificate management.</p></div>
<div class="paragraph">
<p>The ACME protocol specifies different types of challenges, for example the
<span class="monospaced">http-01</span> where a web server provides a file with a certain content to prove
that it controls a domain. Sometimes this isn’t possible, either because of
technical limitations or if the address of a record to is not reachable from
the public internet. The <span class="monospaced">dns-01</span> challenge can be used in these cases.  This
challenge is fulfilled by creating a certain DNS record in the domain’s zone.</p></div>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-datacenter-acme-overview.png">
<img src="images/screenshot/gui-datacenter-acme-overview.png" alt="screenshot/gui-datacenter-acme-overview.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>Proxmox VE supports both of those challenge types out of the box, you can configure
plugins either over the web interface under <span class="monospaced">Datacenter -&gt; ACME</span>, or using the
<span class="monospaced">pvenode acme plugin add</span> command.</p></div>
<div class="paragraph">
<p>ACME Plugin configurations are stored in <span class="monospaced">/etc/pve/priv/acme/plugins.cfg</span>.
A plugin is available for all nodes in the cluster.</p></div>
</div>
<div class="sect4">
<h5 id="_node_domains">Node Domains
 <a class="headerlink" href="#_node_domains" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Each domain is node specific. You can add new or manage existing domain entries
under <span class="monospaced">Node -&gt; Certificates</span>, or using the <span class="monospaced">pvenode config</span> command.</p></div>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-node-certs-add-domain.png">
<img src="images/screenshot/gui-node-certs-add-domain.png" alt="screenshot/gui-node-certs-add-domain.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>After configuring the desired domain(s) for a node and ensuring that the
desired ACME account is selected, you can order your new certificate over the
web interface. On success the interface will reload after 10 seconds.</p></div>
<div class="paragraph">
<p>Renewal will happen <a href="#sysadmin_certs_acme_automatic_renewal">automatically</a>.</p></div>
</div>
</div>
<div class="sect3">
<h4 id="sysadmin_certs_acme_http_challenge">3.12.5. ACME HTTP Challenge Plugin
 <a class="headerlink" href="#sysadmin_certs_acme_http_challenge" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>There is always an implicitly configured <span class="monospaced">standalone</span> plugin for validating
<span class="monospaced">http-01</span> challenges via the built-in webserver spawned on port 80.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">The name <span class="monospaced">standalone</span> means that it can provide the validation on it’s
own, without any third party service. So, this plugin works also for cluster
nodes.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>There are a few prerequisites to use it for certificate management with Let’s
Encrypts ACME.</p></div>
<div class="ulist"><ul>
<li>
<p>
You have to accept the ToS of Let’s Encrypt to register an account.
</p>
</li>
<li>
<p>
<strong>Port 80</strong> of the node needs to be reachable from the internet.
</p>
</li>
<li>
<p>
There <strong>must</strong> be no other listener on port 80.
</p>
</li>
<li>
<p>
The requested (sub)domain needs to resolve to a public IP of the Node.
</p>
</li>
</ul></div>
</div>
<div class="sect3">
<h4 id="sysadmin_certs_acme_dns_challenge">3.12.6. ACME DNS API Challenge Plugin
 <a class="headerlink" href="#sysadmin_certs_acme_dns_challenge" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>On systems where external access for validation via the <span class="monospaced">http-01</span> method is
not possible or desired, it is possible to use the <span class="monospaced">dns-01</span> validation method.
This validation method requires a DNS server that allows provisioning of <span class="monospaced">TXT</span>
records via an API.</p></div>
<div class="sect4">
<h5 id="sysadmin_certs_acme_dns_api_config">Configuring ACME DNS APIs for validation
 <a class="headerlink" href="#sysadmin_certs_acme_dns_api_config" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Proxmox VE re-uses the DNS plugins developed for the <span class="monospaced">acme.sh</span>
<span class="footnote" data-note="acme.sh <a href=&quot;https://github.com/acmesh-official/acme.sh&quot;>https://github.com/acmesh-official/acme.sh</a>">[<a id="_footnoteref_6" href="#_footnote_6" title="View footnote" class="footnote">6</a>]</span> project, please
refer to its documentation for details on configuration of specific APIs.</p></div>
<div class="paragraph">
<p>The easiest way to configure a new plugin with the DNS API is using the web
interface (<span class="monospaced">Datacenter -&gt; ACME</span>).</p></div>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-datacenter-acme-add-dns-plugin.png">
<img src="images/screenshot/gui-datacenter-acme-add-dns-plugin.png" alt="screenshot/gui-datacenter-acme-add-dns-plugin.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>Choose <span class="monospaced">DNS</span> as challenge type. Then you can select your API provider, enter
the credential data to access your account over their API.
The validation delay determines the time in seconds between setting the DNS
record and prompting the ACME provider to validate it, as providers often need
some time to propagate the record in their infrastructure.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">See the acme.sh
<a href="https://github.com/acmesh-official/acme.sh/wiki/dnsapi#how-to-use-dns-api">How to use DNS API</a>
wiki for more detailed information about getting API credentials for your
provider.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>As there are many DNS providers and API endpoints Proxmox VE automatically generates
the form for the credentials for some providers. For the others you will see a
bigger text area, simply copy all the credentials <span class="monospaced">KEY</span>=<span class="monospaced">VALUE</span> pairs in there.</p></div>
</div>
<div class="sect4">
<h5 id="_dns_validation_through_cname_alias">DNS Validation through CNAME Alias
 <a class="headerlink" href="#_dns_validation_through_cname_alias" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>A special <span class="monospaced">alias</span> mode can be used to handle the validation on a different
domain/DNS server, in case your primary/real DNS does not support provisioning
via an API. Manually set up a permanent <span class="monospaced">CNAME</span> record for
<span class="monospaced">_acme-challenge.domain1.example</span> pointing to <span class="monospaced">_acme-challenge.domain2.example</span>
and set the <span class="monospaced">alias</span> property on the corresponding <span class="monospaced">acmedomainX</span> key in the
Proxmox VE node configuration file to <span class="monospaced">domain2.example</span> to allow the DNS server of
<span class="monospaced">domain2.example</span> to validate all challenges for <span class="monospaced">domain1.example</span>.</p></div>
</div>
<div class="sect4">
<h5 id="_combination_of_plugins">Combination of Plugins
 <a class="headerlink" href="#_combination_of_plugins" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Combining <span class="monospaced">http-01</span> and <span class="monospaced">dns-01</span> validation is possible in case your node is
reachable via multiple domains with different requirements / DNS provisioning
capabilities. Mixing DNS APIs from multiple providers or instances is also
possible by specifying different plugin instances per domain.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">Accessing the same service over multiple domains increases complexity and
should be avoided if possible.</td>
</tr></tbody></table>
</div>
</div>
</div>
<div class="sect3">
<h4 id="sysadmin_certs_acme_automatic_renewal">3.12.7. Automatic renewal of ACME certificates
 <a class="headerlink" href="#sysadmin_certs_acme_automatic_renewal" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>If a node has been successfully configured with an ACME-provided certificate
(either via pvenode or via the GUI), the certificate will be automatically
renewed by the <span class="monospaced">pve-daily-update.service</span>. Currently, renewal will be attempted
if the certificate has expired already, or will expire in the next 30 days.</p></div>
</div>
<div class="sect3">
<h4 id="_acme_examples_with_span_class_monospaced_pvenode_span">3.12.8. ACME Examples with <span class="monospaced">pvenode</span>
 <a class="headerlink" href="#_acme_examples_with_span_class_monospaced_pvenode_span" title="Permalink to this heading"></a>
</h4>
<div class="sect4">
<h5 id="_example_sample_span_class_monospaced_pvenode_span_invocation_for_using_let_8217_s_encrypt_certificates">Example: Sample <span class="monospaced">pvenode</span> invocation for using Let’s Encrypt certificates
 <a class="headerlink" href="#_example_sample_span_class_monospaced_pvenode_span_invocation_for_using_let_8217_s_encrypt_certificates" title="Permalink to this heading"></a>
</h5>
<div class="listingblock">
<div class="content monospaced">
<pre>root@proxmox:~# pvenode acme account register default [email protected]
Directory endpoints:
0) Let's Encrypt V2 (https://acme-v02.api.letsencrypt.org/directory)
1) Let's Encrypt V2 Staging (https://acme-staging-v02.api.letsencrypt.org/directory)
2) Custom
Enter selection: 1

Terms of Service: https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf
Do you agree to the above terms? [y|N]y
...
Task OK
root@proxmox:~# pvenode config set --acme domains=example.invalid
root@proxmox:~# pvenode acme cert order
Loading ACME account details
Placing ACME order
...
Status is 'valid'!

All domains validated!
...
Downloading certificate
Setting pveproxy certificate and key
Restarting pveproxy
Task OK</pre>
</div></div>
</div>
<div class="sect4">
<h5 id="_example_setting_up_the_ovh_api_for_validating_a_domain">Example: Setting up the OVH API for validating a domain
 <a class="headerlink" href="#_example_setting_up_the_ovh_api_for_validating_a_domain" title="Permalink to this heading"></a>
</h5>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">the account registration steps are the same no matter which plugins are
used, and are not repeated here.</td>
</tr></tbody></table>
</div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content"><span class="monospaced">OVH_AK</span> and <span class="monospaced">OVH_AS</span> need to be obtained from OVH according to the OVH
API documentation</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>First you need to get all information so you and Proxmox VE can access the API.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>root@proxmox:~# cat /path/to/api-token
OVH_AK=XXXXXXXXXXXXXXXX
OVH_AS=YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
root@proxmox:~# source /path/to/api-token
root@proxmox:~# curl -XPOST -H"X-Ovh-Application: $OVH_AK" -H "Content-type: application/json" \
https://eu.api.ovh.com/1.0/auth/credential  -d '{
  "accessRules": [
    {"method": "GET","path": "/auth/time"},
    {"method": "GET","path": "/domain"},
    {"method": "GET","path": "/domain/zone/*"},
    {"method": "GET","path": "/domain/zone/*/record"},
    {"method": "POST","path": "/domain/zone/*/record"},
    {"method": "POST","path": "/domain/zone/*/refresh"},
    {"method": "PUT","path": "/domain/zone/*/record/"},
    {"method": "DELETE","path": "/domain/zone/*/record/*"}
]
}'
{"consumerKey":"ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ","state":"pendingValidation","validationUrl":"https://eu.api.ovh.com/auth/?credentialToken=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"}

(open validation URL and follow instructions to link Application Key with account/Consumer Key)

root@proxmox:~# echo "OVH_CK=ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ" &gt;&gt; /path/to/api-token</pre>
</div></div>
<div class="paragraph">
<p>Now you can setup the the ACME plugin:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>root@proxmox:~# pvenode acme plugin add dns example_plugin --api ovh --data /path/to/api_token
root@proxmox:~# pvenode acme plugin config example_plugin
┌────────┬──────────────────────────────────────────┐
│ key    │ value                                    │
╞════════╪══════════════════════════════════════════╡
│ api    │ ovh                                      │
├────────┼──────────────────────────────────────────┤
│ data   │ OVH_AK=XXXXXXXXXXXXXXXX                  │
│        │ OVH_AS=YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY  │
│        │ OVH_CK=ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ  │
├────────┼──────────────────────────────────────────┤
│ digest │ 867fcf556363ca1bea866863093fcab83edf47a1 │
├────────┼──────────────────────────────────────────┤
│ plugin │ example_plugin                           │
├────────┼──────────────────────────────────────────┤
│ type   │ dns                                      │
└────────┴──────────────────────────────────────────┘</pre>
</div></div>
<div class="paragraph">
<p>At last you can configure the domain you want to get certificates for and
place the certificate order for it:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>root@proxmox:~# pvenode config set -acmedomain0 example.proxmox.com,plugin=example_plugin
root@proxmox:~# pvenode acme cert order
Loading ACME account details
Placing ACME order
Order URL: https://acme-staging-v02.api.letsencrypt.org/acme/order/11111111/22222222

Getting authorization details from 'https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/33333333'
The validation for example.proxmox.com is pending!
[Wed Apr 22 09:25:30 CEST 2020] Using OVH endpoint: ovh-eu
[Wed Apr 22 09:25:30 CEST 2020] Checking authentication
[Wed Apr 22 09:25:30 CEST 2020] Consumer key is ok.
[Wed Apr 22 09:25:31 CEST 2020] Adding record
[Wed Apr 22 09:25:32 CEST 2020] Added, sleep 10 seconds.
Add TXT record: _acme-challenge.example.proxmox.com
Triggering validation
Sleeping for 5 seconds
Status is 'valid'!
[Wed Apr 22 09:25:48 CEST 2020] Using OVH endpoint: ovh-eu
[Wed Apr 22 09:25:48 CEST 2020] Checking authentication
[Wed Apr 22 09:25:48 CEST 2020] Consumer key is ok.
Remove TXT record: _acme-challenge.example.proxmox.com

All domains validated!

Creating CSR
Checking order status
Order is ready, finalizing order
valid!

Downloading certificate
Setting pveproxy certificate and key
Restarting pveproxy
Task OK</pre>
</div></div>
</div>
<div class="sect4">
<h5 id="sysadmin_certs_acme_switch_from_staging">Example: Switching from the <span class="monospaced">staging</span> to the regular ACME directory
 <a class="headerlink" href="#sysadmin_certs_acme_switch_from_staging" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Changing the ACME directory for an account is unsupported, but as Proxmox VE
supports more than one account you can just create a new one with the
production (trusted) ACME directory as endpoint.  You can also deactivate the
staging account and recreate it.</p></div>
<div class="listingblock">
<div class="title">Example: Changing the <span class="monospaced">default</span> ACME account from <span class="monospaced">staging</span> to directory using <span class="monospaced">pvenode</span></div>
<div class="content monospaced">
<pre>root@proxmox:~# pvenode acme account deactivate default
Renaming account file from '/etc/pve/priv/acme/default' to '/etc/pve/priv/acme/_deactivated_default_4'
Task OK

root@proxmox:~# pvenode acme account register default [email protected]
Directory endpoints:
0) Let's Encrypt V2 (https://acme-v02.api.letsencrypt.org/directory)
1) Let's Encrypt V2 Staging (https://acme-staging-v02.api.letsencrypt.org/directory)
2) Custom
Enter selection: 0

Terms of Service: https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf
Do you agree to the above terms? [y|N]y
...
Task OK</pre>
</div></div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="sysboot">
<span>3.13. Host Bootloader</span>
 <a class="headerlink" href="#sysboot" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Proxmox VE currently uses one of two bootloaders depending on the disk setup
selected in the installer.</p></div>
<div class="paragraph">
<p>For EFI Systems installed with ZFS as the root filesystem <span class="monospaced">systemd-boot</span> is
used, unless Secure Boot is enabled. All other deployments use the standard
GRUB bootloader (this usually also applies to systems which are installed on
top of Debian).</p></div>
<div class="sect3">
<h4 id="sysboot_installer_part_scheme">3.13.1. Partitioning Scheme Used by the Installer
 <a class="headerlink" href="#sysboot_installer_part_scheme" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The Proxmox VE installer creates 3 partitions on all disks selected for
installation.</p></div>
<div class="paragraph">
<p>The created partitions are:</p></div>
<div class="ulist"><ul>
<li>
<p>
a 1 MB BIOS Boot Partition (gdisk type EF02)
</p>
</li>
<li>
<p>
a 512 MB EFI System Partition (ESP, gdisk type EF00)
</p>
</li>
<li>
<p>
a third partition spanning the set <span class="monospaced">hdsize</span> parameter or the remaining space
    used for the chosen storage type
</p>
</li>
</ul></div>
<div class="paragraph">
<p>Systems using ZFS as root filesystem are booted with a kernel and initrd image
stored on the 512 MB EFI System Partition. For legacy BIOS systems, and EFI
systems with Secure Boot enabled, GRUB is used, for EFI systems without
Secure Boot, <span class="monospaced">systemd-boot</span> is used. Both are installed and configured to point
to the ESPs.</p></div>
<div class="paragraph">
<p>GRUB in BIOS mode (<span class="monospaced">--target i386-pc</span>) is installed onto the BIOS Boot
Partition of all selected disks on all systems booted with GRUB
<span class="footnote" data-note="These are all installs with root on <span class=&quot;monospaced&quot;>ext4</span> or <span class=&quot;monospaced&quot;>xfs</span> and installs
with root on ZFS on non-EFI systems">[<a id="_footnoteref_7" href="#_footnote_7" title="View footnote" class="footnote">7</a>]</span>.</p></div>
</div>
<div class="sect3">
<h4 id="sysboot_proxmox_boot_tool">3.13.2. Synchronizing the content of the ESP with <span class="monospaced">proxmox-boot-tool</span>
 <a class="headerlink" href="#sysboot_proxmox_boot_tool" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p><span class="monospaced">proxmox-boot-tool</span> is a utility used to keep the contents of the EFI System
Partitions properly configured and synchronized. It copies certain kernel
versions to all ESPs and configures the respective bootloader to boot from
the <span class="monospaced">vfat</span> formatted ESPs. In the context of ZFS as root filesystem this means
that you can use all optional features on your root pool instead of the subset
which is also present in the ZFS implementation in GRUB or having to create a
separate small boot-pool <span class="footnote" data-note="Booting ZFS on root with GRUB
<a href=&quot;https://github.com/zfsonlinux/zfs/wiki/Debian-Stretch-Root-on-ZFS&quot;>https://github.com/zfsonlinux/zfs/wiki/Debian-Stretch-Root-on-ZFS</a>">[<a id="_footnoteref_8" href="#_footnote_8" title="View footnote" class="footnote">8</a>]</span>.</p></div>
<div class="paragraph">
<p>In setups with redundancy all disks are partitioned with an ESP, by the
installer. This ensures the system boots even if the first boot device fails
or if the BIOS can only boot from a particular disk.</p></div>
<div class="paragraph">
<p>The ESPs are not kept mounted during regular operation. This helps to prevent
filesystem corruption to the <span class="monospaced">vfat</span> formatted ESPs in case of a system crash,
and removes the need to manually adapt <span class="monospaced">/etc/fstab</span> in case the primary boot
device fails.</p></div>
<div class="paragraph">
<p><span class="monospaced">proxmox-boot-tool</span> handles the following tasks:</p></div>
<div class="ulist"><ul>
<li>
<p>
formatting and setting up a new partition
</p>
</li>
<li>
<p>
copying and configuring new kernel images and initrd images to all listed ESPs
</p>
</li>
<li>
<p>
synchronizing the configuration on kernel upgrades and other maintenance tasks
</p>
</li>
<li>
<p>
managing the list of kernel versions which are synchronized
</p>
</li>
<li>
<p>
configuring the boot-loader to boot a particular kernel version (pinning)
</p>
</li>
</ul></div>
<div class="paragraph">
<p>You can view the currently configured ESPs and their state by running:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># proxmox-boot-tool status</pre>
</div></div>
<div class="paragraph" id="sysboot_proxmox_boot_setup">
<div class="title">Setting up a new partition for use as synced ESP</div><p>To format and initialize a partition as synced ESP, e.g., after replacing a
failed vdev in an rpool, or when converting an existing system that pre-dates
the sync mechanism, <span class="monospaced">proxmox-boot-tool</span> from <span class="monospaced">proxmox-kernel-helper</span> can be used.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">the <span class="monospaced">format</span> command will format the <span class="monospaced">&lt;partition&gt;</span>, make sure to pass
in the right device/partition!</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>For example, to format an empty partition <span class="monospaced">/dev/sda2</span> as ESP, run the following:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># proxmox-boot-tool format /dev/sda2</pre>
</div></div>
<div class="paragraph">
<p>To setup an existing, unmounted ESP located on <span class="monospaced">/dev/sda2</span> for inclusion in
Proxmox VE’s kernel update synchronization mechanism, use the following:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># proxmox-boot-tool init /dev/sda2</pre>
</div></div>
<div class="paragraph">
<p>or</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># proxmox-boot-tool init /dev/sda2 grub</pre>
</div></div>
<div class="paragraph">
<p>to force initialization with GRUB instead of <span class="monospaced">systemd-boot</span>, for example for
Secure Boot support.</p></div>
<div class="paragraph">
<p>Afterwards <span class="monospaced">/etc/kernel/proxmox-boot-uuids</span> should contain a new line with the
UUID of the newly added partition. The <span class="monospaced">init</span> command will also automatically
trigger a refresh of all configured ESPs.</p></div>
<div class="paragraph" id="sysboot_proxmox_boot_refresh">
<div class="title">Updating the configuration on all ESPs</div><p>To copy and configure all bootable kernels and keep all ESPs listed in
<span class="monospaced">/etc/kernel/proxmox-boot-uuids</span> in sync you just need to run:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># proxmox-boot-tool refresh</pre>
</div></div>
<div class="paragraph">
<p>(The equivalent to running <span class="monospaced">update-grub</span> systems with <span class="monospaced">ext4</span> or <span class="monospaced">xfs</span> on root).</p></div>
<div class="paragraph">
<p>This is necessary should you make changes to the kernel commandline, or want to
sync all kernels and initrds.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Both <span class="monospaced">update-initramfs</span> and <span class="monospaced">apt</span> (when necessary) will automatically
trigger a refresh.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<div class="title">Kernel Versions considered by <span class="monospaced">proxmox-boot-tool</span></div><p>The following kernel versions are configured by default:</p></div>
<div class="ulist"><ul>
<li>
<p>
the currently running kernel
</p>
</li>
<li>
<p>
the version being newly installed on package updates
</p>
</li>
<li>
<p>
the two latest already installed kernels
</p>
</li>
<li>
<p>
the latest version of the second-to-last kernel series (e.g. 5.0, 5.3), if applicable
</p>
</li>
<li>
<p>
any manually selected kernels
</p>
</li>
</ul></div>
<div class="paragraph">
<div class="title">Manually keeping a kernel bootable</div><p>Should you wish to add a certain kernel and initrd image to the list of
bootable kernels use <span class="monospaced">proxmox-boot-tool kernel add</span>.</p></div>
<div class="paragraph">
<p>For example run the following to add the kernel with ABI version <span class="monospaced">5.0.15-1-pve</span>
to the list of kernels to keep installed and synced to all ESPs:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># proxmox-boot-tool kernel add 5.0.15-1-pve</pre>
</div></div>
<div class="paragraph">
<p><span class="monospaced">proxmox-boot-tool kernel list</span> will list all kernel versions currently selected
for booting:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># proxmox-boot-tool kernel list
Manually selected kernels:
5.0.15-1-pve

Automatically selected kernels:
5.0.12-1-pve
4.15.18-18-pve</pre>
</div></div>
<div class="paragraph">
<p>Run <span class="monospaced">proxmox-boot-tool kernel remove</span> to remove a kernel from the list of
manually selected kernels, for example:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># proxmox-boot-tool kernel remove 5.0.15-1-pve</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">It’s required to run <span class="monospaced">proxmox-boot-tool refresh</span> to update all EFI System
Partitions (ESPs) after a manual kernel addition or removal from above.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="sysboot_determine_bootloader_used">3.13.3. Determine which Bootloader is Used
 <a class="headerlink" href="#sysboot_determine_bootloader_used" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<a class="image" href="images/screenshot/boot-grub.png">
<img src="images/screenshot/boot-grub.png" alt="screenshot/boot-grub.png" width="250" style="padding: 0 10px 0 0;float:left;"></a>
<p>The simplest and most reliable way to determine which bootloader is used, is to
watch the boot process of the Proxmox VE node.</p></div>
<div class="paragraph">
<p>You will either see the blue box of GRUB or the simple black on white
<span class="monospaced">systemd-boot</span>.</p></div>
<div class="paragraph">
<a class="image" href="images/screenshot/boot-systemdboot.png">
<img src="images/screenshot/boot-systemdboot.png" alt="screenshot/boot-systemdboot.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>Determining the bootloader from a running system might not be 100% accurate. The
safest way is to run the following command:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># efibootmgr -v</pre>
</div></div>
<div class="paragraph">
<p>If it returns a message that EFI variables are not supported, GRUB is used in
BIOS/Legacy mode.</p></div>
<div class="paragraph">
<p>If the output contains a line that looks similar to the following, GRUB is
used in UEFI mode.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>Boot0005* proxmox       [...] File(\EFI\proxmox\grubx64.efi)</pre>
</div></div>
<div class="paragraph">
<p>If the output contains a line similar to the following, <span class="monospaced">systemd-boot</span> is used.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>Boot0006* Linux Boot Manager    [...] File(\EFI\systemd\systemd-bootx64.efi)</pre>
</div></div>
<div class="paragraph">
<p>By running:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># proxmox-boot-tool status</pre>
</div></div>
<div class="paragraph">
<p>you can find out if <span class="monospaced">proxmox-boot-tool</span> is configured, which is a good
indication of how the system is booted.</p></div>
</div>
<div class="sect3">
<h4 id="sysboot_grub">3.13.4. GRUB
 <a class="headerlink" href="#sysboot_grub" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>GRUB has been the de-facto standard for booting Linux systems for many years
and is quite well documented
<span class="footnote" data-note="GRUB Manual <a href=&quot;https://www.gnu.org/software/grub/manual/grub/grub.html&quot;>https://www.gnu.org/software/grub/manual/grub/grub.html</a>">[<a id="_footnoteref_9" href="#_footnote_9" title="View footnote" class="footnote">9</a>]</span>.</p></div>
<div class="sect4">
<h5 id="_configuration">Configuration
 <a class="headerlink" href="#_configuration" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Changes to the GRUB configuration are done via the defaults file
<span class="monospaced">/etc/default/grub</span> or config snippets in <span class="monospaced">/etc/default/grub.d</span>. To regenerate
the configuration file after a change to the configuration run:
<span class="footnote" data-note="Systems using <span class=&quot;monospaced&quot;>proxmox-boot-tool</span> will call <span class=&quot;monospaced&quot;>proxmox-boot-tool
refresh</span> upon <span class=&quot;monospaced&quot;>update-grub</span>.">[<a id="_footnoteref_10" href="#_footnote_10" title="View footnote" class="footnote">10</a>]</span></p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># update-grub</pre>
</div></div>
</div>
</div>
<div class="sect3">
<h4 id="sysboot_systemd_boot">3.13.5. Systemd-boot
 <a class="headerlink" href="#sysboot_systemd_boot" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p><span class="monospaced">systemd-boot</span> is a lightweight EFI bootloader. It reads the kernel and initrd
images directly from the EFI Service Partition (ESP) where it is installed.
The main advantage of directly loading the kernel from the ESP is that it does
not need to reimplement the drivers for accessing the storage. In Proxmox VE
<a href="#sysboot_proxmox_boot_tool"><span class="monospaced">proxmox-boot-tool</span></a> is used to keep the
configuration on the ESPs synchronized.</p></div>
<div class="sect4">
<h5 id="sysboot_systemd_boot_config">Configuration
 <a class="headerlink" href="#sysboot_systemd_boot_config" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p><span class="monospaced">systemd-boot</span> is configured via the file <span class="monospaced">loader/loader.conf</span> in the root
directory of an EFI System Partition (ESP). See the <span class="monospaced">loader.conf(5)</span> manpage
for details.</p></div>
<div class="paragraph">
<p>Each bootloader entry is placed in a file of its own in the directory
<span class="monospaced">loader/entries/</span></p></div>
<div class="paragraph">
<p>An example entry.conf looks like this (<span class="monospaced">/</span> refers to the root of the ESP):</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>title    Proxmox
version  5.0.15-1-pve
options   root=ZFS=rpool/ROOT/pve-1 boot=zfs
linux    /EFI/proxmox/5.0.15-1-pve/vmlinuz-5.0.15-1-pve
initrd   /EFI/proxmox/5.0.15-1-pve/initrd.img-5.0.15-1-pve</pre>
</div></div>
</div>
</div>
<div class="sect3">
<h4 id="sysboot_edit_kernel_cmdline">3.13.6. Editing the Kernel Commandline
 <a class="headerlink" href="#sysboot_edit_kernel_cmdline" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>You can modify the kernel commandline in the following places, depending on the
bootloader used:</p></div>
<div class="paragraph">
<div class="title">GRUB</div><p>The kernel commandline needs to be placed in the variable
<span class="monospaced">GRUB_CMDLINE_LINUX_DEFAULT</span> in the file <span class="monospaced">/etc/default/grub</span>. Running
<span class="monospaced">update-grub</span> appends its content to all <span class="monospaced">linux</span> entries in
<span class="monospaced">/boot/grub/grub.cfg</span>.</p></div>
<div class="paragraph">
<div class="title">Systemd-boot</div><p>The kernel commandline needs to be placed as one line in <span class="monospaced">/etc/kernel/cmdline</span>.
To apply your changes, run <span class="monospaced">proxmox-boot-tool refresh</span>, which sets it as the
<span class="monospaced">option</span> line for all config files in <span class="monospaced">loader/entries/proxmox-*.conf</span>.</p></div>
<div class="paragraph">
<p>A complete list of kernel parameters can be found at
<em>https://www.kernel.org/doc/html/v&lt;YOUR-KERNEL-VERSION&gt;/admin-guide/kernel-parameters.html</em>.
replace &lt;YOUR-KERNEL-VERSION&gt; with the major.minor version, for example, for
kernels based on version 6.5 the URL would be:
<a href="https://www.kernel.org/doc/html/v6.5/admin-guide/kernel-parameters.html">https://www.kernel.org/doc/html/v6.5/admin-guide/kernel-parameters.html</a></p></div>
<div class="paragraph">
<p>You can find your kernel version by checking the web interface (<em>Node →
Summary</em>), or by running</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># uname -r</pre>
</div></div>
<div class="paragraph">
<p>Use the first two numbers at the front of the output.</p></div>
</div>
<div class="sect3">
<h4 id="sysboot_kernel_pin">3.13.7. Override the Kernel-Version for next Boot
 <a class="headerlink" href="#sysboot_kernel_pin" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>To select a kernel that is not currently the default kernel, you can either:</p></div>
<div class="ulist"><ul>
<li>
<p>
use the boot loader menu that is displayed at the beginning of the boot
  process
</p>
</li>
<li>
<p>
use the <span class="monospaced">proxmox-boot-tool</span> to <span class="monospaced">pin</span> the system to a kernel version either
  once or permanently (until pin is reset).
</p>
</li>
</ul></div>
<div class="paragraph">
<p>This should help you work around incompatibilities between a newer kernel
version and the hardware.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Such a pin should be removed as soon as possible so that all current
security patches of the latest kernel are also applied to the system.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>For example: To permanently select the version <span class="monospaced">5.15.30-1-pve</span> for booting you
would run:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># proxmox-boot-tool kernel pin 5.15.30-1-pve</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">The pinning functionality works for all Proxmox VE systems, not only those using
<span class="monospaced">proxmox-boot-tool</span> to synchronize the contents of the ESPs, if your system
does not use <span class="monospaced">proxmox-boot-tool</span> for synchronizing you can also skip the
<span class="monospaced">proxmox-boot-tool refresh</span> call in the end.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>You can also set a kernel version to be booted on the next system boot only.
This is for example useful to test if an updated kernel has resolved an issue,
which caused you to <span class="monospaced">pin</span> a version in the first place:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># proxmox-boot-tool kernel pin 5.15.30-1-pve --next-boot</pre>
</div></div>
<div class="paragraph">
<p>To remove any pinned version configuration use the <span class="monospaced">unpin</span> subcommand:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># proxmox-boot-tool kernel unpin</pre>
</div></div>
<div class="paragraph">
<p>While <span class="monospaced">unpin</span> has a <span class="monospaced">--next-boot</span> option as well, it is used to clear a pinned
version set with <span class="monospaced">--next-boot</span>. As that happens already automatically on boot,
invonking it manually is of little use.</p></div>
<div class="paragraph">
<p>After setting, or clearing pinned versions you also need to synchronize the
content and configuration on the ESPs by running the <span class="monospaced">refresh</span> subcommand.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">You will be prompted to automatically do for  <span class="monospaced">proxmox-boot-tool</span> managed
systems if you call the tool interactively.</td>
</tr></tbody></table>
</div>
<div class="listingblock">
<div class="content monospaced">
<pre># proxmox-boot-tool refresh</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="sysboot_secure_boot">3.13.8. Secure Boot
 <a class="headerlink" href="#sysboot_secure_boot" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Since Proxmox VE 8.1, Secure Boot is supported out of the box via signed packages
and integration in <span class="monospaced">proxmox-boot-tool</span>.</p></div>
<div class="paragraph">
<p>The following packages are required for secure boot to work. You can
install them all at once by using the ‘proxmox-secure-boot-support’
meta-package.</p></div>
<div class="ulist"><ul>
<li>
<p>
<span class="monospaced">shim-signed</span> (shim bootloader signed by Microsoft)
</p>
</li>
<li>
<p>
<span class="monospaced">shim-helpers-amd64-signed</span> (fallback bootloader and MOKManager, signed by
  Proxmox)
</p>
</li>
<li>
<p>
<span class="monospaced">grub-efi-amd64-signed</span> (GRUB EFI bootloader, signed by Proxmox)
</p>
</li>
<li>
<p>
<span class="monospaced">proxmox-kernel-6.X.Y-Z-pve-signed</span> (Kernel image, signed by Proxmox)
</p>
</li>
</ul></div>
<div class="paragraph">
<p>Only GRUB is supported as bootloader out of the box, since other bootloader are
currently not eligible for secure boot code-signing.</p></div>
<div class="paragraph">
<p>Any new installation of Proxmox VE will automatically have all of the above packages
included.</p></div>
<div class="paragraph">
<p>More details about how Secure Boot works, and how to customize the setup, are
available in <a href="https://pve.proxmox.com/wiki/Secure_Boot_Setup">our wiki</a>.</p></div>
<div class="sect4">
<h5 id="_switching_an_existing_installation_to_secure_boot">Switching an Existing Installation to Secure Boot
 <a class="headerlink" href="#_switching_an_existing_installation_to_secure_boot" title="Permalink to this heading"></a>
</h5>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">This can lead to an unbootable installation in some cases if not done
correctly. Reinstalling the host will setup Secure Boot automatically if
available, without any extra interactions. <strong>Make sure you have a working and
well-tested backup of your Proxmox VE host!</strong></td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>An existing UEFI installation can be switched over to Secure Boot if desired,
without having to reinstall Proxmox VE from scratch.</p></div>
<div class="paragraph">
<p>First, ensure all your system is up-to-date. Next, install
<span class="monospaced">proxmox-secure-boot-support</span>. GRUB automatically creates the needed EFI boot
entry for booting via the default shim.</p></div>
<div class="paragraph">
<div class="title">systemd-boot</div><p>If <span class="monospaced">systemd-boot</span> is used as a bootloader (see
<a href="#sysboot_determine_bootloader_used">Determine which Bootloader is used</a>),
some additional setup is needed. This is only the case if Proxmox VE was installed
with ZFS-on-root.</p></div>
<div class="paragraph">
<p>To check the latter, run:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># findmnt /</pre>
</div></div>
<div class="paragraph">
<p>If the host is indeed using ZFS as root filesystem, the <span class="monospaced">FSTYPE</span> column
should contain <span class="monospaced">zfs</span>:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>TARGET SOURCE           FSTYPE OPTIONS
/      rpool/ROOT/pve-1 zfs    rw,relatime,xattr,noacl,casesensitive</pre>
</div></div>
<div class="paragraph">
<p>Next, a suitable potential ESP (EFI system partition) must be found. This can be
done using the <span class="monospaced">lsblk</span> command as following:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># lsblk -o +FSTYPE</pre>
</div></div>
<div class="paragraph">
<p>The output should look something like this:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS FSTYPE
sda      8:0    0   32G  0 disk
├─sda1   8:1    0 1007K  0 part
├─sda2   8:2    0  512M  0 part             vfat
└─sda3   8:3    0 31.5G  0 part             zfs_member
sdb      8:16   0   32G  0 disk
├─sdb1   8:17   0 1007K  0 part
├─sdb2   8:18   0  512M  0 part             vfat
└─sdb3   8:19   0 31.5G  0 part             zfs_member</pre>
</div></div>
<div class="paragraph">
<p>In this case, the partitions <span class="monospaced">sda2</span> and <span class="monospaced">sdb2</span> are the targets. They can be
identified by the their size of 512M and their <span class="monospaced">FSTYPE</span> being <span class="monospaced">vfat</span>, in this
case on a ZFS RAID-1 installation.</p></div>
<div class="paragraph">
<p>These partitions must be properly set up for booting through GRUB using
<span class="monospaced">proxmox-boot-tool</span>. This command (using <span class="monospaced">sda2</span> as an example) must be run
separately for each individual ESP:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># proxmox-boot-tool init /dev/sda2 grub</pre>
</div></div>
<div class="paragraph">
<p>Afterwards, you can sanity-check the setup by running the following command:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># efibootmgr -v</pre>
</div></div>
<div class="paragraph">
<p>This list should contain an entry looking similar to this:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>[..]
Boot0009* proxmox       HD(2,GPT,..,0x800,0x100000)/File(\EFI\proxmox\shimx64.efi)
[..]</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">The old <span class="monospaced">systemd-boot</span> bootloader will be kept, but GRUB will be
preferred. This way, if booting using GRUB in Secure Boot mode does not work for
any reason, the system can still be booted using <span class="monospaced">systemd-boot</span> with Secure Boot
turned off.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>Now the host can be rebooted and Secure Boot enabled in the UEFI firmware setup
utility.</p></div>
<div class="paragraph">
<p>On reboot, a new entry named <span class="monospaced">proxmox</span> should be selectable in the UEFI firmware
boot menu, which boots using the pre-signed EFI shim.</p></div>
<div class="paragraph">
<p>If, for any reason, no <span class="monospaced">proxmox</span> entry can be found in the UEFI boot menu, you
can try adding it manually (if supported by the firmware), by adding the file
<span class="monospaced">\EFI\proxmox\shimx64.efi</span> as a custom boot entry.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Some UEFI firmwares are known to drop the <span class="monospaced">proxmox</span> boot option on reboot.
This can happen if the <span class="monospaced">proxmox</span> boot entry is pointing to a GRUB installation
on a disk, where the disk itself is not a boot option. If possible, try adding
the disk as a boot option in the UEFI firmware setup utility and run
<span class="monospaced">proxmox-boot-tool</span> again.</td>
</tr></tbody></table>
</div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">To enroll custom keys, see the accompanying
<a href="https://pve.proxmox.com/wiki/Secure_Boot_Setup#Setup_instructions_for_db_key_variant">Secure
Boot wiki page</a>.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect4">
<h5 id="_using_dkms_third_party_modules_with_secure_boot">Using DKMS/Third Party Modules With Secure Boot
 <a class="headerlink" href="#_using_dkms_third_party_modules_with_secure_boot" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>On systems with Secure Boot enabled, the kernel will refuse to load modules
which are not signed by a trusted key. The default set of modules shipped with
the kernel packages is signed with an ephemeral key embedded in the kernel
image which is trusted by that specific version of the kernel image.</p></div>
<div class="paragraph">
<p>In order to load other modules, such as those built with DKMS or manually, they
need to be signed with a key trusted by the Secure Boot stack. The easiest way
to achieve this is to enroll them as Machine Owner Key (<span class="monospaced">MOK</span>) with <span class="monospaced">mokutil</span>.</p></div>
<div class="paragraph">
<p>The <span class="monospaced">dkms</span> tool will automatically generate a keypair and certificate in
<span class="monospaced">/var/lib/dkms/mok.key</span> and <span class="monospaced">/var/lib/dkms/mok.pub</span> and use it for signing
the kernel modules it builds and installs.</p></div>
<div class="paragraph">
<p>You can view the certificate contents with</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># openssl x509 -in /var/lib/dkms/mok.pub -noout -text</pre>
</div></div>
<div class="paragraph">
<p>and enroll it on your system using the following command:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># mokutil --import /var/lib/dkms/mok.pub
input password:
input password again:</pre>
</div></div>
<div class="paragraph">
<p>The <span class="monospaced">mokutil</span> command will ask for a (temporary) password twice, this password
needs to be entered one more time in the next step of the process! Rebooting
the system should automatically boot into the <span class="monospaced">MOKManager</span> EFI binary, which
allows you to verify the key/certificate and confirm the enrollment using the
password selected when starting the enrollment using <span class="monospaced">mokutil</span>. Afterwards, the
kernel should allow loading modules built with DKMS (which are signed with the
enrolled <span class="monospaced">MOK</span>). The <span class="monospaced">MOK</span> can also be used to sign custom EFI binaries and
kernel images if desired.</p></div>
<div class="paragraph">
<p>The same procedure can also be used for custom/third-party modules not managed
with DKMS, but the key/certificate generation and signing steps need to be done
manually in that case.</p></div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="kernel_samepage_merging">
<span>3.14. Kernel Samepage Merging (KSM)</span>
 <a class="headerlink" href="#kernel_samepage_merging" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Kernel Samepage Merging (KSM) is an optional memory deduplication feature
offered by the Linux kernel, which is enabled by default in Proxmox VE. KSM
works by scanning a range of physical memory pages for identical content, and
identifying the virtual pages that are mapped to them. If identical pages are
found, the corresponding virtual pages are re-mapped so that they all point to
the same physical page, and the old pages are freed. The virtual pages are
marked as "copy-on-write", so that any writes to them will be written to a new
area of memory, leaving the shared physical page intact.</p></div>
<div class="sect3">
<h4 id="_implications_of_ksm">3.14.1. Implications of KSM
 <a class="headerlink" href="#_implications_of_ksm" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>KSM can optimize memory usage in virtualization environments, as multiple VMs
running similar operating systems or workloads could potentially share a lot of
common memory pages.</p></div>
<div class="paragraph">
<p>However, while KSM can reduce memory usage, it also comes with some security
risks, as it can expose VMs to side-channel attacks. Research has shown that it
is possible to infer information about a running VM via a second VM on the same
host, by exploiting certain characteristics of KSM.</p></div>
<div class="paragraph">
<p>Thus, if you are using Proxmox VE to provide hosting services, you should consider
disabling KSM, in order to provide your users with additional security.
Furthermore, you should check your country’s regulations, as disabling KSM may
be a legal requirement.</p></div>
</div>
<div class="sect3">
<h4 id="_disabling_ksm">3.14.2. Disabling KSM
 <a class="headerlink" href="#_disabling_ksm" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>To see if KSM is active, you can check the output of:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># systemctl status ksmtuned</pre>
</div></div>
<div class="paragraph">
<p>If it is, it can be disabled immediately with:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># systemctl disable --now ksmtuned</pre>
</div></div>
<div class="paragraph">
<p>Finally, to unmerge all the currently merged pages, run:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># echo 2 &gt; /sys/kernel/mm/ksm/run</pre>
</div></div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="chapter_gui">
4. Graphical User Interface
 <a class="headerlink" href="#chapter_gui" title="Permalink to this heading"></a>
</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Proxmox VE is simple. There is no need to install a separate management
tool, and everything can be done through your web browser (Latest
Firefox or Google Chrome is preferred). A built-in HTML5 console is
used to access the guest console. As an alternative,
<a href="https://www.spice-space.org/">SPICE</a> can be used.</p></div>
<div class="paragraph">
<p>Because we use the Proxmox cluster file system (pmxcfs), you can
connect to any node to manage the entire cluster. Each node can manage
the entire cluster. There is no need for a dedicated manager node.</p></div>
<div class="paragraph">
<p>You can use the web-based administration interface with any modern
browser. When Proxmox VE detects that you are connecting from a mobile
device, you are redirected to a simpler, touch-based user interface.</p></div>
<div class="paragraph">
<p>The web interface can be reached via <a href="https://youripaddress:8006">https://youripaddress:8006</a>
(default login is: <em>root</em>, and the password is specified during the
installation process).</p></div>
<div class="sect2">
<h3 id="_features">
<span>4.1. Features</span>
 <a class="headerlink" href="#_features" title="Permalink to this heading"></a>
</h3>
<div class="ulist"><ul>
<li>
<p>
Seamless integration and management of Proxmox VE clusters
</p>
</li>
<li>
<p>
AJAX technologies for dynamic updates of resources
</p>
</li>
<li>
<p>
Secure access to all Virtual Machines and Containers via SSL
  encryption (https)
</p>
</li>
<li>
<p>
Fast search-driven interface, capable of handling hundreds and
  probably thousands of VMs
</p>
</li>
<li>
<p>
Secure HTML5 console or SPICE
</p>
</li>
<li>
<p>
Role based permission management for all objects (VMs, storages,
  nodes, etc.)
</p>
</li>
<li>
<p>
Support for multiple authentication sources (e.g. local, MS ADS,
  LDAP, …)
</p>
</li>
<li>
<p>
Two-Factor Authentication (OATH, Yubikey)
</p>
</li>
<li>
<p>
Based on ExtJS 7.x JavaScript framework
</p>
</li>
</ul></div>
</div>
<div class="sect2">
<h3 id="_login">
<span>4.2. Login</span>
 <a class="headerlink" href="#_login" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-login-window.png">
<img src="images/screenshot/gui-login-window.png" alt="screenshot/gui-login-window.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>When you connect to the server, you will first see the login window.
Proxmox VE supports various authentication backends (<em>Realm</em>), and
you can select the language here. The GUI is translated to more
than 20 languages.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">You can save the user name on the client side by selecting the
checkbox at the bottom. This saves some typing when you login next
time.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect2">
<h3 id="_gui_overview">
<span>4.3. GUI Overview</span>
 <a class="headerlink" href="#_gui_overview" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-datacenter-summary.png">
<img src="images/screenshot/gui-datacenter-summary.png" alt="screenshot/gui-datacenter-summary.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>The Proxmox VE user interface consists of four regions.</p></div>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
Header
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
On top. Shows status information and contains buttons for
most important actions.
</p>
</td>
</tr>
<tr>
<td class="hdlist1">
Resource Tree
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
At the left side. A navigation tree where you can select
specific objects.
</p>
</td>
</tr>
<tr>
<td class="hdlist1">
Content Panel
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Center region. Selected objects display configuration
options and status here.
</p>
</td>
</tr>
<tr>
<td class="hdlist1">
Log Panel
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
At the bottom. Displays log entries for recent tasks. You
can double-click on those log entries to get more details, or to abort
a running task.
</p>
</td>
</tr>
</tbody></table></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">You can shrink and expand the size of the resource tree and log
panel, or completely hide the log panel. This can be helpful when you
work on small displays and want more space to view other content.</td>
</tr></tbody></table>
</div>
<div class="sect3">
<h4 id="_header">4.3.1. Header
 <a class="headerlink" href="#_header" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>On the top left side, the first thing you see is the Proxmox
logo. Next to it is the current running version of Proxmox VE. In the
search bar nearside you can search for specific objects (VMs,
containers, nodes, …). This is sometimes faster than selecting an
object in the resource tree.</p></div>
<div class="paragraph">
<p>The right part of the header contains four buttons:</p></div>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
Documentation 
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Opens a new browser window showing the reference documentation.
</p>
</td>
</tr>
<tr>
<td class="hdlist1">
Create&nbsp;VM 
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Opens the virtual machine creation wizard.
</p>
</td>
</tr>
<tr>
<td class="hdlist1">
Create&nbsp;CT 
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Open the container creation wizard.
</p>
</td>
</tr>
<tr>
<td class="hdlist1">
User&nbsp;Menu 
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Displays the identity of the user you’re currently logged in
with, and clicking it opens a menu with user-specific options.
</p>
<div class="paragraph">
<p>In the user menu, you’ll find the <em>My Settings</em> dialog, which provides local UI
settings. Below that, there are shortcuts for <em>TFA</em> (Two-Factor Authentication)
and <em>Password</em> self-service. You’ll also find options to change the <em>Language</em>
and the <em>Color Theme.</em> Finally, at the bottom of the menu is the <em>Logout</em>
option.</p></div>
</td>
</tr>
</tbody></table></div>
</div>
<div class="sect3">
<h4 id="gui_my_settings">4.3.2. My Settings
 <a class="headerlink" href="#gui_my_settings" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-my-settings.png">
<img src="images/screenshot/gui-my-settings.png" alt="screenshot/gui-my-settings.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>The <em>My Settings</em> window allows you to set locally stored settings. These
include the <em>Dashboard Storages</em> which allow you to enable or disable specific
storages to be counted towards the total amount visible in the datacenter
summary. If no storage is checked the total is the sum of all storages, same
as enabling every single one.</p></div>
<div class="paragraph">
<p>Below the dashboard settings you find the stored user name and a button to
clear it as well as a button to reset every layout in the GUI to its default.</p></div>
<div class="paragraph">
<p>On the right side there are <em>xterm.js Settings</em>. These contain the following
options:</p></div>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
Font-Family 
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
The font to be used in xterm.js (e.g. Arial).
</p>
</td>
</tr>
<tr>
<td class="hdlist1">
Font-Size 
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
The preferred font size to be used.
</p>
</td>
</tr>
<tr>
<td class="hdlist1">
Letter Spacing 
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Increases or decreases spacing between letters in text.
</p>
</td>
</tr>
<tr>
<td class="hdlist1">
Line Height 
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Specify the absolute height of a line.
</p>
</td>
</tr>
</tbody></table></div>
</div>
<div class="sect3">
<h4 id="_resource_tree">4.3.3. Resource Tree
 <a class="headerlink" href="#_resource_tree" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>This is the main navigation tree. On top of the tree you can select
some predefined views, which change the structure of the tree
below. The default view is the <strong>Server View</strong>, and it shows the following
object types:</p></div>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
Datacenter
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Contains cluster-wide settings (relevant for all nodes).
</p>
</td>
</tr>
<tr>
<td class="hdlist1">
Node
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Represents the hosts inside a cluster, where the guests run.
</p>
</td>
</tr>
<tr>
<td class="hdlist1">
Guest
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
VMs, containers and templates.
</p>
</td>
</tr>
<tr>
<td class="hdlist1">
Storage
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Data Storage.
</p>
</td>
</tr>
<tr>
<td class="hdlist1">
Pool
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
It is possible to group guests using a pool to simplify
management.
</p>
</td>
</tr>
</tbody></table></div>
<div class="paragraph">
<p>The following view types are available:</p></div>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
Server View
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Shows all kinds of objects, grouped by nodes.
</p>
</td>
</tr>
<tr>
<td class="hdlist1">
Folder View
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Shows all kinds of objects, grouped by object type.
</p>
</td>
</tr>
<tr>
<td class="hdlist1">
Pool View
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Show VMs and containers, grouped by pool.
</p>
</td>
</tr>
</tbody></table></div>
</div>
<div class="sect3">
<h4 id="_log_panel">4.3.4. Log Panel
 <a class="headerlink" href="#_log_panel" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The main purpose of the log panel is to show you what is currently
going on in your cluster. Actions like creating an new VM are executed
in the background, and we call such a background job a <em>task</em>.</p></div>
<div class="paragraph">
<p>Any output from such a task is saved into a separate log file. You can
view that log by simply double-click a task log entry. It is also
possible to abort a running task there.</p></div>
<div class="paragraph">
<p>Please note that we display the most recent tasks from all cluster nodes
here. So you can see when somebody else is working on another cluster
node in real-time.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">We remove older and finished task from the log panel to keep
that list short. But you can still find those tasks within the node panel in the
<em>Task History</em>.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>Some short-running actions simply send logs to all cluster
members. You can see those messages in the <em>Cluster log</em> panel.</p></div>
</div>
</div>
<div class="sect2">
<h3 id="_content_panels">
<span>4.4. Content Panels</span>
 <a class="headerlink" href="#_content_panels" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>When you select an item from the resource tree, the corresponding
object displays configuration and status information in the content
panel. The following sections provide a brief overview of this
functionality. Please refer to the corresponding chapters in the
reference documentation to get more detailed information.</p></div>
<div class="sect3">
<h4 id="_datacenter">4.4.1. Datacenter
 <a class="headerlink" href="#_datacenter" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-datacenter-search.png">
<img src="images/screenshot/gui-datacenter-search.png" alt="screenshot/gui-datacenter-search.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>On the datacenter level, you can access cluster-wide settings and information.</p></div>
<div class="ulist"><ul>
<li>
<p>
<strong>Search:</strong> perform a cluster-wide search for nodes, VMs, containers, storage
   devices, and pools.
</p>
</li>
<li>
<p>
<strong>Summary:</strong> gives a brief overview of the cluster’s health and resource usage.
</p>
</li>
<li>
<p>
<strong>Cluster:</strong> provides the functionality and information necessary to create or
   join a cluster.
</p>
</li>
<li>
<p>
<strong>Options:</strong> view and manage cluster-wide default settings.
</p>
</li>
<li>
<p>
<strong>Storage:</strong> provides an interface for managing cluster storage.
</p>
</li>
<li>
<p>
<strong>Backup:</strong> schedule backup jobs. This operates cluster wide, so it doesn’t
   matter where the VMs/containers are on your cluster when scheduling.
</p>
</li>
<li>
<p>
<strong>Replication:</strong> view and manage replication jobs.
</p>
</li>
<li>
<p>
<strong>Permissions:</strong> manage user, group, and API token permissions, and LDAP,
   MS-AD and Two-Factor authentication.
</p>
</li>
<li>
<p>
<strong>HA:</strong> manage Proxmox VE High Availability.
</p>
</li>
<li>
<p>
<strong>ACME:</strong> set up ACME (Let’s Encrypt) certificates for server nodes.
</p>
</li>
<li>
<p>
<strong>Firewall:</strong> configure and make templates for the Proxmox Firewall cluster wide.
</p>
</li>
<li>
<p>
<strong>Metric Server:</strong> define external metric servers for Proxmox VE.
</p>
</li>
<li>
<p>
<strong>Notifications:</strong> configurate notification behavior and targets for  Proxmox VE.
</p>
</li>
<li>
<p>
<strong>Support:</strong> display information about your support subscription.
</p>
</li>
</ul></div>
</div>
<div class="sect3">
<h4 id="_nodes">4.4.2. Nodes
 <a class="headerlink" href="#_nodes" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-node-summary.png">
<img src="images/screenshot/gui-node-summary.png" alt="screenshot/gui-node-summary.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>Nodes in your cluster can be managed individually at this level.</p></div>
<div class="paragraph">
<p>The top header has useful buttons such as <em>Reboot</em>, <em>Shutdown</em>, <em>Shell</em>,
<em>Bulk Actions</em> and <em>Help</em>.
<em>Shell</em> has the options <em>noVNC</em>, <em>SPICE</em> and <em>xterm.js</em>.
<em>Bulk Actions</em> has the options <em>Bulk Start</em>, <em>Bulk Shutdown</em> and <em>Bulk Migrate</em>.</p></div>
<div class="ulist"><ul>
<li>
<p>
<strong>Search:</strong> search a node for VMs, containers, storage devices, and pools.
</p>
</li>
<li>
<p>
<strong>Summary:</strong> display a brief overview of the node’s resource usage.
</p>
</li>
<li>
<p>
<strong>Notes:</strong> write custom comments in <a href="#markdown_basics">Markdown syntax</a>.
</p>
</li>
<li>
<p>
<strong>Shell:</strong> access to a shell interface for the node.
</p>
</li>
<li>
<p>
<strong>System:</strong> configure network, DNS and time settings, and access the syslog.
</p>
</li>
<li>
<p>
<strong>Updates:</strong> upgrade the system and see the available new packages.
</p>
</li>
<li>
<p>
<strong>Firewall:</strong> manage the Proxmox Firewall for a specific node.
</p>
</li>
<li>
<p>
<strong>Disks:</strong> get an overview of the attached disks, and manage how they are used.
</p>
</li>
<li>
<p>
<strong>Ceph:</strong> is only used if you have installed a Ceph server on your
   host. In this case, you can manage your Ceph cluster and see the status
   of it here.
</p>
</li>
<li>
<p>
<strong>Replication:</strong> view and manage replication jobs.
</p>
</li>
<li>
<p>
<strong>Task History:</strong> see a list of past tasks.
</p>
</li>
<li>
<p>
<strong>Subscription:</strong> upload a subscription key, and generate a system report for
   use in support cases.
</p>
</li>
</ul></div>
</div>
<div class="sect3">
<h4 id="_guests">4.4.3. Guests
 <a class="headerlink" href="#_guests" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-qemu-summary.png">
<img src="images/screenshot/gui-qemu-summary.png" alt="screenshot/gui-qemu-summary.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>There are two different kinds of guests and both can be converted to a template.
One of them is a Kernel-based Virtual Machine (KVM) and the other is a Linux Container (LXC).
Navigation for these are mostly the same; only some options are different.</p></div>
<div class="paragraph">
<p>To access the various guest management interfaces, select a VM or container from
the menu on the left.</p></div>
<div class="paragraph">
<p>The header contains commands for items such as power management, migration,
console access and type, cloning, HA, and help.
Some of these buttons contain drop-down menus, for example, <em>Shutdown</em> also contains
other power options, and <em>Console</em> contains the different console types:
<em>SPICE</em>, <em>noVNC</em> and <em>xterm.js</em>.</p></div>
<div class="paragraph">
<p>The panel on the right contains an interface for whatever item is selected from
the menu on the left.</p></div>
<div class="paragraph">
<p>The available interfaces are as follows.</p></div>
<div class="ulist"><ul>
<li>
<p>
<strong>Summary:</strong> provides a brief overview of the VM’s activity and a <span class="monospaced">Notes</span> field
  for <a href="#markdown_basics">Markdown syntax</a> comments.
</p>
</li>
<li>
<p>
<strong>Console:</strong> access to an interactive console for the VM/container.
</p>
</li>
<li>
<p>
<strong>(KVM)Hardware:</strong> define the hardware available to the KVM VM.
</p>
</li>
<li>
<p>
<strong>(LXC)Resources:</strong> define the system resources available to the LXC.
</p>
</li>
<li>
<p>
<strong>(LXC)Network:</strong> configure a container’s network settings.
</p>
</li>
<li>
<p>
<strong>(LXC)DNS:</strong> configure a container’s DNS settings.
</p>
</li>
<li>
<p>
<strong>Options:</strong> manage guest options.
</p>
</li>
<li>
<p>
<strong>Task History:</strong> view all previous tasks related to the selected guest.
</p>
</li>
<li>
<p>
<strong>(KVM) Monitor:</strong> an interactive communication interface to the KVM process.
</p>
</li>
<li>
<p>
<strong>Backup:</strong> create and restore system backups.
</p>
</li>
<li>
<p>
<strong>Replication:</strong> view and manage the replication jobs for the selected guest.
</p>
</li>
<li>
<p>
<strong>Snapshots:</strong> create and restore VM snapshots.
</p>
</li>
<li>
<p>
<strong>Firewall:</strong> configure the firewall on the VM level.
</p>
</li>
<li>
<p>
<strong>Permissions:</strong> manage permissions for the selected guest.
</p>
</li>
</ul></div>
</div>
<div class="sect3">
<h4 id="_storage">4.4.4. Storage
 <a class="headerlink" href="#_storage" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-storage-summary-local.png">
<img src="images/screenshot/gui-storage-summary-local.png" alt="screenshot/gui-storage-summary-local.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>As with the guest interface, the interface for storage consists of a menu on the
left for certain storage elements and an interface on the right to manage
these elements.</p></div>
<div class="paragraph">
<p>In this view we have a two partition split-view.
On the left side we have the storage options
and on the right side the content of the selected option will be shown.</p></div>
<div class="ulist"><ul>
<li>
<p>
<strong>Summary:</strong> shows important information about the storage, such as the type,
   usage, and content which it stores.
</p>
</li>
<li>
<p>
<strong>Content:</strong> a menu item for each content type which the storage
   stores, for example, Backups, ISO Images, CT Templates.
</p>
</li>
<li>
<p>
<strong>Permissions:</strong> manage permissions for the storage.
</p>
</li>
</ul></div>
</div>
<div class="sect3">
<h4 id="_pools">4.4.5. Pools
 <a class="headerlink" href="#_pools" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-pool-summary-development.png">
<img src="images/screenshot/gui-pool-summary-development.png" alt="screenshot/gui-pool-summary-development.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>Again, the pools view comprises two partitions: a menu on the left,
and the corresponding interfaces for each menu item on the right.</p></div>
<div class="ulist"><ul>
<li>
<p>
<strong>Summary:</strong> shows a description of the pool.
</p>
</li>
<li>
<p>
<strong>Members:</strong> display and manage pool members (guests and storage).
</p>
</li>
<li>
<p>
<strong>Permissions:</strong> manage the permissions for the pool.
</p>
</li>
</ul></div>
</div>
</div>
<div class="sect2">
<h3 id="_tags">
<span>4.5. Tags</span>
 <a class="headerlink" href="#_tags" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-qemu-summary-tags-edit.png">
<img src="images/screenshot/gui-qemu-summary-tags-edit.png" alt="screenshot/gui-qemu-summary-tags-edit.png" width="250" style="padding: 0 10px 0 0;float:left;"></a>
<p>For organizational purposes, it is possible to set <span class="monospaced">tags</span> for guests.
Currently, these only provide informational value to users.
Tags are displayed in two places in the web interface: in the <span class="monospaced">Resource Tree</span> and
in the status line when a guest is selected.</p></div>
<div class="paragraph">
<p>Tags can be added, edited, and removed in the status line of the guest by
clicking on the <span class="monospaced">pencil</span> icon. You can add multiple tags by pressing the <span class="monospaced">+</span>
button and remove them by pressing the <span class="monospaced">-</span> button. To save or cancel the changes,
you can use the <span class="monospaced">✓</span> and <span class="monospaced">x</span> button respectively.</p></div>
<div class="paragraph">
<p>Tags can also be set via the CLI, where multiple tags are separated by semicolons.
For example:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># qm set ID --tags myfirsttag;mysecondtag</pre>
</div></div>
<div class="sect3">
<h4 id="_style_configuration">4.5.1. Style Configuration
 <a class="headerlink" href="#_style_configuration" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-datacenter-tag-style.png">
<img src="images/screenshot/gui-datacenter-tag-style.png" alt="screenshot/gui-datacenter-tag-style.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>By default, the tag colors are derived from their text in a deterministic way.
The color, shape in the resource tree, and case-sensitivity, as well as how tags
are sorted, can be customized. This can be done via the web interface under
<em>Datacenter → Options → Tag Style Override</em>. Alternatively, this can be done
via the CLI. For example:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># pvesh set /cluster/options --tag-style color-map=example:000000:FFFFFF</pre>
</div></div>
<div class="paragraph">
<p>sets the background color of the tag <span class="monospaced">example</span> to black (#000000) and the text
color to white (#FFFFFF).</p></div>
</div>
<div class="sect3">
<h4 id="_permissions">4.5.2. Permissions
 <a class="headerlink" href="#_permissions" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-datacenter-options.png">
<img src="images/screenshot/gui-datacenter-options.png" alt="screenshot/gui-datacenter-options.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>By default, users with the privilege <span class="monospaced">VM.Config.Options</span> on a guest (<span class="monospaced">/vms/ID</span>)
can set any tags they want (see
<a href="#pveum_permission_management">Permission Management</a>). If you want to
restrict this behavior, appropriate permissions can be set under
<em>Datacenter → Options → User Tag Access</em>:</p></div>
<div class="ulist"><ul>
<li>
<p>
<span class="monospaced">free</span>: users are not restricted in setting tags (Default)
</p>
</li>
<li>
<p>
<span class="monospaced">list</span>: users can set tags based on a predefined list of tags
</p>
</li>
<li>
<p>
<span class="monospaced">existing</span>: like list but users can also use already existing tags
</p>
</li>
<li>
<p>
<span class="monospaced">none</span>: users are restricted from using tags
</p>
</li>
</ul></div>
<div class="paragraph">
<p>The same can also be done via the CLI.</p></div>
<div class="paragraph">
<p>Note that a user with the <span class="monospaced">Sys.Modify</span> privileges on <span class="monospaced">/</span> is always able to set
or delete any tags, regardless of the settings here. Additionally, there is a
configurable list of <span class="monospaced">registered tags</span> which can only be added and removed by
users with the privilege <span class="monospaced">Sys.Modify</span> on <span class="monospaced">/</span>. The list of registered tags can be
edited under <em>Datacenter → Options → Registered Tags</em> or via the CLI.</p></div>
<div class="paragraph">
<p>For more details on the exact options and how to invoke them in the CLI, see
<a href="#datacenter_configuration_file">Datacenter Configuration</a>.</p></div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="chapter_pvecm">
5. Cluster Manager
 <a class="headerlink" href="#chapter_pvecm" title="Permalink to this heading"></a>
</h2>
<div class="sectionbody">
<div class="paragraph">
<p>The Proxmox VE cluster manager <span class="monospaced">pvecm</span> is a tool to create a group of
physical servers. Such a group is called a <strong>cluster</strong>. We use the
<a href="http://www.corosync.org">Corosync Cluster Engine</a> for reliable group
communication. There’s no explicit limit for the number of nodes in a cluster.
In practice, the actual possible node count may be limited by the host and
network performance. Currently (2021), there are reports of clusters (using
high-end enterprise hardware) with over 50 nodes in production.</p></div>
<div class="paragraph">
<p><span class="monospaced">pvecm</span> can be used to create a new cluster, join nodes to a cluster,
leave the cluster, get status information, and do various other cluster-related
tasks. The <strong>P</strong>rox<strong>m</strong>o<strong>x</strong> <strong>C</strong>luster <strong>F</strong>ile <strong>S</strong>ystem (“pmxcfs”)
is used to transparently distribute the cluster configuration to all cluster
nodes.</p></div>
<div class="paragraph">
<p>Grouping nodes into a cluster has the following advantages:</p></div>
<div class="ulist"><ul>
<li>
<p>
Centralized, web-based management
</p>
</li>
<li>
<p>
Multi-master clusters: each node can do all management tasks
</p>
</li>
<li>
<p>
Use of <span class="monospaced">pmxcfs</span>, a database-driven file system, for storing configuration
  files, replicated in real-time on all nodes using <span class="monospaced">corosync</span>
</p>
</li>
<li>
<p>
Easy migration of virtual machines and containers between physical
  hosts
</p>
</li>
<li>
<p>
Fast deployment
</p>
</li>
<li>
<p>
Cluster-wide services like firewall and HA
</p>
</li>
</ul></div>
<div class="sect2">
<h3 id="_requirements">
<span>5.1. Requirements</span>
 <a class="headerlink" href="#_requirements" title="Permalink to this heading"></a>
</h3>
<div class="ulist"><ul>
<li>
<p>
All nodes must be able to connect to each other via UDP ports 5405-5412
 for corosync to work.
</p>
</li>
<li>
<p>
Date and time must be synchronized.
</p>
</li>
<li>
<p>
An SSH tunnel on TCP port 22 between nodes is required.
</p>
</li>
<li>
<p>
If you are interested in High Availability, you need to have at
  least three nodes for reliable quorum. All nodes should have the
  same version.
</p>
</li>
<li>
<p>
We recommend a dedicated NIC for the cluster traffic, especially if
  you use shared storage.
</p>
</li>
<li>
<p>
The root password of a cluster node is required for adding nodes.
</p>
</li>
<li>
<p>
Online migration of virtual machines is only supported when nodes have CPUs
  from the same vendor. It might work otherwise, but this is never guaranteed.
</p>
</li>
</ul></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">It is not possible to mix Proxmox VE 3.x and earlier with Proxmox VE 4.X cluster
nodes.</td>
</tr></tbody></table>
</div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">While it’s possible to mix Proxmox VE 4.4 and Proxmox VE 5.0 nodes, doing so is
not supported as a production configuration and should only be done temporarily,
during an upgrade of the whole cluster from one major version to another.</td>
</tr></tbody></table>
</div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Running a cluster of Proxmox VE 6.x with earlier versions is not possible. The
cluster protocol (corosync) between Proxmox VE 6.x and earlier versions changed
fundamentally. The corosync 3 packages for Proxmox VE 5.4 are only intended for the
upgrade procedure to Proxmox VE 6.0.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect2">
<h3 id="_preparing_nodes">
<span>5.2. Preparing Nodes</span>
 <a class="headerlink" href="#_preparing_nodes" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>First, install Proxmox VE on all nodes. Make sure that each node is
installed with the final hostname and IP configuration. Changing the
hostname and IP is not possible after cluster creation.</p></div>
<div class="paragraph">
<p>While it’s common to reference all node names and their IPs in <span class="monospaced">/etc/hosts</span> (or
make their names resolvable through other means), this is not necessary for a
cluster to work. It may be useful however, as you can then connect from one node
to another via SSH, using the easier to remember node name (see also
<a href="#pvecm_corosync_addresses">Link Address Types</a>). Note that we always
recommend referencing nodes by their IP addresses in the cluster configuration.</p></div>
</div>
<div class="sect2">
<h3 id="pvecm_create_cluster">
<span>5.3. Create a Cluster</span>
 <a class="headerlink" href="#pvecm_create_cluster" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>You can either create a cluster on the console (login via <span class="monospaced">ssh</span>), or through
the API using the Proxmox VE web interface (<em>Datacenter → Cluster</em>).</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Use a unique name for your cluster. This name cannot be changed later.
The cluster name follows the same rules as node names.</td>
</tr></tbody></table>
</div>
<div class="sect3">
<h4 id="pvecm_cluster_create_via_gui">5.3.1. Create via Web GUI
 <a class="headerlink" href="#pvecm_cluster_create_via_gui" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-cluster-create.png">
<img src="images/screenshot/gui-cluster-create.png" alt="screenshot/gui-cluster-create.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>Under <em>Datacenter → Cluster</em>, click on <strong>Create Cluster</strong>. Enter the cluster
name and select a network connection from the drop-down list to serve as the
main cluster network (Link 0). It defaults to the IP resolved via the node’s
hostname.</p></div>
<div class="paragraph">
<p>As of Proxmox VE 6.2, up to 8 fallback links can be added to a cluster. To add a
redundant link, click the <em>Add</em> button and select a link number and IP address
from the respective fields. Prior to Proxmox VE 6.2, to add a second link as
fallback, you can select the <em>Advanced</em> checkbox and choose an additional
network interface (Link 1, see also <a href="#pvecm_redundancy">Corosync Redundancy</a>).</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Ensure that the network selected for cluster communication is not used for
any high traffic purposes, like network storage or live-migration.
While the cluster network itself produces small amounts of data, it is very
sensitive to latency. Check out full
<a href="#pvecm_cluster_network_requirements">cluster network requirements</a>.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="pvecm_cluster_create_via_cli">5.3.2. Create via the Command Line
 <a class="headerlink" href="#pvecm_cluster_create_via_cli" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Login via <span class="monospaced">ssh</span> to the first Proxmox VE node and run the following command:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre> hp1# pvecm create CLUSTERNAME</pre>
</div></div>
<div class="paragraph">
<p>To check the state of the new cluster use:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre> hp1# pvecm status</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_multiple_clusters_in_the_same_network">5.3.3. Multiple Clusters in the Same Network
 <a class="headerlink" href="#_multiple_clusters_in_the_same_network" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>It is possible to create multiple clusters in the same physical or logical
network. In this case, each cluster must have a unique name to avoid possible
clashes in the cluster communication stack. Furthermore, this helps avoid human
confusion by making clusters clearly distinguishable.</p></div>
<div class="paragraph">
<p>While the bandwidth requirement of a corosync cluster is relatively low, the
latency of packages and the package per second (PPS) rate is the limiting
factor. Different clusters in the same network can compete with each other for
these resources, so it may still make sense to use separate physical network
infrastructure for bigger clusters.</p></div>
</div>
</div>
<div class="sect2">
<h3 id="pvecm_join_node_to_cluster">
<span>5.4. Adding Nodes to the Cluster</span>
 <a class="headerlink" href="#pvecm_join_node_to_cluster" title="Permalink to this heading"></a>
</h3>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Caution" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKdUlEQVRoge1Ze1AV1x3+zt279wEi
DWCYGzRVktqa1MEmmtbWR22ncXxUrTDWV/5IqG2wUAUfmUwSkk6V+EAJCEQC6figwZBqZtRxqukf
tebRZBpFG1S0hiRErwiaKK977+6eX//YPXt37+UiIJlMZnpmdnb33PP4vt97z2VEhG9yc3zdAO60
/Z/A192+8QScX8Wifr+fWltbzffU1FT4fD72Vew15ASampqovr4eBw8eNPvGjRuHzMxMmj9//tCT
IKIhu958801yuVwEoNdr48aNNJT7EdHQEdi/fz/JshwTvLiKioqGlMSQLFJfX0+MMRvQsWNSqXLz
H2jez78fRWLTpk1DRuKOF6irqyOn02kDOP8XGdTz+VFSP91Hatu79NRvZ0SR2LJly5CQuKPJtbW1
JEmSDdivHp1AwctHSblYSsqFElLOF5PavIvWZU+LIrF169Y7JjHoibt3744CnzXrIQpeOUbKhVJS
mkpIPV9MyrnNpJwtIuVSNa15/MdRJLZt23ZHJBjRwIu5N954gxYvXgzOudm3aO5E1L5SBNZ1Hoxr
INLAyH6HJxVP/WkXtu9+37ZeSUkJVq9ePagQO+BMXFNTEwV+ybxHsPeVIjg6zwOkgSWNhuOuUUBC
KhxJY0AggDSgpwWbn1mO1csftq1ZUFCA0tLSQZXFA0pkVVVVlJOTA6vWsmY9hF07N0AS4OU48LgR
WLO+FPFxboxMS8G5hlPYmPMI4jwM1P0Jtj67DJyrKHvtNADdjPPz8+FwOCgvL29gmuivrVVUVESF
yqxZD1PoyjFSL+7QHbapmJTml+n9EzVRtl676ZekNKwl5VQ+KSfzKHR2E+UutodYxhiVl5cPyCf6
ZUJlZWWUm5trk/wTi6bgtVdf1M2Gq2DEwUgDNBWhUDBqjfSRwwGoIOEP3c3Y/uxirFw0zibMvLw8
VFZW9tucbkugoqKCVq1aZQO/YslUVJU+D9bVBIDroLgK4hygEDweybYGY8AD6d8yftfASL+j+2O8
9NwiPJk51kYiNzcXO3fu7BeJPgkUFxdTbm6ure/JZdNRub0QrOMcGFcBrgLgAHEAGqD2ID7OZZsz
Ji0RcR6HAV4DkaqPJxXouoSywkz8buF9NhIrV65EVVXVbUnEJHDixAlat26drS9n+U+xo/hZMMNh
yZA+uAbAIKOGMCJlmG3eA/clAYaJEamGBlRdC6QBXZewo3AhVswfbSORk5ODDz/8sE8SvRLw+/1U
XV0dBb5MgOe6LUNIEqpOwgCUlOCESw4HuAfG3KX7B2kAGaTJ0Jro77yI8ucX4DfzRtlIHD58GH6/
PyaJXgk0NjZi79695vuUiffrkr91Vt8cHAwc4FZJGiZEKqDcwsi0ZHP+d749POy8MS7iGtBxARWF
8/CDsQnm3BdeeAHWj6N+EYhspSVbgI5zYYkZGmAwzIEbwA0tUM9NfO+7aeb89HuGRQBWLcSNZxj3
zvOYMzk5NpjBEHi99mXD5lXdjqGDJh42h7CEOXhnK3448X5z/shUb1jSpIEM8yFuzOUcjAsNcnR0
KXdG4MEHH8TEiRPN9z0H3sfRE41m+GNcs9i0LjkGTTcrUoGem5g7/V5z/ohElw4O+jxGYQ3qJqmB
jN/+cbINtcfCJvPYY4+hr3qtVwI+n49lZmaa71fbbuH3hX/FycbPQKSCoIIMQEQcehjVo5Gw8/F3
3cSvH03H2HuHY1icQzcvYWKGIMiITMwQwjtnruPpqk9w/ZZq7p2RkYEJEyaAYrCIWY36/X7KyMhA
W1ub2XfP3Qmo3ZKFaQ+P0gmYIdSoOkU4lWSwxGRAdqHL/xm86k27w0blAw3/PncD6yo/xrsfdZj7
paSkoLGxESNGjAgDZsxWK8X0AZ/Px44fP460tLAzXrnWgUX5r+PYuxf1mG/Gf80sofWy2YuWbg/+
uONtPLHhPRw+0WIHL8hbwG95rcUGPiEhAYcOHUJSUhI456YZRWqiz+8BIqLm5mZMmjQJN27cMPsT
E9zYtWE25k4bYzh3mABIBRwOHPgPx+LH/6yDiXPi6pHZcEpk+I5qmtGpCzdQecCPPUevmevLsowj
R45gypQpcDgc5mVowKaJmBoQTEePHo2GhgaMGxcuum52BLHsqUOoP3oWgLB9Sz7QgpgxJoThw/SS
IutnaXBKZEpcgD998Uvs+3ubDbzT6URNTQ0mTZoEVVWhaRo459A0TeCy4etVA+JH8RvnHK2trZgz
Zw5Onz5tjnPJDrxSOAPLZt1nSVIqwDmIVJw6dx1nLt3A3J/cjZThkhE29cvf3o2KA59ja91lcz3G
GDZv3oylS5fC7XbD6XTC6XRCkiRIktS7Jm5HQLDXNA3Xr19HVlYWPvjgA4vEGLav+RHmTxsJX4rb
8AsO4kbOIG6GzTD4HvzlmB/PVH9q2zc/Px/Z2dnwer1wu92QZRkulwuSJEGWZZMEY8wkEGVCVvCC
AOcciqIgPj4e+/btw9SpU83xqkpYteU9HHm7Bf62rnC2RjikisTFSMPV9m4ceucannvVDn7p0qVY
uHAhgsEggsEgQqFQlAlZnVm0PjOxIKFpGogIqqrC5XKhuroa06dPt4wDcl78F+qPNeNKe6cRoTjs
ZYOKK+3dON7wBQrKm2H5pMbMmTOxZMkSBAIB9PT0oKenB4FAAIqiQFEUU4hRRyq3I2AlIZ4553A6
nSgvL8fs2bNtY9eVNaDub58YmhDZWpe+v70bJ5tu4clt/4WihqU4efJkrFixAkRkAg4Gg1AUBaqq
muDF/pGtX7WQNXeIZ0mSUFRUhAULFtjGPl3ZiJdev4QrbZ16zUQqrrZ342JLN5ZvaEJ3ICz68ePH
IycnB5qmmZIW4EOhUJTZWEKo+Rx1KsEYY9Zk4XA4IEmSKXnOuRkVJElCYWEhvF4v6urqzDVK9n2M
zu4Qnl5+L9q/7MHltgBWbr9kA5+eno7c3FzIsoyI5GrTeCzgosU8VmGMmSGLiEwSsixH2WNBQQEA
2EhUH/wc7330BaaOH479/2zHtS/CFabP58OaNWvg9Xp1EE4nZFmGLMtwu93wer1wuVy2ECpJko2I
iTNWJrZGI+HEwi6FisUVCAQQCASwZ88eVFZW9lk9JicnY/369fD5fPB4PCbo+Ph4k4TH44HL5YLX
64XX6zXH9EKE9aUBRkTEGLNJP1Kd4nI4HMjOzkZ8fDyKi4ttJ3eiJSYmYu3atRg1apQJzuv1wuPx
wOPxwOFwmEBFHnA6nXC5XGYSiywlbns2SnoDANN0hMNZnS0UCpkaOnPmDN566y20tLSY2ktLS8OM
GTOQkpICt9uNuLg4M2EJ8CJZSZJkZmFBwkrAWpH2+3BXEBEkRIhTVdVMOCL5qKqKy5cvQ1EUaJoG
VVWRnKx/Jlpt3eVy2QCKu9C68AFr9o0spwd0Oh1ZYggCIlNaiQktiXdhZgKQABcp3ci7LMtM7B0J
fsAErJoAYItG1mdr+hfvVlACtHDIXkploYXbHvQOikDEe6/ZmjFmAhf9ZvJxOk2Qol88W4kYz32S
GNQfHLG+TyMTkAVEr88xQfVnkBg7GAKixSIykDYQsL21/wFkW/B5QqT9lwAAAABJRU5ErkJggg==">
</td>
<td class="content">All existing configuration in <span class="monospaced">/etc/pve</span> is overwritten when joining a
cluster. In particular, a joining node cannot hold any guests, since guest IDs
could otherwise conflict, and the node will inherit the cluster’s storage
configuration. To join a node with existing guest, as a workaround, you can
create a backup of each guest (using <span class="monospaced">vzdump</span>) and restore it under a different
ID after joining. If the node’s storage layout differs, you will need to re-add
the node’s storages, and adapt each storage’s node restriction to reflect on
which nodes the storage is actually available.</td>
</tr></tbody></table>
</div>
<div class="sect3">
<h4 id="_join_node_to_cluster_via_gui">5.4.1. Join Node to Cluster via GUI
 <a class="headerlink" href="#_join_node_to_cluster_via_gui" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-cluster-join-information.png">
<img src="images/screenshot/gui-cluster-join-information.png" alt="screenshot/gui-cluster-join-information.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>Log in to the web interface on an existing cluster node. Under <em>Datacenter →
Cluster</em>, click the <strong>Join Information</strong> button at the top. Then, click on the
button <strong>Copy Information</strong>. Alternatively, copy the string from the <em>Information</em>
field manually.</p></div>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-cluster-join.png">
<img src="images/screenshot/gui-cluster-join.png" alt="screenshot/gui-cluster-join.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>Next, log in to the web interface on the node you want to add.
Under <em>Datacenter → Cluster</em>, click on <strong>Join Cluster</strong>. Fill in the
<em>Information</em> field with the <em>Join Information</em> text you copied earlier.
Most settings required for joining the cluster will be filled out
automatically. For security reasons, the cluster password has to be entered
manually.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">To enter all required data manually, you can disable the <em>Assisted Join</em>
checkbox.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>After clicking the <strong>Join</strong> button, the cluster join process will start
immediately. After the node has joined the cluster, its current node certificate
will be replaced by one signed from the cluster certificate authority (CA).
This means that the current session will stop working after a few seconds. You
then might need to force-reload the web interface and log in again with the
cluster credentials.</p></div>
<div class="paragraph">
<p>Now your node should be visible under <em>Datacenter → Cluster</em>.</p></div>
</div>
<div class="sect3">
<h4 id="_join_node_to_cluster_via_command_line">5.4.2. Join Node to Cluster via Command Line
 <a class="headerlink" href="#_join_node_to_cluster_via_command_line" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Log in to the node you want to join into an existing cluster via <span class="monospaced">ssh</span>.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre> # pvecm add IP-ADDRESS-CLUSTER</pre>
</div></div>
<div class="paragraph">
<p>For <span class="monospaced">IP-ADDRESS-CLUSTER</span>, use the IP or hostname of an existing cluster node.
An IP address is recommended (see <a href="#pvecm_corosync_addresses">Link Address Types</a>).</p></div>
<div class="paragraph">
<p>To check the state of the cluster use:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre> # pvecm status</pre>
</div></div>
<div class="listingblock">
<div class="title">Cluster status after adding 4 nodes</div>
<div class="content monospaced">
<pre> # pvecm status
Cluster information
~~~~~~~~~~~~~~~~~~~
Name:             prod-central
Config Version:   3
Transport:        knet
Secure auth:      on

Quorum information
~~~~~~~~~~~~~~~~~~
Date:             Tue Sep 14 11:06:47 2021
Quorum provider:  corosync_votequorum
Nodes:            4
Node ID:          0x00000001
Ring ID:          1.1a8
Quorate:          Yes

Votequorum information
~~~~~~~~~~~~~~~~~~~~~~
Expected votes:   4
Highest expected: 4
Total votes:      4
Quorum:           3
Flags:            Quorate

Membership information
~~~~~~~~~~~~~~~~~~~~~~
    Nodeid      Votes Name
0x00000001          1 192.168.15.91
0x00000002          1 192.168.15.92 (local)
0x00000003          1 192.168.15.93
0x00000004          1 192.168.15.94</pre>
</div></div>
<div class="paragraph">
<p>If you only want a list of all nodes, use:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre> # pvecm nodes</pre>
</div></div>
<div class="listingblock">
<div class="title">List nodes in a cluster</div>
<div class="content monospaced">
<pre> # pvecm nodes

Membership information
~~~~~~~~~~~~~~~~~~~~~~
    Nodeid      Votes Name
         1          1 hp1
         2          1 hp2 (local)
         3          1 hp3
         4          1 hp4</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="pvecm_adding_nodes_with_separated_cluster_network">5.4.3. Adding Nodes with Separated Cluster Network
 <a class="headerlink" href="#pvecm_adding_nodes_with_separated_cluster_network" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>When adding a node to a cluster with a separated cluster network, you need to
use the <em>link0</em> parameter to set the nodes address on that network:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt><span style="font-style: italic"><span style="color: #9A1900"># pvecm add IP-ADDRESS-CLUSTER --link0 LOCAL-IP-ADDRESS-LINK0</span></span></tt></pre></div></div>
<div class="paragraph">
<p>If you want to use the built-in <a href="#pvecm_redundancy">redundancy</a> of the
Kronosnet transport layer, also use the <em>link1</em> parameter.</p></div>
<div class="paragraph">
<p>Using the GUI, you can select the correct interface from the corresponding
<em>Link X</em> fields in the <strong>Cluster Join</strong> dialog.</p></div>
</div>
</div>
<div class="sect2">
<h3 id="_remove_a_cluster_node">
<span>5.5. Remove a Cluster Node</span>
 <a class="headerlink" href="#_remove_a_cluster_node" title="Permalink to this heading"></a>
</h3>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Caution" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKdUlEQVRoge1Ze1AV1x3+zt279wEi
DWCYGzRVktqa1MEmmtbWR22ncXxUrTDWV/5IqG2wUAUfmUwSkk6V+EAJCEQC6figwZBqZtRxqukf
tebRZBpFG1S0hiRErwiaKK977+6eX//YPXt37+UiIJlMZnpmdnb33PP4vt97z2VEhG9yc3zdAO60
/Z/A192+8QScX8Wifr+fWltbzffU1FT4fD72Vew15ASampqovr4eBw8eNPvGjRuHzMxMmj9//tCT
IKIhu958801yuVwEoNdr48aNNJT7EdHQEdi/fz/JshwTvLiKioqGlMSQLFJfX0+MMRvQsWNSqXLz
H2jez78fRWLTpk1DRuKOF6irqyOn02kDOP8XGdTz+VFSP91Hatu79NRvZ0SR2LJly5CQuKPJtbW1
JEmSDdivHp1AwctHSblYSsqFElLOF5PavIvWZU+LIrF169Y7JjHoibt3744CnzXrIQpeOUbKhVJS
mkpIPV9MyrnNpJwtIuVSNa15/MdRJLZt23ZHJBjRwIu5N954gxYvXgzOudm3aO5E1L5SBNZ1Hoxr
INLAyH6HJxVP/WkXtu9+37ZeSUkJVq9ePagQO+BMXFNTEwV+ybxHsPeVIjg6zwOkgSWNhuOuUUBC
KhxJY0AggDSgpwWbn1mO1csftq1ZUFCA0tLSQZXFA0pkVVVVlJOTA6vWsmY9hF07N0AS4OU48LgR
WLO+FPFxboxMS8G5hlPYmPMI4jwM1P0Jtj67DJyrKHvtNADdjPPz8+FwOCgvL29gmuivrVVUVESF
yqxZD1PoyjFSL+7QHbapmJTml+n9EzVRtl676ZekNKwl5VQ+KSfzKHR2E+UutodYxhiVl5cPyCf6
ZUJlZWWUm5trk/wTi6bgtVdf1M2Gq2DEwUgDNBWhUDBqjfSRwwGoIOEP3c3Y/uxirFw0zibMvLw8
VFZW9tucbkugoqKCVq1aZQO/YslUVJU+D9bVBIDroLgK4hygEDweybYGY8AD6d8yftfASL+j+2O8
9NwiPJk51kYiNzcXO3fu7BeJPgkUFxdTbm6ure/JZdNRub0QrOMcGFcBrgLgAHEAGqD2ID7OZZsz
Ji0RcR6HAV4DkaqPJxXouoSywkz8buF9NhIrV65EVVXVbUnEJHDixAlat26drS9n+U+xo/hZMMNh
yZA+uAbAIKOGMCJlmG3eA/clAYaJEamGBlRdC6QBXZewo3AhVswfbSORk5ODDz/8sE8SvRLw+/1U
XV0dBb5MgOe6LUNIEqpOwgCUlOCESw4HuAfG3KX7B2kAGaTJ0Jro77yI8ucX4DfzRtlIHD58GH6/
PyaJXgk0NjZi79695vuUiffrkr91Vt8cHAwc4FZJGiZEKqDcwsi0ZHP+d749POy8MS7iGtBxARWF
8/CDsQnm3BdeeAHWj6N+EYhspSVbgI5zYYkZGmAwzIEbwA0tUM9NfO+7aeb89HuGRQBWLcSNZxj3
zvOYMzk5NpjBEHi99mXD5lXdjqGDJh42h7CEOXhnK3448X5z/shUb1jSpIEM8yFuzOUcjAsNcnR0
KXdG4MEHH8TEiRPN9z0H3sfRE41m+GNcs9i0LjkGTTcrUoGem5g7/V5z/ohElw4O+jxGYQ3qJqmB
jN/+cbINtcfCJvPYY4+hr3qtVwI+n49lZmaa71fbbuH3hX/FycbPQKSCoIIMQEQcehjVo5Gw8/F3
3cSvH03H2HuHY1icQzcvYWKGIMiITMwQwjtnruPpqk9w/ZZq7p2RkYEJEyaAYrCIWY36/X7KyMhA
W1ub2XfP3Qmo3ZKFaQ+P0gmYIdSoOkU4lWSwxGRAdqHL/xm86k27w0blAw3/PncD6yo/xrsfdZj7
paSkoLGxESNGjAgDZsxWK8X0AZ/Px44fP460tLAzXrnWgUX5r+PYuxf1mG/Gf80sofWy2YuWbg/+
uONtPLHhPRw+0WIHL8hbwG95rcUGPiEhAYcOHUJSUhI456YZRWqiz+8BIqLm5mZMmjQJN27cMPsT
E9zYtWE25k4bYzh3mABIBRwOHPgPx+LH/6yDiXPi6pHZcEpk+I5qmtGpCzdQecCPPUevmevLsowj
R45gypQpcDgc5mVowKaJmBoQTEePHo2GhgaMGxcuum52BLHsqUOoP3oWgLB9Sz7QgpgxJoThw/SS
IutnaXBKZEpcgD998Uvs+3ubDbzT6URNTQ0mTZoEVVWhaRo459A0TeCy4etVA+JH8RvnHK2trZgz
Zw5Onz5tjnPJDrxSOAPLZt1nSVIqwDmIVJw6dx1nLt3A3J/cjZThkhE29cvf3o2KA59ja91lcz3G
GDZv3oylS5fC7XbD6XTC6XRCkiRIktS7Jm5HQLDXNA3Xr19HVlYWPvjgA4vEGLav+RHmTxsJX4rb
8AsO4kbOIG6GzTD4HvzlmB/PVH9q2zc/Px/Z2dnwer1wu92QZRkulwuSJEGWZZMEY8wkEGVCVvCC
AOcciqIgPj4e+/btw9SpU83xqkpYteU9HHm7Bf62rnC2RjikisTFSMPV9m4ceucannvVDn7p0qVY
uHAhgsEggsEgQqFQlAlZnVm0PjOxIKFpGogIqqrC5XKhuroa06dPt4wDcl78F+qPNeNKe6cRoTjs
ZYOKK+3dON7wBQrKm2H5pMbMmTOxZMkSBAIB9PT0oKenB4FAAIqiQFEUU4hRRyq3I2AlIZ4553A6
nSgvL8fs2bNtY9eVNaDub58YmhDZWpe+v70bJ5tu4clt/4WihqU4efJkrFixAkRkAg4Gg1AUBaqq
muDF/pGtX7WQNXeIZ0mSUFRUhAULFtjGPl3ZiJdev4QrbZ16zUQqrrZ342JLN5ZvaEJ3ICz68ePH
IycnB5qmmZIW4EOhUJTZWEKo+Rx1KsEYY9Zk4XA4IEmSKXnOuRkVJElCYWEhvF4v6urqzDVK9n2M
zu4Qnl5+L9q/7MHltgBWbr9kA5+eno7c3FzIsoyI5GrTeCzgosU8VmGMmSGLiEwSsixH2WNBQQEA
2EhUH/wc7330BaaOH479/2zHtS/CFabP58OaNWvg9Xp1EE4nZFmGLMtwu93wer1wuVy2ECpJko2I
iTNWJrZGI+HEwi6FisUVCAQQCASwZ88eVFZW9lk9JicnY/369fD5fPB4PCbo+Ph4k4TH44HL5YLX
64XX6zXH9EKE9aUBRkTEGLNJP1Kd4nI4HMjOzkZ8fDyKi4ttJ3eiJSYmYu3atRg1apQJzuv1wuPx
wOPxwOFwmEBFHnA6nXC5XGYSiywlbns2SnoDANN0hMNZnS0UCpkaOnPmDN566y20tLSY2ktLS8OM
GTOQkpICt9uNuLg4M2EJ8CJZSZJkZmFBwkrAWpH2+3BXEBEkRIhTVdVMOCL5qKqKy5cvQ1EUaJoG
VVWRnKx/Jlpt3eVy2QCKu9C68AFr9o0spwd0Oh1ZYggCIlNaiQktiXdhZgKQABcp3ci7LMtM7B0J
fsAErJoAYItG1mdr+hfvVlACtHDIXkploYXbHvQOikDEe6/ZmjFmAhf9ZvJxOk2Qol88W4kYz32S
GNQfHLG+TyMTkAVEr88xQfVnkBg7GAKixSIykDYQsL21/wFkW/B5QqT9lwAAAABJRU5ErkJggg==">
</td>
<td class="content">Read the procedure carefully before proceeding, as it may
not be what you want or need.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>Move all virtual machines from the node. Ensure that you have made copies of any
local data or backups that you want to keep. In addition, make sure to remove
any scheduled replication jobs to the node to be removed.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Caution" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKdUlEQVRoge1Ze1AV1x3+zt279wEi
DWCYGzRVktqa1MEmmtbWR22ncXxUrTDWV/5IqG2wUAUfmUwSkk6V+EAJCEQC6figwZBqZtRxqukf
tebRZBpFG1S0hiRErwiaKK977+6eX//YPXt37+UiIJlMZnpmdnb33PP4vt97z2VEhG9yc3zdAO60
/Z/A192+8QScX8Wifr+fWltbzffU1FT4fD72Vew15ASampqovr4eBw8eNPvGjRuHzMxMmj9//tCT
IKIhu958801yuVwEoNdr48aNNJT7EdHQEdi/fz/JshwTvLiKioqGlMSQLFJfX0+MMRvQsWNSqXLz
H2jez78fRWLTpk1DRuKOF6irqyOn02kDOP8XGdTz+VFSP91Hatu79NRvZ0SR2LJly5CQuKPJtbW1
JEmSDdivHp1AwctHSblYSsqFElLOF5PavIvWZU+LIrF169Y7JjHoibt3744CnzXrIQpeOUbKhVJS
mkpIPV9MyrnNpJwtIuVSNa15/MdRJLZt23ZHJBjRwIu5N954gxYvXgzOudm3aO5E1L5SBNZ1Hoxr
INLAyH6HJxVP/WkXtu9+37ZeSUkJVq9ePagQO+BMXFNTEwV+ybxHsPeVIjg6zwOkgSWNhuOuUUBC
KhxJY0AggDSgpwWbn1mO1csftq1ZUFCA0tLSQZXFA0pkVVVVlJOTA6vWsmY9hF07N0AS4OU48LgR
WLO+FPFxboxMS8G5hlPYmPMI4jwM1P0Jtj67DJyrKHvtNADdjPPz8+FwOCgvL29gmuivrVVUVESF
yqxZD1PoyjFSL+7QHbapmJTml+n9EzVRtl676ZekNKwl5VQ+KSfzKHR2E+UutodYxhiVl5cPyCf6
ZUJlZWWUm5trk/wTi6bgtVdf1M2Gq2DEwUgDNBWhUDBqjfSRwwGoIOEP3c3Y/uxirFw0zibMvLw8
VFZW9tucbkugoqKCVq1aZQO/YslUVJU+D9bVBIDroLgK4hygEDweybYGY8AD6d8yftfASL+j+2O8
9NwiPJk51kYiNzcXO3fu7BeJPgkUFxdTbm6ure/JZdNRub0QrOMcGFcBrgLgAHEAGqD2ID7OZZsz
Ji0RcR6HAV4DkaqPJxXouoSywkz8buF9NhIrV65EVVXVbUnEJHDixAlat26drS9n+U+xo/hZMMNh
yZA+uAbAIKOGMCJlmG3eA/clAYaJEamGBlRdC6QBXZewo3AhVswfbSORk5ODDz/8sE8SvRLw+/1U
XV0dBb5MgOe6LUNIEqpOwgCUlOCESw4HuAfG3KX7B2kAGaTJ0Jro77yI8ucX4DfzRtlIHD58GH6/
PyaJXgk0NjZi79695vuUiffrkr91Vt8cHAwc4FZJGiZEKqDcwsi0ZHP+d749POy8MS7iGtBxARWF
8/CDsQnm3BdeeAHWj6N+EYhspSVbgI5zYYkZGmAwzIEbwA0tUM9NfO+7aeb89HuGRQBWLcSNZxj3
zvOYMzk5NpjBEHi99mXD5lXdjqGDJh42h7CEOXhnK3448X5z/shUb1jSpIEM8yFuzOUcjAsNcnR0
KXdG4MEHH8TEiRPN9z0H3sfRE41m+GNcs9i0LjkGTTcrUoGem5g7/V5z/ohElw4O+jxGYQ3qJqmB
jN/+cbINtcfCJvPYY4+hr3qtVwI+n49lZmaa71fbbuH3hX/FycbPQKSCoIIMQEQcehjVo5Gw8/F3
3cSvH03H2HuHY1icQzcvYWKGIMiITMwQwjtnruPpqk9w/ZZq7p2RkYEJEyaAYrCIWY36/X7KyMhA
W1ub2XfP3Qmo3ZKFaQ+P0gmYIdSoOkU4lWSwxGRAdqHL/xm86k27w0blAw3/PncD6yo/xrsfdZj7
paSkoLGxESNGjAgDZsxWK8X0AZ/Px44fP460tLAzXrnWgUX5r+PYuxf1mG/Gf80sofWy2YuWbg/+
uONtPLHhPRw+0WIHL8hbwG95rcUGPiEhAYcOHUJSUhI456YZRWqiz+8BIqLm5mZMmjQJN27cMPsT
E9zYtWE25k4bYzh3mABIBRwOHPgPx+LH/6yDiXPi6pHZcEpk+I5qmtGpCzdQecCPPUevmevLsowj
R45gypQpcDgc5mVowKaJmBoQTEePHo2GhgaMGxcuum52BLHsqUOoP3oWgLB9Sz7QgpgxJoThw/SS
IutnaXBKZEpcgD998Uvs+3ubDbzT6URNTQ0mTZoEVVWhaRo459A0TeCy4etVA+JH8RvnHK2trZgz
Zw5Onz5tjnPJDrxSOAPLZt1nSVIqwDmIVJw6dx1nLt3A3J/cjZThkhE29cvf3o2KA59ja91lcz3G
GDZv3oylS5fC7XbD6XTC6XRCkiRIktS7Jm5HQLDXNA3Xr19HVlYWPvjgA4vEGLav+RHmTxsJX4rb
8AsO4kbOIG6GzTD4HvzlmB/PVH9q2zc/Px/Z2dnwer1wu92QZRkulwuSJEGWZZMEY8wkEGVCVvCC
AOcciqIgPj4e+/btw9SpU83xqkpYteU9HHm7Bf62rnC2RjikisTFSMPV9m4ceucannvVDn7p0qVY
uHAhgsEggsEgQqFQlAlZnVm0PjOxIKFpGogIqqrC5XKhuroa06dPt4wDcl78F+qPNeNKe6cRoTjs
ZYOKK+3dON7wBQrKm2H5pMbMmTOxZMkSBAIB9PT0oKenB4FAAIqiQFEUU4hRRyq3I2AlIZ4553A6
nSgvL8fs2bNtY9eVNaDub58YmhDZWpe+v70bJ5tu4clt/4WihqU4efJkrFixAkRkAg4Gg1AUBaqq
muDF/pGtX7WQNXeIZ0mSUFRUhAULFtjGPl3ZiJdev4QrbZ16zUQqrrZ342JLN5ZvaEJ3ICz68ePH
IycnB5qmmZIW4EOhUJTZWEKo+Rx1KsEYY9Zk4XA4IEmSKXnOuRkVJElCYWEhvF4v6urqzDVK9n2M
zu4Qnl5+L9q/7MHltgBWbr9kA5+eno7c3FzIsoyI5GrTeCzgosU8VmGMmSGLiEwSsixH2WNBQQEA
2EhUH/wc7330BaaOH479/2zHtS/CFabP58OaNWvg9Xp1EE4nZFmGLMtwu93wer1wuVy2ECpJko2I
iTNWJrZGI+HEwi6FisUVCAQQCASwZ88eVFZW9lk9JicnY/369fD5fPB4PCbo+Ph4k4TH44HL5YLX
64XX6zXH9EKE9aUBRkTEGLNJP1Kd4nI4HMjOzkZ8fDyKi4ttJ3eiJSYmYu3atRg1apQJzuv1wuPx
wOPxwOFwmEBFHnA6nXC5XGYSiywlbns2SnoDANN0hMNZnS0UCpkaOnPmDN566y20tLSY2ktLS8OM
GTOQkpICt9uNuLg4M2EJ8CJZSZJkZmFBwkrAWpH2+3BXEBEkRIhTVdVMOCL5qKqKy5cvQ1EUaJoG
VVWRnKx/Jlpt3eVy2QCKu9C68AFr9o0spwd0Oh1ZYggCIlNaiQktiXdhZgKQABcp3ci7LMtM7B0J
fsAErJoAYItG1mdr+hfvVlACtHDIXkploYXbHvQOikDEe6/ZmjFmAhf9ZvJxOk2Qol88W4kYz32S
GNQfHLG+TyMTkAVEr88xQfVnkBg7GAKixSIykDYQsL21/wFkW/B5QqT9lwAAAABJRU5ErkJggg==">
</td>
<td class="content">Failure to remove replication jobs to a node before removing said node
will result in the replication job becoming irremovable. Especially note that
replication automatically switches direction if a replicated VM is migrated, so
by migrating a replicated VM from a node to be deleted, replication jobs will be
set up to that node automatically.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>In the following example, we will remove the node hp4 from the cluster.</p></div>
<div class="paragraph">
<p>Log in to a <strong>different</strong> cluster node (not hp4), and issue a <span class="monospaced">pvecm nodes</span>
command to identify the node ID to remove:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre> hp1# pvecm nodes

Membership information
~~~~~~~~~~~~~~~~~~~~~~
    Nodeid      Votes Name
         1          1 hp1 (local)
         2          1 hp2
         3          1 hp3
         4          1 hp4</pre>
</div></div>
<div class="paragraph">
<p>At this point, you must power off hp4 and ensure that it will not power on
again (in the network) with its current configuration.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Important" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAALa0lEQVRogdWZa2wc1RXHfzM7O/te
P9e1vXHSmEdjx3YeDkkaF6REKRRCEDSEFNkRjdSWSsgC2iqoRWqLQKiqIBg1NOQDiMeHtkQIQkRR
S9S4aWwgCQoUgl232KkT28J21l6vd3d2dx79sDuTXdtre03Uqlc62tl53Pv/n3PuOffcKxiGwf9z
E//XAL5sk65WR0a6AaDrOjMtKwgCoigiCAKCIAhXa9wvTcAEHhoc5KMjR7jw3nucf+MN67k/GCS4
YQOr77yTpl27kGUZm81mXC0ywlLnQDbwEwcO0P2b3yz4jV5RwZq2Nm5/5BH8fj+SJGGz2b4UiSUR
MAzD0DSNEwcO8MdHHln8d0AcsAUCfOe551h7yy04HA7sdvuSrVEwAV3XjdDgIC/v2cOl06dznqWA
JCAAMvn9MwlEgF1PP8037rsPt9uN3W5fkjUKIqDrujF+4QLP3XQTU0NDOc8UIAa4y8sJtrQQqK/H
7XQSfv99Pn/nHQRRxNB16/0EEAUatm9n76uv4vP5cDgcBZNYNIF84I0MEK20lIZvf5sNe/dSWlqK
1+vFxBEZHubT3/+evqNHmR4dJRWP55BevW0bra+8gt/vx+l0IknSokksikA+8ElgClixYwc3799P
ZWUlTqczbz+x0VFOPvEEQ6dPE/niCwxdt0jUbd1K68svU1xcXBCJBQnMBz5eWkrL/v2s++Y3KSsr
QxQXlxc7f/5zBjo7CQ8NoadSKKQn943t7dyyf79FYjHuNC8BwzAMVVV56Z57+OzNN3PAJ8vK+FZH
B2s2b55X6/nahRMnOP7TnxLOKEXJyO2/+hVf37uXoqIinE6nmfzyksirMjNUvvGjH+WAVwG1vJxb
Dhxg1bp1uFwuMwQWJCu3bWPN3r24SkoAcGbA/OXXv2bws8+IRqOkUqlZGX1RBAzDMHRd51+nTvHe
wYNX7gPTQMO+faxav56SkpIlgTdlU3s7TXv2IGUs6AGioRDH2tuZmJggFouhqirGPCzyEUBVVX7X
1pZzPwI0fP/7bNmzh7KysrzADh8+zP3338++ffs4ePBg3vdEUWTDD3/IsuZmRLsdW4bEpd5eOp99
lnA4jKIoaJqWl8QsAqb2Txw4kDNp40BxXR0bWlspLy/PC+qZZ55hdHQUSZIYHByko6ODxx57bBZw
cy3kq6xk6y9/ib+qCgQBmbQ7nT50iIs9PUSjUZLJJIZhzEliLgKMDQzwp5/9zLqnkQ51W3/xCwKB
AE6nMy+B3t5exsfH6ezspLe3l3g8zuHDh2cBz5bKpibWtrXhLi0FwA3owNsPP8zExATxeBxVVecy
QC4Bc+L+taNjluvU33031dddN6/2BUGgpKSE/v5+JicncweaA3i2tDz8MIHrr0eU0gsQBzDW10ff
yZNEIhESicScrpRDQNd1xgYG+OC3v7XuJQBXIMDWn/yEioqKeUGMjY0hCAKKoszS1MjIyLzfAmx/
4gmKli0DQcBJOmicfOopJicnicfjJoG5LWBq/+Szz+a8oABNe/bgdrvndR1BEKiqqiIejxOLxWYR
CAaDC0alqjVrqGxowFNWhkDalUKff07f3/5mWSFTLFksLAK6rjN+4UKO9lXAUV7O2tbWeaNOthZl
WZ4FHlh0aN3y4IPIHk+6L8AGvNfRQTgctuZCthVEU/u6rnPutddyBk0A9bt3W4us+QY2fbyurg6v
15vTzw033LBoAsHmZoLNzdgcDgBcQKi/n391dTE9PU0ymcyxgkmAVCrF+4cO5WjfFgiwfoGwOTOy
lJWVUVtba1mkUAKCILClvR1veTkA9owVPnzllZy8YFnA1H5/dzfTw8PWgziwcutWfD4fsiwvCNyU
xsZG3G43lZWVVl933XVXQQSq161j+ebNlhXcwBdnzxIOh60lhmkFEUDTNP7++uvWgBrp6mrNHBk3
H/BsAtFoFEdm8GAwyMaNGwsiIAgCjbt346uosKwgAp8cPWolNj1THInmsqHn6FGLQApY1thIxcqV
+Hy+WX6+0ETesWMHLpcLWZZ54IEHCgYvCAKrbrsNyem0JrQEXMzkhKzlBaKu61z+97+ZHhnJmbwr
tm2zJu5CWp8pra2tNDU1sXPnTtrb25dEQBAEvrplC57MXJCB8XPniEQixONxa6Uq6brOQHd3jvvo
QP2ttxZUpMxsL7zwwpK+AyxLXrt9Oxe6uxFEEXvGZc4fO0bVD35AMplM1wuapjGYtbugAp5AgKLq
aquu/W/JTEs37dqFqihWzSADk8PD1jzQNC3tQtmrThUINDbm+H4h0tfXx6OPPsodd9xBW1sbPT09
BQPPluu2b7fWRxIQ6usjFotZBCRN06wZbbZAfT1FRUWWKQtpzz//PMPDw/T09DA1NcXo6CjHjx+f
11Xma95AwHJjEdB0HUVRrhCYuRGrAiXBoFUqFtq6u7sZHx9nJBMUurq65uxnsX0Hm5v56A9/ANIJ
LXrpEolEwgqlUqZQyPlo6OOPlzx5ly1bxnBWQmxpackBW6hSBEC02axrpqasNZGmaek8kN0kYPDj
j1FVdUlz4Mknn6Surs4Cb5aUhYZiUy59+GGOgm3V1WiadiUPCILAV9autV5wAaM9PXQeOWJprBBp
aGjgxIkTpFIpOjs7aWxsXBJwUwZOnUKZmgLS2zl4vUjSlV1XURRFqtavv3IDKAVe/O53efPxx69a
SCxUBs+c4dV77+Vyfz+xy5eBdG3iX7ECm81m9S2JokiwuZmiujrCPT1AuqgOAqcef5zzL77Imt27
uemhhyipqVnYZ5cw8bPbR0eOcOall7g8MEB4aIhEJAJk9qOAwKZNyLJsHpQgRKNRIxQK0XPmDH/+
3vdQQ6G8na/YtIk199zDypYWVmzceFWAT1y8yMWzZzn/9tv0vfsudqeTlKIQHRvL2QSOAqU7d/K1
W2/l2muvpba2Nl3iJhIJIxKJMDw8TO/Zs3zw4x+jTUzkHVCw2fCUleHw+QiuXYu/upqqhgb8VVUY
hkFNczPFweCs7z556y3r+vyxYwiiyD+PHwdBwNB1krEYhqYRC4UwdB2D9LJmmnRtXHz77VTfeCPL
ly+ntraWmpqa9MaaqqqGoiiEQiGGhoYYHBzk00OHiHR2LkqDNlnG7nJhd7nQswqN+ZogiuipFMlY
DDWRgBl5KEF6woqAUFFByc03U756NZWVlSxfvpyamhoCgQAejwdB13VDVVVisRihUIiRkRGGh4cZ
+sc/GO3qIt7VhZGJAle7mVpWuVKDCBng8jXX4N68Gc8111BUVER5eTnV1dVUV1cTCASuFFqGYaDr
upFKpYjH40xMTDA+Ps7Y2Bjj4+OEw2EmenqYPneO1OAgZCWpxTY9S9QMcLhyHGUDRI8HqbISZ309
zoYGnE4nHo8Hv99PaWkpgUCAiooK6/DE4XCkI1HW2a6hqiqKojA9Pc3k5KQl4XCY6elpotFoetuk
v59UKIR66RLToRBJTcMYHUWIRvMSMPO6rbY2/SsIOFatQpYk5NWrkSQJWZZxOBy43W68Xi9+v5+i
oiJKSkooLi7G5/PhdruRZdnads85HzD3hpLJJIqiEIvFiEajRCIRIpEI0WiUWCxGPB631iOKopBK
pazUbi4Oc7Y+snKCzWbDZrMhSRKSJGG3262w6HK5cLlceDwevF4vPp8Pr9eL2+3G5XLhcDjMkxuE
TNibdcBhnv9qmkYqlSKZTJJIJFAUhXg8jqIoKIpCIpGwSCSTSVRVtWSuk3qTiAnebrfngHc4HDid
TpxOJy6Xy7qWZdk6wZzrxCbvCY1JRNd1NE3LAWgSM69ngp+PgCiK1gF3NhFTzHvmO+Y3wCzw8xLI
JpL5tYCZpEyw2f/nWt1ag2UtM0x3Mq1iAp1j+bH0M7KFCGWDnfmbd8AMnuxfUzL/C0rp/wFnFd4n
EQn3XQAAAABJRU5ErkJggg==">
</td>
<td class="content">As mentioned above, it is critical to power off the node
<strong>before</strong> removal, and make sure that it will <strong>not</strong> power on again
(in the existing cluster network) with its current configuration.
If you power on the node as it is, the cluster could end up broken,
and it could be difficult to restore it to a functioning state.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>After powering off the node hp4, we can safely remove it from the cluster.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre> hp1# pvecm delnode hp4
 Killing node 4</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">At this point, it is possible that you will receive an error message
stating <span class="monospaced">Could not kill node (error = CS_ERR_NOT_EXIST)</span>. This does not
signify an actual failure in the deletion of the node, but rather a failure in
corosync trying to kill an offline node. Thus, it can be safely ignored.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>Use <span class="monospaced">pvecm nodes</span> or <span class="monospaced">pvecm status</span> to check the node list again. It should
look something like:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>hp1# pvecm status

...

Votequorum information
~~~~~~~~~~~~~~~~~~~~~~
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2
Flags:            Quorate

Membership information
~~~~~~~~~~~~~~~~~~~~~~
    Nodeid      Votes Name
0x00000001          1 192.168.15.90 (local)
0x00000002          1 192.168.15.91
0x00000003          1 192.168.15.92</pre>
</div></div>
<div class="paragraph">
<p>If, for whatever reason, you want this server to join the same cluster again,
you have to:</p></div>
<div class="ulist"><ul>
<li>
<p>
do a fresh install of Proxmox VE on it,
</p>
</li>
<li>
<p>
then join it, as explained in the previous section.
</p>
</li>
</ul></div>
<div class="paragraph">
<p>The configuration files for the removed node will still reside in
<em>/etc/pve/nodes/hp4</em>. Recover any configuration you still need and remove the
directory afterwards.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">After removal of the node, its SSH fingerprint will still reside in the
<em>known_hosts</em> of the other nodes. If you receive an SSH error after rejoining
a node with the same IP or hostname, run <span class="monospaced">pvecm updatecerts</span> once on the
re-added node to update its fingerprint cluster wide.</td>
</tr></tbody></table>
</div>
<div class="sect3">
<h4 id="pvecm_separate_node_without_reinstall">5.5.1. Separate a Node Without Reinstalling
 <a class="headerlink" href="#pvecm_separate_node_without_reinstall" title="Permalink to this heading"></a>
</h4>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Caution" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKdUlEQVRoge1Ze1AV1x3+zt279wEi
DWCYGzRVktqa1MEmmtbWR22ncXxUrTDWV/5IqG2wUAUfmUwSkk6V+EAJCEQC6figwZBqZtRxqukf
tebRZBpFG1S0hiRErwiaKK977+6eX//YPXt37+UiIJlMZnpmdnb33PP4vt97z2VEhG9yc3zdAO60
/Z/A192+8QScX8Wifr+fWltbzffU1FT4fD72Vew15ASampqovr4eBw8eNPvGjRuHzMxMmj9//tCT
IKIhu958801yuVwEoNdr48aNNJT7EdHQEdi/fz/JshwTvLiKioqGlMSQLFJfX0+MMRvQsWNSqXLz
H2jez78fRWLTpk1DRuKOF6irqyOn02kDOP8XGdTz+VFSP91Hatu79NRvZ0SR2LJly5CQuKPJtbW1
JEmSDdivHp1AwctHSblYSsqFElLOF5PavIvWZU+LIrF169Y7JjHoibt3744CnzXrIQpeOUbKhVJS
mkpIPV9MyrnNpJwtIuVSNa15/MdRJLZt23ZHJBjRwIu5N954gxYvXgzOudm3aO5E1L5SBNZ1Hoxr
INLAyH6HJxVP/WkXtu9+37ZeSUkJVq9ePagQO+BMXFNTEwV+ybxHsPeVIjg6zwOkgSWNhuOuUUBC
KhxJY0AggDSgpwWbn1mO1csftq1ZUFCA0tLSQZXFA0pkVVVVlJOTA6vWsmY9hF07N0AS4OU48LgR
WLO+FPFxboxMS8G5hlPYmPMI4jwM1P0Jtj67DJyrKHvtNADdjPPz8+FwOCgvL29gmuivrVVUVESF
yqxZD1PoyjFSL+7QHbapmJTml+n9EzVRtl676ZekNKwl5VQ+KSfzKHR2E+UutodYxhiVl5cPyCf6
ZUJlZWWUm5trk/wTi6bgtVdf1M2Gq2DEwUgDNBWhUDBqjfSRwwGoIOEP3c3Y/uxirFw0zibMvLw8
VFZW9tucbkugoqKCVq1aZQO/YslUVJU+D9bVBIDroLgK4hygEDweybYGY8AD6d8yftfASL+j+2O8
9NwiPJk51kYiNzcXO3fu7BeJPgkUFxdTbm6ure/JZdNRub0QrOMcGFcBrgLgAHEAGqD2ID7OZZsz
Ji0RcR6HAV4DkaqPJxXouoSywkz8buF9NhIrV65EVVXVbUnEJHDixAlat26drS9n+U+xo/hZMMNh
yZA+uAbAIKOGMCJlmG3eA/clAYaJEamGBlRdC6QBXZewo3AhVswfbSORk5ODDz/8sE8SvRLw+/1U
XV0dBb5MgOe6LUNIEqpOwgCUlOCESw4HuAfG3KX7B2kAGaTJ0Jro77yI8ucX4DfzRtlIHD58GH6/
PyaJXgk0NjZi79695vuUiffrkr91Vt8cHAwc4FZJGiZEKqDcwsi0ZHP+d749POy8MS7iGtBxARWF
8/CDsQnm3BdeeAHWj6N+EYhspSVbgI5zYYkZGmAwzIEbwA0tUM9NfO+7aeb89HuGRQBWLcSNZxj3
zvOYMzk5NpjBEHi99mXD5lXdjqGDJh42h7CEOXhnK3448X5z/shUb1jSpIEM8yFuzOUcjAsNcnR0
KXdG4MEHH8TEiRPN9z0H3sfRE41m+GNcs9i0LjkGTTcrUoGem5g7/V5z/ohElw4O+jxGYQ3qJqmB
jN/+cbINtcfCJvPYY4+hr3qtVwI+n49lZmaa71fbbuH3hX/FycbPQKSCoIIMQEQcehjVo5Gw8/F3
3cSvH03H2HuHY1icQzcvYWKGIMiITMwQwjtnruPpqk9w/ZZq7p2RkYEJEyaAYrCIWY36/X7KyMhA
W1ub2XfP3Qmo3ZKFaQ+P0gmYIdSoOkU4lWSwxGRAdqHL/xm86k27w0blAw3/PncD6yo/xrsfdZj7
paSkoLGxESNGjAgDZsxWK8X0AZ/Px44fP460tLAzXrnWgUX5r+PYuxf1mG/Gf80sofWy2YuWbg/+
uONtPLHhPRw+0WIHL8hbwG95rcUGPiEhAYcOHUJSUhI456YZRWqiz+8BIqLm5mZMmjQJN27cMPsT
E9zYtWE25k4bYzh3mABIBRwOHPgPx+LH/6yDiXPi6pHZcEpk+I5qmtGpCzdQecCPPUevmevLsowj
R45gypQpcDgc5mVowKaJmBoQTEePHo2GhgaMGxcuum52BLHsqUOoP3oWgLB9Sz7QgpgxJoThw/SS
IutnaXBKZEpcgD998Uvs+3ubDbzT6URNTQ0mTZoEVVWhaRo459A0TeCy4etVA+JH8RvnHK2trZgz
Zw5Onz5tjnPJDrxSOAPLZt1nSVIqwDmIVJw6dx1nLt3A3J/cjZThkhE29cvf3o2KA59ja91lcz3G
GDZv3oylS5fC7XbD6XTC6XRCkiRIktS7Jm5HQLDXNA3Xr19HVlYWPvjgA4vEGLav+RHmTxsJX4rb
8AsO4kbOIG6GzTD4HvzlmB/PVH9q2zc/Px/Z2dnwer1wu92QZRkulwuSJEGWZZMEY8wkEGVCVvCC
AOcciqIgPj4e+/btw9SpU83xqkpYteU9HHm7Bf62rnC2RjikisTFSMPV9m4ceucannvVDn7p0qVY
uHAhgsEggsEgQqFQlAlZnVm0PjOxIKFpGogIqqrC5XKhuroa06dPt4wDcl78F+qPNeNKe6cRoTjs
ZYOKK+3dON7wBQrKm2H5pMbMmTOxZMkSBAIB9PT0oKenB4FAAIqiQFEUU4hRRyq3I2AlIZ4553A6
nSgvL8fs2bNtY9eVNaDub58YmhDZWpe+v70bJ5tu4clt/4WihqU4efJkrFixAkRkAg4Gg1AUBaqq
muDF/pGtX7WQNXeIZ0mSUFRUhAULFtjGPl3ZiJdev4QrbZ16zUQqrrZ342JLN5ZvaEJ3ICz68ePH
IycnB5qmmZIW4EOhUJTZWEKo+Rx1KsEYY9Zk4XA4IEmSKXnOuRkVJElCYWEhvF4v6urqzDVK9n2M
zu4Qnl5+L9q/7MHltgBWbr9kA5+eno7c3FzIsoyI5GrTeCzgosU8VmGMmSGLiEwSsixH2WNBQQEA
2EhUH/wc7330BaaOH479/2zHtS/CFabP58OaNWvg9Xp1EE4nZFmGLMtwu93wer1wuVy2ECpJko2I
iTNWJrZGI+HEwi6FisUVCAQQCASwZ88eVFZW9lk9JicnY/369fD5fPB4PCbo+Ph4k4TH44HL5YLX
64XX6zXH9EKE9aUBRkTEGLNJP1Kd4nI4HMjOzkZ8fDyKi4ttJ3eiJSYmYu3atRg1apQJzuv1wuPx
wOPxwOFwmEBFHnA6nXC5XGYSiywlbns2SnoDANN0hMNZnS0UCpkaOnPmDN566y20tLSY2ktLS8OM
GTOQkpICt9uNuLg4M2EJ8CJZSZJkZmFBwkrAWpH2+3BXEBEkRIhTVdVMOCL5qKqKy5cvQ1EUaJoG
VVWRnKx/Jlpt3eVy2QCKu9C68AFr9o0spwd0Oh1ZYggCIlNaiQktiXdhZgKQABcp3ci7LMtM7B0J
fsAErJoAYItG1mdr+hfvVlACtHDIXkploYXbHvQOikDEe6/ZmjFmAhf9ZvJxOk2Qol88W4kYz32S
GNQfHLG+TyMTkAVEr88xQfVnkBg7GAKixSIykDYQsL21/wFkW/B5QqT9lwAAAABJRU5ErkJggg==">
</td>
<td class="content">This is <strong>not</strong> the recommended method, proceed with caution. Use the
previous method if you’re unsure.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>You can also separate a node from a cluster without reinstalling it from
scratch. But after removing the node from the cluster, it will still have
access to any shared storage. This must be resolved before you start removing
the node from the cluster. A Proxmox VE cluster cannot share the exact same
storage with another cluster, as storage locking doesn’t work over the cluster
boundary. Furthermore, it may also lead to VMID conflicts.</p></div>
<div class="paragraph">
<p>It’s suggested that you create a new storage, where only the node which you want
to separate has access. This can be a new export on your NFS or a new Ceph
pool, to name a few examples. It’s just important that the exact same storage
does not get accessed by multiple clusters. After setting up this storage, move
all data and VMs from the node to it. Then you are ready to separate the
node from the cluster.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">Ensure that all shared resources are cleanly separated! Otherwise you
will run into conflicts and problems.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>First, stop the corosync and pve-cluster services on the node:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>systemctl stop pve-cluster
systemctl stop corosync</tt></pre></div></div>
<div class="paragraph">
<p>Start the cluster file system again in local mode:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>pmxcfs -l</tt></pre></div></div>
<div class="paragraph">
<p>Delete the corosync configuration files:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>rm /etc/pve/corosync<span style="color: #990000">.</span>conf
rm -r /etc/corosync<span style="color: #990000">/*</span></tt></pre></div></div>
<div class="paragraph">
<p>You can now start the file system again as a normal service:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>killall pmxcfs
systemctl start pve-cluster</tt></pre></div></div>
<div class="paragraph">
<p>The node is now separated from the cluster. You can deleted it from any
remaining node of the cluster with:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>pvecm delnode oldnode</tt></pre></div></div>
<div class="paragraph">
<p>If the command fails due to a loss of quorum in the remaining node, you can set
the expected votes to 1 as a workaround:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>pvecm expected <span style="color: #993399">1</span></tt></pre></div></div>
<div class="paragraph">
<p>And then repeat the <em>pvecm delnode</em> command.</p></div>
<div class="paragraph">
<p>Now switch back to the separated node and delete all the remaining cluster
files on it. This ensures that the node can be added to another cluster again
without problems.</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>rm /var/lib/corosync<span style="color: #990000">/*</span></tt></pre></div></div>
<div class="paragraph">
<p>As the configuration files from the other nodes are still in the cluster
file system, you may want to clean those up too. After making absolutely sure
that you have the correct node name, you can simply remove the entire
directory recursively from <em>/etc/pve/nodes/NODENAME</em>.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Caution" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKdUlEQVRoge1Ze1AV1x3+zt279wEi
DWCYGzRVktqa1MEmmtbWR22ncXxUrTDWV/5IqG2wUAUfmUwSkk6V+EAJCEQC6figwZBqZtRxqukf
tebRZBpFG1S0hiRErwiaKK977+6eX//YPXt37+UiIJlMZnpmdnb33PP4vt97z2VEhG9yc3zdAO60
/Z/A192+8QScX8Wifr+fWltbzffU1FT4fD72Vew15ASampqovr4eBw8eNPvGjRuHzMxMmj9//tCT
IKIhu958801yuVwEoNdr48aNNJT7EdHQEdi/fz/JshwTvLiKioqGlMSQLFJfX0+MMRvQsWNSqXLz
H2jez78fRWLTpk1DRuKOF6irqyOn02kDOP8XGdTz+VFSP91Hatu79NRvZ0SR2LJly5CQuKPJtbW1
JEmSDdivHp1AwctHSblYSsqFElLOF5PavIvWZU+LIrF169Y7JjHoibt3744CnzXrIQpeOUbKhVJS
mkpIPV9MyrnNpJwtIuVSNa15/MdRJLZt23ZHJBjRwIu5N954gxYvXgzOudm3aO5E1L5SBNZ1Hoxr
INLAyH6HJxVP/WkXtu9+37ZeSUkJVq9ePagQO+BMXFNTEwV+ybxHsPeVIjg6zwOkgSWNhuOuUUBC
KhxJY0AggDSgpwWbn1mO1csftq1ZUFCA0tLSQZXFA0pkVVVVlJOTA6vWsmY9hF07N0AS4OU48LgR
WLO+FPFxboxMS8G5hlPYmPMI4jwM1P0Jtj67DJyrKHvtNADdjPPz8+FwOCgvL29gmuivrVVUVESF
yqxZD1PoyjFSL+7QHbapmJTml+n9EzVRtl676ZekNKwl5VQ+KSfzKHR2E+UutodYxhiVl5cPyCf6
ZUJlZWWUm5trk/wTi6bgtVdf1M2Gq2DEwUgDNBWhUDBqjfSRwwGoIOEP3c3Y/uxirFw0zibMvLw8
VFZW9tucbkugoqKCVq1aZQO/YslUVJU+D9bVBIDroLgK4hygEDweybYGY8AD6d8yftfASL+j+2O8
9NwiPJk51kYiNzcXO3fu7BeJPgkUFxdTbm6ure/JZdNRub0QrOMcGFcBrgLgAHEAGqD2ID7OZZsz
Ji0RcR6HAV4DkaqPJxXouoSywkz8buF9NhIrV65EVVXVbUnEJHDixAlat26drS9n+U+xo/hZMMNh
yZA+uAbAIKOGMCJlmG3eA/clAYaJEamGBlRdC6QBXZewo3AhVswfbSORk5ODDz/8sE8SvRLw+/1U
XV0dBb5MgOe6LUNIEqpOwgCUlOCESw4HuAfG3KX7B2kAGaTJ0Jro77yI8ucX4DfzRtlIHD58GH6/
PyaJXgk0NjZi79695vuUiffrkr91Vt8cHAwc4FZJGiZEKqDcwsi0ZHP+d749POy8MS7iGtBxARWF
8/CDsQnm3BdeeAHWj6N+EYhspSVbgI5zYYkZGmAwzIEbwA0tUM9NfO+7aeb89HuGRQBWLcSNZxj3
zvOYMzk5NpjBEHi99mXD5lXdjqGDJh42h7CEOXhnK3448X5z/shUb1jSpIEM8yFuzOUcjAsNcnR0
KXdG4MEHH8TEiRPN9z0H3sfRE41m+GNcs9i0LjkGTTcrUoGem5g7/V5z/ohElw4O+jxGYQ3qJqmB
jN/+cbINtcfCJvPYY4+hr3qtVwI+n49lZmaa71fbbuH3hX/FycbPQKSCoIIMQEQcehjVo5Gw8/F3
3cSvH03H2HuHY1icQzcvYWKGIMiITMwQwjtnruPpqk9w/ZZq7p2RkYEJEyaAYrCIWY36/X7KyMhA
W1ub2XfP3Qmo3ZKFaQ+P0gmYIdSoOkU4lWSwxGRAdqHL/xm86k27w0blAw3/PncD6yo/xrsfdZj7
paSkoLGxESNGjAgDZsxWK8X0AZ/Px44fP460tLAzXrnWgUX5r+PYuxf1mG/Gf80sofWy2YuWbg/+
uONtPLHhPRw+0WIHL8hbwG95rcUGPiEhAYcOHUJSUhI456YZRWqiz+8BIqLm5mZMmjQJN27cMPsT
E9zYtWE25k4bYzh3mABIBRwOHPgPx+LH/6yDiXPi6pHZcEpk+I5qmtGpCzdQecCPPUevmevLsowj
R45gypQpcDgc5mVowKaJmBoQTEePHo2GhgaMGxcuum52BLHsqUOoP3oWgLB9Sz7QgpgxJoThw/SS
IutnaXBKZEpcgD998Uvs+3ubDbzT6URNTQ0mTZoEVVWhaRo459A0TeCy4etVA+JH8RvnHK2trZgz
Zw5Onz5tjnPJDrxSOAPLZt1nSVIqwDmIVJw6dx1nLt3A3J/cjZThkhE29cvf3o2KA59ja91lcz3G
GDZv3oylS5fC7XbD6XTC6XRCkiRIktS7Jm5HQLDXNA3Xr19HVlYWPvjgA4vEGLav+RHmTxsJX4rb
8AsO4kbOIG6GzTD4HvzlmB/PVH9q2zc/Px/Z2dnwer1wu92QZRkulwuSJEGWZZMEY8wkEGVCVvCC
AOcciqIgPj4e+/btw9SpU83xqkpYteU9HHm7Bf62rnC2RjikisTFSMPV9m4ceucannvVDn7p0qVY
uHAhgsEggsEgQqFQlAlZnVm0PjOxIKFpGogIqqrC5XKhuroa06dPt4wDcl78F+qPNeNKe6cRoTjs
ZYOKK+3dON7wBQrKm2H5pMbMmTOxZMkSBAIB9PT0oKenB4FAAIqiQFEUU4hRRyq3I2AlIZ4553A6
nSgvL8fs2bNtY9eVNaDub58YmhDZWpe+v70bJ5tu4clt/4WihqU4efJkrFixAkRkAg4Gg1AUBaqq
muDF/pGtX7WQNXeIZ0mSUFRUhAULFtjGPl3ZiJdev4QrbZ16zUQqrrZ342JLN5ZvaEJ3ICz68ePH
IycnB5qmmZIW4EOhUJTZWEKo+Rx1KsEYY9Zk4XA4IEmSKXnOuRkVJElCYWEhvF4v6urqzDVK9n2M
zu4Qnl5+L9q/7MHltgBWbr9kA5+eno7c3FzIsoyI5GrTeCzgosU8VmGMmSGLiEwSsixH2WNBQQEA
2EhUH/wc7330BaaOH479/2zHtS/CFabP58OaNWvg9Xp1EE4nZFmGLMtwu93wer1wuVy2ECpJko2I
iTNWJrZGI+HEwi6FisUVCAQQCASwZ88eVFZW9lk9JicnY/369fD5fPB4PCbo+Ph4k4TH44HL5YLX
64XX6zXH9EKE9aUBRkTEGLNJP1Kd4nI4HMjOzkZ8fDyKi4ttJ3eiJSYmYu3atRg1apQJzuv1wuPx
wOPxwOFwmEBFHnA6nXC5XGYSiywlbns2SnoDANN0hMNZnS0UCpkaOnPmDN566y20tLSY2ktLS8OM
GTOQkpICt9uNuLg4M2EJ8CJZSZJkZmFBwkrAWpH2+3BXEBEkRIhTVdVMOCL5qKqKy5cvQ1EUaJoG
VVWRnKx/Jlpt3eVy2QCKu9C68AFr9o0spwd0Oh1ZYggCIlNaiQktiXdhZgKQABcp3ci7LMtM7B0J
fsAErJoAYItG1mdr+hfvVlACtHDIXkploYXbHvQOikDEe6/ZmjFmAhf9ZvJxOk2Qol88W4kYz32S
GNQfHLG+TyMTkAVEr88xQfVnkBg7GAKixSIykDYQsL21/wFkW/B5QqT9lwAAAABJRU5ErkJggg==">
</td>
<td class="content">The node’s SSH keys will remain in the <em>authorized_key</em> file. This
means that the nodes can still connect to each other with public key
authentication. You should fix this by removing the respective keys from the
<em>/etc/pve/priv/authorized_keys</em> file.</td>
</tr></tbody></table>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_quorum">
<span>5.6. Quorum</span>
 <a class="headerlink" href="#_quorum" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Proxmox VE use a quorum-based technique to provide a consistent state among
all cluster nodes.</p></div>
<div class="quoteblock">
<div class="content">
<div class="paragraph">
<p>A quorum is the minimum number of votes that a distributed transaction
has to obtain in order to be allowed to perform an operation in a
distributed system.</p></div>
</div>
<div class="attribution">
<em>Quorum (distributed computing)</em><br>
— from Wikipedia
</div></div>
<div class="paragraph">
<p>In case of network partitioning, state changes requires that a
majority of nodes are online. The cluster switches to read-only mode
if it loses quorum.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Proxmox VE assigns a single vote to each node by default.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect2">
<h3 id="_cluster_network">
<span>5.7. Cluster Network</span>
 <a class="headerlink" href="#_cluster_network" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>The cluster network is the core of a cluster. All messages sent over it have to
be delivered reliably to all nodes in their respective order. In Proxmox VE this
part is done by corosync, an implementation of a high performance, low overhead,
high availability development toolkit. It serves our decentralized configuration
file system (<span class="monospaced">pmxcfs</span>).</p></div>
<div class="sect3">
<h4 id="pvecm_cluster_network_requirements">5.7.1. Network Requirements
 <a class="headerlink" href="#pvecm_cluster_network_requirements" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The Proxmox VE cluster stack requires a reliable network with latencies under 5
milliseconds (LAN performance) between all nodes to operate stably. While on
setups with a small node count a network with higher latencies <em>may</em> work, this
is not guaranteed and gets rather unlikely with more than three nodes and
latencies above around 10 ms.</p></div>
<div class="paragraph">
<p>The network should not be used heavily by other members, as while corosync does
not uses much bandwidth it is sensitive to latency jitters; ideally corosync
runs on its own physically separated network.  Especially do not use a shared
network for corosync and storage (except as a potential low-priority fallback
in a <a href="#pvecm_redundancy">redundant</a> configuration).</p></div>
<div class="paragraph">
<p>Before setting up a cluster, it is good practice to check if the network is fit
for that purpose. To ensure that the nodes can connect to each other on the
cluster network, you can test the connectivity between them with the <span class="monospaced">ping</span>
tool.</p></div>
<div class="paragraph">
<p>If the Proxmox VE firewall is enabled, ACCEPT rules for corosync will automatically
be generated - no manual action is required.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Corosync used Multicast before version 3.0 (introduced in Proxmox VE 6.0).
Modern versions rely on <a href="https://kronosnet.org/">Kronosnet</a> for cluster
communication, which, for now, only supports regular UDP unicast.</td>
</tr></tbody></table>
</div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Caution" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKdUlEQVRoge1Ze1AV1x3+zt279wEi
DWCYGzRVktqa1MEmmtbWR22ncXxUrTDWV/5IqG2wUAUfmUwSkk6V+EAJCEQC6figwZBqZtRxqukf
tebRZBpFG1S0hiRErwiaKK977+6eX//YPXt37+UiIJlMZnpmdnb33PP4vt97z2VEhG9yc3zdAO60
/Z/A192+8QScX8Wifr+fWltbzffU1FT4fD72Vew15ASampqovr4eBw8eNPvGjRuHzMxMmj9//tCT
IKIhu958801yuVwEoNdr48aNNJT7EdHQEdi/fz/JshwTvLiKioqGlMSQLFJfX0+MMRvQsWNSqXLz
H2jez78fRWLTpk1DRuKOF6irqyOn02kDOP8XGdTz+VFSP91Hatu79NRvZ0SR2LJly5CQuKPJtbW1
JEmSDdivHp1AwctHSblYSsqFElLOF5PavIvWZU+LIrF169Y7JjHoibt3744CnzXrIQpeOUbKhVJS
mkpIPV9MyrnNpJwtIuVSNa15/MdRJLZt23ZHJBjRwIu5N954gxYvXgzOudm3aO5E1L5SBNZ1Hoxr
INLAyH6HJxVP/WkXtu9+37ZeSUkJVq9ePagQO+BMXFNTEwV+ybxHsPeVIjg6zwOkgSWNhuOuUUBC
KhxJY0AggDSgpwWbn1mO1csftq1ZUFCA0tLSQZXFA0pkVVVVlJOTA6vWsmY9hF07N0AS4OU48LgR
WLO+FPFxboxMS8G5hlPYmPMI4jwM1P0Jtj67DJyrKHvtNADdjPPz8+FwOCgvL29gmuivrVVUVESF
yqxZD1PoyjFSL+7QHbapmJTml+n9EzVRtl676ZekNKwl5VQ+KSfzKHR2E+UutodYxhiVl5cPyCf6
ZUJlZWWUm5trk/wTi6bgtVdf1M2Gq2DEwUgDNBWhUDBqjfSRwwGoIOEP3c3Y/uxirFw0zibMvLw8
VFZW9tucbkugoqKCVq1aZQO/YslUVJU+D9bVBIDroLgK4hygEDweybYGY8AD6d8yftfASL+j+2O8
9NwiPJk51kYiNzcXO3fu7BeJPgkUFxdTbm6ure/JZdNRub0QrOMcGFcBrgLgAHEAGqD2ID7OZZsz
Ji0RcR6HAV4DkaqPJxXouoSywkz8buF9NhIrV65EVVXVbUnEJHDixAlat26drS9n+U+xo/hZMMNh
yZA+uAbAIKOGMCJlmG3eA/clAYaJEamGBlRdC6QBXZewo3AhVswfbSORk5ODDz/8sE8SvRLw+/1U
XV0dBb5MgOe6LUNIEqpOwgCUlOCESw4HuAfG3KX7B2kAGaTJ0Jro77yI8ucX4DfzRtlIHD58GH6/
PyaJXgk0NjZi79695vuUiffrkr91Vt8cHAwc4FZJGiZEKqDcwsi0ZHP+d749POy8MS7iGtBxARWF
8/CDsQnm3BdeeAHWj6N+EYhspSVbgI5zYYkZGmAwzIEbwA0tUM9NfO+7aeb89HuGRQBWLcSNZxj3
zvOYMzk5NpjBEHi99mXD5lXdjqGDJh42h7CEOXhnK3448X5z/shUb1jSpIEM8yFuzOUcjAsNcnR0
KXdG4MEHH8TEiRPN9z0H3sfRE41m+GNcs9i0LjkGTTcrUoGem5g7/V5z/ohElw4O+jxGYQ3qJqmB
jN/+cbINtcfCJvPYY4+hr3qtVwI+n49lZmaa71fbbuH3hX/FycbPQKSCoIIMQEQcehjVo5Gw8/F3
3cSvH03H2HuHY1icQzcvYWKGIMiITMwQwjtnruPpqk9w/ZZq7p2RkYEJEyaAYrCIWY36/X7KyMhA
W1ub2XfP3Qmo3ZKFaQ+P0gmYIdSoOkU4lWSwxGRAdqHL/xm86k27w0blAw3/PncD6yo/xrsfdZj7
paSkoLGxESNGjAgDZsxWK8X0AZ/Px44fP460tLAzXrnWgUX5r+PYuxf1mG/Gf80sofWy2YuWbg/+
uONtPLHhPRw+0WIHL8hbwG95rcUGPiEhAYcOHUJSUhI456YZRWqiz+8BIqLm5mZMmjQJN27cMPsT
E9zYtWE25k4bYzh3mABIBRwOHPgPx+LH/6yDiXPi6pHZcEpk+I5qmtGpCzdQecCPPUevmevLsowj
R45gypQpcDgc5mVowKaJmBoQTEePHo2GhgaMGxcuum52BLHsqUOoP3oWgLB9Sz7QgpgxJoThw/SS
IutnaXBKZEpcgD998Uvs+3ubDbzT6URNTQ0mTZoEVVWhaRo459A0TeCy4etVA+JH8RvnHK2trZgz
Zw5Onz5tjnPJDrxSOAPLZt1nSVIqwDmIVJw6dx1nLt3A3J/cjZThkhE29cvf3o2KA59ja91lcz3G
GDZv3oylS5fC7XbD6XTC6XRCkiRIktS7Jm5HQLDXNA3Xr19HVlYWPvjgA4vEGLav+RHmTxsJX4rb
8AsO4kbOIG6GzTD4HvzlmB/PVH9q2zc/Px/Z2dnwer1wu92QZRkulwuSJEGWZZMEY8wkEGVCVvCC
AOcciqIgPj4e+/btw9SpU83xqkpYteU9HHm7Bf62rnC2RjikisTFSMPV9m4ceucannvVDn7p0qVY
uHAhgsEggsEgQqFQlAlZnVm0PjOxIKFpGogIqqrC5XKhuroa06dPt4wDcl78F+qPNeNKe6cRoTjs
ZYOKK+3dON7wBQrKm2H5pMbMmTOxZMkSBAIB9PT0oKenB4FAAIqiQFEUU4hRRyq3I2AlIZ4553A6
nSgvL8fs2bNtY9eVNaDub58YmhDZWpe+v70bJ5tu4clt/4WihqU4efJkrFixAkRkAg4Gg1AUBaqq
muDF/pGtX7WQNXeIZ0mSUFRUhAULFtjGPl3ZiJdev4QrbZ16zUQqrrZ342JLN5ZvaEJ3ICz68ePH
IycnB5qmmZIW4EOhUJTZWEKo+Rx1KsEYY9Zk4XA4IEmSKXnOuRkVJElCYWEhvF4v6urqzDVK9n2M
zu4Qnl5+L9q/7MHltgBWbr9kA5+eno7c3FzIsoyI5GrTeCzgosU8VmGMmSGLiEwSsixH2WNBQQEA
2EhUH/wc7330BaaOH479/2zHtS/CFabP58OaNWvg9Xp1EE4nZFmGLMtwu93wer1wuVy2ECpJko2I
iTNWJrZGI+HEwi6FisUVCAQQCASwZ88eVFZW9lk9JicnY/369fD5fPB4PCbo+Ph4k4TH44HL5YLX
64XX6zXH9EKE9aUBRkTEGLNJP1Kd4nI4HMjOzkZ8fDyKi4ttJ3eiJSYmYu3atRg1apQJzuv1wuPx
wOPxwOFwmEBFHnA6nXC5XGYSiywlbns2SnoDANN0hMNZnS0UCpkaOnPmDN566y20tLSY2ktLS8OM
GTOQkpICt9uNuLg4M2EJ8CJZSZJkZmFBwkrAWpH2+3BXEBEkRIhTVdVMOCL5qKqKy5cvQ1EUaJoG
VVWRnKx/Jlpt3eVy2QCKu9C68AFr9o0spwd0Oh1ZYggCIlNaiQktiXdhZgKQABcp3ci7LMtM7B0J
fsAErJoAYItG1mdr+hfvVlACtHDIXkploYXbHvQOikDEe6/ZmjFmAhf9ZvJxOk2Qol88W4kYz32S
GNQfHLG+TyMTkAVEr88xQfVnkBg7GAKixSIykDYQsL21/wFkW/B5QqT9lwAAAABJRU5ErkJggg==">
</td>
<td class="content">You can still enable Multicast or legacy unicast by setting your
transport to <span class="monospaced">udp</span> or <span class="monospaced">udpu</span> in your <a href="#pvecm_edit_corosync_conf">corosync.conf</a>,
but keep in mind that this will disable all cryptography and redundancy support.
This is therefore not recommended.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="_separate_cluster_network">5.7.2. Separate Cluster Network
 <a class="headerlink" href="#_separate_cluster_network" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>When creating a cluster without any parameters, the corosync cluster network is
generally shared with the web interface and the VMs' network. Depending on
your setup, even storage traffic may get sent over the same network. It’s
recommended to change that, as corosync is a time-critical, real-time
application.</p></div>
<div class="sect4">
<h5 id="_setting_up_a_new_network">Setting Up a New Network
 <a class="headerlink" href="#_setting_up_a_new_network" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>First, you have to set up a new network interface. It should be on a physically
separate network. Ensure that your network fulfills the
<a href="#pvecm_cluster_network_requirements">cluster network requirements</a>.</p></div>
</div>
<div class="sect4">
<h5 id="_separate_on_cluster_creation">Separate On Cluster Creation
 <a class="headerlink" href="#_separate_on_cluster_creation" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>This is possible via the <em>linkX</em> parameters of the <em>pvecm create</em>
command, used for creating a new cluster.</p></div>
<div class="paragraph">
<p>If you have set up an additional NIC with a static address on 10.10.10.1/25,
and want to send and receive all cluster communication over this interface,
you would execute:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>pvecm create <span style="font-weight: bold"><span style="color: #0000FF">test</span></span> --link<span style="color: #993399">0</span> <span style="color: #993399">10.10</span><span style="color: #990000">.</span><span style="color: #993399">10.1</span></tt></pre></div></div>
<div class="paragraph">
<p>To check if everything is working properly, execute:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>systemctl status corosync</tt></pre></div></div>
<div class="paragraph">
<p>Afterwards, proceed as described above to
<a href="#pvecm_adding_nodes_with_separated_cluster_network">add nodes with a separated cluster network</a>.</p></div>
</div>
<div class="sect4">
<h5 id="pvecm_separate_cluster_net_after_creation">Separate After Cluster Creation
 <a class="headerlink" href="#pvecm_separate_cluster_net_after_creation" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>You can do this if you have already created a cluster and want to switch
its communication to another network, without rebuilding the whole cluster.
This change may lead to short periods of quorum loss in the cluster, as nodes
have to restart corosync and come up one after the other on the new network.</p></div>
<div class="paragraph">
<p>Check how to <a href="#pvecm_edit_corosync_conf">edit the corosync.conf file</a> first.
Then, open it and you should see a file similar to:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>logging {
  debug: off
  to_syslog: yes
}

nodelist {

  node {
    name: due
    nodeid: 2
    quorum_votes: 1
    ring0_addr: due
  }

  node {
    name: tre
    nodeid: 3
    quorum_votes: 1
    ring0_addr: tre
  }

  node {
    name: uno
    nodeid: 1
    quorum_votes: 1
    ring0_addr: uno
  }

}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: testcluster
  config_version: 3
  ip_version: ipv4-6
  secauth: on
  version: 2
  interface {
    linknumber: 0
  }

}</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content"><span class="monospaced">ringX_addr</span> actually specifies a corosync <strong>link address</strong>. The name "ring"
is a remnant of older corosync versions that is kept for backwards
compatibility.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>The first thing you want to do is add the <em>name</em> properties in the node entries,
if you do not see them already. Those <strong>must</strong> match the node name.</p></div>
<div class="paragraph">
<p>Then replace all addresses from the <em>ring0_addr</em> properties of all nodes with
the new addresses. You may use plain IP addresses or hostnames here. If you use
hostnames, ensure that they are resolvable from all nodes (see also
<a href="#pvecm_corosync_addresses">Link Address Types</a>).</p></div>
<div class="paragraph">
<p>In this example, we want to switch cluster communication to the
10.10.10.0/25 network, so we change the <em>ring0_addr</em> of each node respectively.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">The exact same procedure can be used to change other <em>ringX_addr</em> values
as well. However, we recommend only changing one link address at a time, so
that it’s easier to recover if something goes wrong.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>After we increase the <em>config_version</em> property, the new configuration file
should look like:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>logging {
  debug: off
  to_syslog: yes
}

nodelist {

  node {
    name: due
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 10.10.10.2
  }

  node {
    name: tre
    nodeid: 3
    quorum_votes: 1
    ring0_addr: 10.10.10.3
  }

  node {
    name: uno
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 10.10.10.1
  }

}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: testcluster
  config_version: 4
  ip_version: ipv4-6
  secauth: on
  version: 2
  interface {
    linknumber: 0
  }

}</pre>
</div></div>
<div class="paragraph">
<p>Then, after a final check to see that all changed information is correct, we
save it and once again follow the
<a href="#pvecm_edit_corosync_conf">edit corosync.conf file</a> section to bring it into
effect.</p></div>
<div class="paragraph">
<p>The changes will be applied live, so restarting corosync is not strictly
necessary. If you changed other settings as well, or notice corosync
complaining, you can optionally trigger a restart.</p></div>
<div class="paragraph">
<p>On a single node execute:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>systemctl restart corosync</tt></pre></div></div>
<div class="paragraph">
<p>Now check if everything is okay:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>systemctl status corosync</tt></pre></div></div>
<div class="paragraph">
<p>If corosync begins to work again, restart it on all other nodes too.
They will then join the cluster membership one by one on the new network.</p></div>
</div>
</div>
<div class="sect3">
<h4 id="pvecm_corosync_addresses">5.7.3. Corosync Addresses
 <a class="headerlink" href="#pvecm_corosync_addresses" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>A corosync link address (for backwards compatibility denoted by <em>ringX_addr</em> in
<span class="monospaced">corosync.conf</span>) can be specified in two ways:</p></div>
<div class="ulist"><ul>
<li>
<p>
<strong>IPv4/v6 addresses</strong> can be used directly. They are recommended, since they
are static and usually not changed carelessly.
</p>
</li>
<li>
<p>
<strong>Hostnames</strong> will be resolved using <span class="monospaced">getaddrinfo</span>, which means that by
default, IPv6 addresses will be used first, if available (see also
<span class="monospaced">man gai.conf</span>). Keep this in mind, especially when upgrading an existing
cluster to IPv6.
</p>
</li>
</ul></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Caution" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKdUlEQVRoge1Ze1AV1x3+zt279wEi
DWCYGzRVktqa1MEmmtbWR22ncXxUrTDWV/5IqG2wUAUfmUwSkk6V+EAJCEQC6figwZBqZtRxqukf
tebRZBpFG1S0hiRErwiaKK977+6eX//YPXt37+UiIJlMZnpmdnb33PP4vt97z2VEhG9yc3zdAO60
/Z/A192+8QScX8Wifr+fWltbzffU1FT4fD72Vew15ASampqovr4eBw8eNPvGjRuHzMxMmj9//tCT
IKIhu958801yuVwEoNdr48aNNJT7EdHQEdi/fz/JshwTvLiKioqGlMSQLFJfX0+MMRvQsWNSqXLz
H2jez78fRWLTpk1DRuKOF6irqyOn02kDOP8XGdTz+VFSP91Hatu79NRvZ0SR2LJly5CQuKPJtbW1
JEmSDdivHp1AwctHSblYSsqFElLOF5PavIvWZU+LIrF169Y7JjHoibt3744CnzXrIQpeOUbKhVJS
mkpIPV9MyrnNpJwtIuVSNa15/MdRJLZt23ZHJBjRwIu5N954gxYvXgzOudm3aO5E1L5SBNZ1Hoxr
INLAyH6HJxVP/WkXtu9+37ZeSUkJVq9ePagQO+BMXFNTEwV+ybxHsPeVIjg6zwOkgSWNhuOuUUBC
KhxJY0AggDSgpwWbn1mO1csftq1ZUFCA0tLSQZXFA0pkVVVVlJOTA6vWsmY9hF07N0AS4OU48LgR
WLO+FPFxboxMS8G5hlPYmPMI4jwM1P0Jtj67DJyrKHvtNADdjPPz8+FwOCgvL29gmuivrVVUVESF
yqxZD1PoyjFSL+7QHbapmJTml+n9EzVRtl676ZekNKwl5VQ+KSfzKHR2E+UutodYxhiVl5cPyCf6
ZUJlZWWUm5trk/wTi6bgtVdf1M2Gq2DEwUgDNBWhUDBqjfSRwwGoIOEP3c3Y/uxirFw0zibMvLw8
VFZW9tucbkugoqKCVq1aZQO/YslUVJU+D9bVBIDroLgK4hygEDweybYGY8AD6d8yftfASL+j+2O8
9NwiPJk51kYiNzcXO3fu7BeJPgkUFxdTbm6ure/JZdNRub0QrOMcGFcBrgLgAHEAGqD2ID7OZZsz
Ji0RcR6HAV4DkaqPJxXouoSywkz8buF9NhIrV65EVVXVbUnEJHDixAlat26drS9n+U+xo/hZMMNh
yZA+uAbAIKOGMCJlmG3eA/clAYaJEamGBlRdC6QBXZewo3AhVswfbSORk5ODDz/8sE8SvRLw+/1U
XV0dBb5MgOe6LUNIEqpOwgCUlOCESw4HuAfG3KX7B2kAGaTJ0Jro77yI8ucX4DfzRtlIHD58GH6/
PyaJXgk0NjZi79695vuUiffrkr91Vt8cHAwc4FZJGiZEKqDcwsi0ZHP+d749POy8MS7iGtBxARWF
8/CDsQnm3BdeeAHWj6N+EYhspSVbgI5zYYkZGmAwzIEbwA0tUM9NfO+7aeb89HuGRQBWLcSNZxj3
zvOYMzk5NpjBEHi99mXD5lXdjqGDJh42h7CEOXhnK3448X5z/shUb1jSpIEM8yFuzOUcjAsNcnR0
KXdG4MEHH8TEiRPN9z0H3sfRE41m+GNcs9i0LjkGTTcrUoGem5g7/V5z/ohElw4O+jxGYQ3qJqmB
jN/+cbINtcfCJvPYY4+hr3qtVwI+n49lZmaa71fbbuH3hX/FycbPQKSCoIIMQEQcehjVo5Gw8/F3
3cSvH03H2HuHY1icQzcvYWKGIMiITMwQwjtnruPpqk9w/ZZq7p2RkYEJEyaAYrCIWY36/X7KyMhA
W1ub2XfP3Qmo3ZKFaQ+P0gmYIdSoOkU4lWSwxGRAdqHL/xm86k27w0blAw3/PncD6yo/xrsfdZj7
paSkoLGxESNGjAgDZsxWK8X0AZ/Px44fP460tLAzXrnWgUX5r+PYuxf1mG/Gf80sofWy2YuWbg/+
uONtPLHhPRw+0WIHL8hbwG95rcUGPiEhAYcOHUJSUhI456YZRWqiz+8BIqLm5mZMmjQJN27cMPsT
E9zYtWE25k4bYzh3mABIBRwOHPgPx+LH/6yDiXPi6pHZcEpk+I5qmtGpCzdQecCPPUevmevLsowj
R45gypQpcDgc5mVowKaJmBoQTEePHo2GhgaMGxcuum52BLHsqUOoP3oWgLB9Sz7QgpgxJoThw/SS
IutnaXBKZEpcgD998Uvs+3ubDbzT6URNTQ0mTZoEVVWhaRo459A0TeCy4etVA+JH8RvnHK2trZgz
Zw5Onz5tjnPJDrxSOAPLZt1nSVIqwDmIVJw6dx1nLt3A3J/cjZThkhE29cvf3o2KA59ja91lcz3G
GDZv3oylS5fC7XbD6XTC6XRCkiRIktS7Jm5HQLDXNA3Xr19HVlYWPvjgA4vEGLav+RHmTxsJX4rb
8AsO4kbOIG6GzTD4HvzlmB/PVH9q2zc/Px/Z2dnwer1wu92QZRkulwuSJEGWZZMEY8wkEGVCVvCC
AOcciqIgPj4e+/btw9SpU83xqkpYteU9HHm7Bf62rnC2RjikisTFSMPV9m4ceucannvVDn7p0qVY
uHAhgsEggsEgQqFQlAlZnVm0PjOxIKFpGogIqqrC5XKhuroa06dPt4wDcl78F+qPNeNKe6cRoTjs
ZYOKK+3dON7wBQrKm2H5pMbMmTOxZMkSBAIB9PT0oKenB4FAAIqiQFEUU4hRRyq3I2AlIZ4553A6
nSgvL8fs2bNtY9eVNaDub58YmhDZWpe+v70bJ5tu4clt/4WihqU4efJkrFixAkRkAg4Gg1AUBaqq
muDF/pGtX7WQNXeIZ0mSUFRUhAULFtjGPl3ZiJdev4QrbZ16zUQqrrZ342JLN5ZvaEJ3ICz68ePH
IycnB5qmmZIW4EOhUJTZWEKo+Rx1KsEYY9Zk4XA4IEmSKXnOuRkVJElCYWEhvF4v6urqzDVK9n2M
zu4Qnl5+L9q/7MHltgBWbr9kA5+eno7c3FzIsoyI5GrTeCzgosU8VmGMmSGLiEwSsixH2WNBQQEA
2EhUH/wc7330BaaOH479/2zHtS/CFabP58OaNWvg9Xp1EE4nZFmGLMtwu93wer1wuVy2ECpJko2I
iTNWJrZGI+HEwi6FisUVCAQQCASwZ88eVFZW9lk9JicnY/369fD5fPB4PCbo+Ph4k4TH44HL5YLX
64XX6zXH9EKE9aUBRkTEGLNJP1Kd4nI4HMjOzkZ8fDyKi4ttJ3eiJSYmYu3atRg1apQJzuv1wuPx
wOPxwOFwmEBFHnA6nXC5XGYSiywlbns2SnoDANN0hMNZnS0UCpkaOnPmDN566y20tLSY2ktLS8OM
GTOQkpICt9uNuLg4M2EJ8CJZSZJkZmFBwkrAWpH2+3BXEBEkRIhTVdVMOCL5qKqKy5cvQ1EUaJoG
VVWRnKx/Jlpt3eVy2QCKu9C68AFr9o0spwd0Oh1ZYggCIlNaiQktiXdhZgKQABcp3ci7LMtM7B0J
fsAErJoAYItG1mdr+hfvVlACtHDIXkploYXbHvQOikDEe6/ZmjFmAhf9ZvJxOk2Qol88W4kYz32S
GNQfHLG+TyMTkAVEr88xQfVnkBg7GAKixSIykDYQsL21/wFkW/B5QqT9lwAAAABJRU5ErkJggg==">
</td>
<td class="content">Hostnames should be used with care, since the addresses they
resolve to can be changed without touching corosync or the node it runs on -
which may lead to a situation where an address is changed without thinking
about implications for corosync.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>A separate, static hostname specifically for corosync is recommended, if
hostnames are preferred. Also, make sure that every node in the cluster can
resolve all hostnames correctly.</p></div>
<div class="paragraph">
<p>Since Proxmox VE 5.1, while supported, hostnames will be resolved at the time of
entry. Only the resolved IP is saved to the configuration.</p></div>
<div class="paragraph">
<p>Nodes that joined the cluster on earlier versions likely still use their
unresolved hostname in <span class="monospaced">corosync.conf</span>. It might be a good idea to replace
them with IPs or a separate hostname, as mentioned above.</p></div>
</div>
</div>
<div class="sect2">
<h3 id="pvecm_redundancy">
<span>5.8. Corosync Redundancy</span>
 <a class="headerlink" href="#pvecm_redundancy" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Corosync supports redundant networking via its integrated Kronosnet layer by
default (it is not supported on the legacy udp/udpu transports). It can be
enabled by specifying more than one link address, either via the <em>--linkX</em>
parameters of <span class="monospaced">pvecm</span>, in the GUI as <strong>Link 1</strong> (while creating a cluster or
adding a new node) or by specifying more than one <em>ringX_addr</em> in
<span class="monospaced">corosync.conf</span>.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">To provide useful failover, every link should be on its own
physical network connection.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>Links are used according to a priority setting. You can configure this priority
by setting <em>knet_link_priority</em> in the corresponding interface section in
<span class="monospaced">corosync.conf</span>, or, preferably, using the <em>priority</em> parameter when creating
your cluster with <span class="monospaced">pvecm</span>:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre> # pvecm create CLUSTERNAME --link0 10.10.10.1,priority=15 --link1 10.20.20.1,priority=20</pre>
</div></div>
<div class="paragraph">
<p>This would cause <em>link1</em> to be used first, since it has the higher priority.</p></div>
<div class="paragraph">
<p>If no priorities are configured manually (or two links have the same priority),
links will be used in order of their number, with the lower number having higher
priority.</p></div>
<div class="paragraph">
<p>Even if all links are working, only the one with the highest priority will see
corosync traffic. Link priorities cannot be mixed, meaning that links with
different priorities will not be able to communicate with each other.</p></div>
<div class="paragraph">
<p>Since lower priority links will not see traffic unless all higher priorities
have failed, it becomes a useful strategy to specify networks used for
other tasks (VMs, storage, etc.) as low-priority links. If worst comes to
worst, a higher latency or more congested connection might be better than no
connection at all.</p></div>
<div class="sect3">
<h4 id="_adding_redundant_links_to_an_existing_cluster">5.8.1. Adding Redundant Links To An Existing Cluster
 <a class="headerlink" href="#_adding_redundant_links_to_an_existing_cluster" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>To add a new link to a running configuration, first check how to
<a href="#pvecm_edit_corosync_conf">edit the corosync.conf file</a>.</p></div>
<div class="paragraph">
<p>Then, add a new <em>ringX_addr</em> to every node in the <span class="monospaced">nodelist</span> section. Make
sure that your <em>X</em> is the same for every node you add it to, and that it is
unique for each node.</p></div>
<div class="paragraph">
<p>Lastly, add a new <em>interface</em>, as shown below, to your <span class="monospaced">totem</span>
section, replacing <em>X</em> with the link number chosen above.</p></div>
<div class="paragraph">
<p>Assuming you added a link with number 1, the new configuration file could look
like this:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>logging {
  debug: off
  to_syslog: yes
}

nodelist {

  node {
    name: due
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 10.10.10.2
    ring1_addr: 10.20.20.2
  }

  node {
    name: tre
    nodeid: 3
    quorum_votes: 1
    ring0_addr: 10.10.10.3
    ring1_addr: 10.20.20.3
  }

  node {
    name: uno
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 10.10.10.1
    ring1_addr: 10.20.20.1
  }

}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: testcluster
  config_version: 4
  ip_version: ipv4-6
  secauth: on
  version: 2
  interface {
    linknumber: 0
  }
  interface {
    linknumber: 1
  }
}</pre>
</div></div>
<div class="paragraph">
<p>The new link will be enabled as soon as you follow the last steps to
<a href="#pvecm_edit_corosync_conf">edit the corosync.conf file</a>. A restart should not
be necessary. You can check that corosync loaded the new link using:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>journalctl -b -u corosync</pre>
</div></div>
<div class="paragraph">
<p>It might be a good idea to test the new link by temporarily disconnecting the
old link on one node and making sure that its status remains online while
disconnected:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>pvecm status</pre>
</div></div>
<div class="paragraph">
<p>If you see a healthy cluster state, it means that your new link is being used.</p></div>
</div>
</div>
<div class="sect2">
<h3 id="_role_of_ssh_in_proxmox_ve_clusters">
<span>5.9. Role of SSH in Proxmox VE Clusters</span>
 <a class="headerlink" href="#_role_of_ssh_in_proxmox_ve_clusters" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Proxmox VE utilizes SSH tunnels for various features.</p></div>
<div class="ulist"><ul>
<li>
<p>
Proxying console/shell sessions (node and guests)
</p>
<div class="paragraph">
<p>When using the shell for node B while being connected to node A, connects to a
terminal proxy on node A, which is in turn connected to the login shell on node
B via a non-interactive SSH tunnel.</p></div>
</li>
<li>
<p>
VM and CT memory and local-storage migration in <em>secure</em> mode.
</p>
<div class="paragraph">
<p>During the migration, one or more SSH tunnel(s) are established between the
source and target nodes, in order to exchange migration information and
transfer memory and disk contents.</p></div>
</li>
<li>
<p>
Storage replication
</p>
</li>
</ul></div>
<div class="sect3">
<h4 id="_ssh_setup">5.9.1. SSH setup
 <a class="headerlink" href="#_ssh_setup" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>On Proxmox VE systems, the following changes are made to the SSH configuration/setup:</p></div>
<div class="ulist"><ul>
<li>
<p>
the <span class="monospaced">root</span> user’s SSH client config gets setup to prefer <span class="monospaced">AES</span> over <span class="monospaced">ChaCha20</span>
</p>
</li>
<li>
<p>
the <span class="monospaced">root</span> user’s <span class="monospaced">authorized_keys</span> file gets linked to
  <span class="monospaced">/etc/pve/priv/authorized_keys</span>, merging all authorized keys within a cluster
</p>
</li>
<li>
<p>
<span class="monospaced">sshd</span> is configured to allow logging in as root with a password
</p>
</li>
</ul></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Older systems might also have <span class="monospaced">/etc/ssh/ssh_known_hosts</span> set up as symlink
pointing to <span class="monospaced">/etc/pve/priv/known_hosts</span>, containing a merged version of all
node host keys. This system was replaced with explicit host key pinning in
<span class="monospaced">pve-cluster &lt;&lt;INSERT VERSION&gt;&gt;</span>, the symlink can be deconfigured if still in
place by running <span class="monospaced">pvecm updatecerts --unmerge-known-hosts</span>.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="_pitfalls_due_to_automatic_execution_of_span_class_monospaced_bashrc_span_and_siblings">5.9.2. Pitfalls due to automatic execution of <span class="monospaced">.bashrc</span> and siblings
 <a class="headerlink" href="#_pitfalls_due_to_automatic_execution_of_span_class_monospaced_bashrc_span_and_siblings" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>In case you have a custom <span class="monospaced">.bashrc</span>, or similar files that get executed on
login by the configured shell, <span class="monospaced">ssh</span> will automatically run it once the session
is established successfully. This can cause some unexpected behavior, as those
commands may be executed with root permissions on any of the operations
described above. This can cause possible problematic side-effects!</p></div>
<div class="paragraph">
<p>In order to avoid such complications, it’s recommended to add a check in
<span class="monospaced">/root/.bashrc</span> to make sure the session is interactive, and only then run
<span class="monospaced">.bashrc</span> commands.</p></div>
<div class="paragraph">
<p>You can add this snippet at the beginning of your <span class="monospaced">.bashrc</span> file:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># Early exit if not running interactively to avoid side-effects!
case $- in
    *i*) ;;
      *) return;;
esac</pre>
</div></div>
</div>
</div>
<div class="sect2">
<h3 id="_corosync_external_vote_support">
<span>5.10. Corosync External Vote Support</span>
 <a class="headerlink" href="#_corosync_external_vote_support" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>This section describes a way to deploy an external voter in a Proxmox VE cluster.
When configured, the cluster can sustain more node failures without
violating safety properties of the cluster communication.</p></div>
<div class="paragraph">
<p>For this to work, there are two services involved:</p></div>
<div class="ulist"><ul>
<li>
<p>
A QDevice daemon which runs on each Proxmox VE node
</p>
</li>
<li>
<p>
An external vote daemon which runs on an independent server
</p>
</li>
</ul></div>
<div class="paragraph">
<p>As a result, you can achieve higher availability, even in smaller setups (for
example 2+1 nodes).</p></div>
<div class="sect3">
<h4 id="_qdevice_technical_overview">5.10.1. QDevice Technical Overview
 <a class="headerlink" href="#_qdevice_technical_overview" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The Corosync Quorum Device (QDevice) is a daemon which runs on each cluster
node. It provides a configured number of votes to the cluster’s quorum
subsystem, based on an externally running third-party arbitrator’s decision.
Its primary use is to allow a cluster to sustain more node failures than
standard quorum rules allow. This can be done safely as the external device
can see all nodes and thus choose only one set of nodes to give its vote.
This will only be done if said set of nodes can have quorum (again) after
receiving the third-party vote.</p></div>
<div class="paragraph">
<p>Currently, only <em>QDevice Net</em> is supported as a third-party arbitrator. This is
a daemon which provides a vote to a cluster partition, if it can reach the
partition members over the network. It will only give votes to one partition
of a cluster at any time.
It’s designed to support multiple clusters and is almost configuration and
state free. New clusters are handled dynamically and no configuration file
is needed on the host running a QDevice.</p></div>
<div class="paragraph">
<p>The only requirements for the external host are that it needs network access to
the cluster and to have a corosync-qnetd package available. We provide a package
for Debian based hosts, and other Linux distributions should also have a package
available through their respective package manager.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Unlike corosync itself, a QDevice connects to the cluster over TCP/IP.
The daemon can also run outside the LAN of the cluster and isn’t limited to the
low latencies requirements of corosync.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="_supported_setups">5.10.2. Supported Setups
 <a class="headerlink" href="#_supported_setups" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>We support QDevices for clusters with an even number of nodes and recommend
it for 2 node clusters, if they should provide higher availability.
For clusters with an odd node count, we currently discourage the use of
QDevices. The reason for this is the difference in the votes which the QDevice
provides for each cluster type. Even numbered clusters get a single additional
vote, which only increases availability, because if the QDevice
itself fails, you are in the same position as with no QDevice at all.</p></div>
<div class="paragraph">
<p>On the other hand, with an odd numbered cluster size, the QDevice provides
<em>(N-1)</em> votes — where <em>N</em> corresponds to the cluster node count. This
alternative behavior makes sense; if it had only one additional vote, the
cluster could get into a split-brain situation. This algorithm allows for all
nodes but one (and naturally the QDevice itself) to fail. However, there are two
drawbacks to this:</p></div>
<div class="ulist"><ul>
<li>
<p>
If the QNet daemon itself fails, no other node may fail or the cluster
  immediately loses quorum. For example, in a cluster with 15 nodes, 7
  could fail before the cluster becomes inquorate. But, if a QDevice is
  configured here and it itself fails, <strong>no single node</strong> of the 15 may fail.
  The QDevice acts almost as a single point of failure in this case.
</p>
</li>
<li>
<p>
The fact that all but one node plus QDevice may fail sounds promising at
  first, but this may result in a mass recovery of HA services, which could
  overload the single remaining node. Furthermore, a Ceph server will stop
  providing services if only <em>((N-1)/2)</em> nodes or less remain online.
</p>
</li>
</ul></div>
<div class="paragraph">
<p>If you understand the drawbacks and implications, you can decide yourself if
you want to use this technology in an odd numbered cluster setup.</p></div>
</div>
<div class="sect3">
<h4 id="_qdevice_net_setup">5.10.3. QDevice-Net Setup
 <a class="headerlink" href="#_qdevice_net_setup" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>We recommend running any daemon which provides votes to corosync-qdevice as an
unprivileged user. Proxmox VE and Debian provide a package which is already
configured to do so.
The traffic between the daemon and the cluster must be encrypted to ensure a
safe and secure integration of the QDevice in Proxmox VE.</p></div>
<div class="paragraph">
<p>First, install the <em>corosync-qnetd</em> package on your external server</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>external# apt install corosync-qnetd</pre>
</div></div>
<div class="paragraph">
<p>and the <em>corosync-qdevice</em> package on all cluster nodes</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>pve# apt install corosync-qdevice</pre>
</div></div>
<div class="paragraph">
<p>After doing this, ensure that all the nodes in the cluster are online.</p></div>
<div class="paragraph">
<p>You can now set up your QDevice by running the following command on one
of the Proxmox VE nodes:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>pve# pvecm qdevice setup &lt;QDEVICE-IP&gt;</pre>
</div></div>
<div class="paragraph">
<p>The SSH key from the cluster will be automatically copied to the QDevice.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Make sure to setup key-based access for the root user on your external
server, or temporarily allow root login with password during the setup phase.
If you receive an error such as <em>Host key verification failed.</em> at this
stage, running <span class="monospaced">pvecm updatecerts</span> could fix the issue.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>After all the steps have successfully completed, you will see "Done". You can
verify that the QDevice has been set up with:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>pve# pvecm status

...

Votequorum information
~~~~~~~~~~~~~~~~~~~~~
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2
Flags:            Quorate Qdevice

Membership information
~~~~~~~~~~~~~~~~~~~~~~
    Nodeid      Votes    Qdevice Name
    0x00000001      1    A,V,NMW 192.168.22.180 (local)
    0x00000002      1    A,V,NMW 192.168.22.181
    0x00000000      1            Qdevice</pre>
</div></div>
<div class="sect4">
<h5 id="pvecm_qdevice_status_flags">QDevice Status Flags
 <a class="headerlink" href="#pvecm_qdevice_status_flags" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>The status output of the QDevice, as seen above, will usually contain three
columns:</p></div>
<div class="ulist"><ul>
<li>
<p>
<span class="monospaced">A</span> / <span class="monospaced">NA</span>: Alive or Not Alive. Indicates if the communication to the external
    <span class="monospaced">corosync-qnetd</span> daemon works.
</p>
</li>
<li>
<p>
<span class="monospaced">V</span> / <span class="monospaced">NV</span>: If the QDevice will cast a vote for the node. In a split-brain
    situation, where the corosync connection between the nodes is down, but they
    both can still communicate with the external <span class="monospaced">corosync-qnetd</span> daemon,
    only one node will get the vote.
</p>
</li>
<li>
<p>
<span class="monospaced">MW</span> / <span class="monospaced">NMW</span>: Master wins (<span class="monospaced">MV</span>) or not (<span class="monospaced">NMW</span>). Default is <span class="monospaced">NMW</span>, see
   <span class="footnote" data-note="<span class=&quot;monospaced&quot;>votequorum_qdevice_master_wins</span> manual page
   <a href=&quot;https://manpages.debian.org/bookworm/libvotequorum-dev/votequorum_qdevice_master_wins.3.en.html&quot;>https://manpages.debian.org/bookworm/libvotequorum-dev/votequorum_qdevice_master_wins.3.en.html</a>">[<a id="_footnoteref_11" href="#_footnote_11" title="View footnote" class="footnote">11</a>]</span>.
</p>
</li>
<li>
<p>
<span class="monospaced">NR</span>: QDevice is not registered.
</p>
</li>
</ul></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">If your QDevice is listed as <span class="monospaced">Not Alive</span> (<span class="monospaced">NA</span> in the output above),
ensure that port <span class="monospaced">5403</span> (the default port of the qnetd server) of your external
server is reachable via TCP/IP!</td>
</tr></tbody></table>
</div>
</div>
</div>
<div class="sect3">
<h4 id="_frequently_asked_questions">5.10.4. Frequently Asked Questions
 <a class="headerlink" href="#_frequently_asked_questions" title="Permalink to this heading"></a>
</h4>
<div class="sect4">
<h5 id="_tie_breaking">Tie Breaking
 <a class="headerlink" href="#_tie_breaking" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>In case of a tie, where two same-sized cluster partitions cannot see each other
but can see the QDevice, the QDevice chooses one of those partitions randomly
and provides a vote to it.</p></div>
</div>
<div class="sect4">
<h5 id="_possible_negative_implications">Possible Negative Implications
 <a class="headerlink" href="#_possible_negative_implications" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>For clusters with an even node count, there are no negative implications when
using a QDevice. If it fails to work, it is the same as not having a QDevice
at all.</p></div>
</div>
<div class="sect4">
<h5 id="_adding_deleting_nodes_after_qdevice_setup">Adding/Deleting Nodes After QDevice Setup
 <a class="headerlink" href="#_adding_deleting_nodes_after_qdevice_setup" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>If you want to add a new node or remove an existing one from a cluster with a
QDevice setup, you need to remove the QDevice first. After that, you can add or
remove nodes normally. Once you have a cluster with an even node count again,
you can set up the QDevice again as described previously.</p></div>
</div>
<div class="sect4">
<h5 id="_removing_the_qdevice">Removing the QDevice
 <a class="headerlink" href="#_removing_the_qdevice" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>If you used the official <span class="monospaced">pvecm</span> tool to add the QDevice, you can remove it
by running:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>pve# pvecm qdevice remove</pre>
</div></div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_corosync_configuration">
<span>5.11. Corosync Configuration</span>
 <a class="headerlink" href="#_corosync_configuration" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>The <span class="monospaced">/etc/pve/corosync.conf</span> file plays a central role in a Proxmox VE cluster. It
controls the cluster membership and its network.
For further information about it, check the corosync.conf man page:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>man corosync<span style="color: #990000">.</span>conf</tt></pre></div></div>
<div class="paragraph">
<p>For node membership, you should always use the <span class="monospaced">pvecm</span> tool provided by Proxmox VE.
You may have to edit the configuration file manually for other changes.
Here are a few best practice tips for doing this.</p></div>
<div class="sect3">
<h4 id="pvecm_edit_corosync_conf">5.11.1. Edit corosync.conf
 <a class="headerlink" href="#pvecm_edit_corosync_conf" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Editing the corosync.conf file is not always very straightforward. There are
two on each cluster node, one in <span class="monospaced">/etc/pve/corosync.conf</span> and the other in
<span class="monospaced">/etc/corosync/corosync.conf</span>. Editing the one in our cluster file system will
propagate the changes to the local one, but not vice versa.</p></div>
<div class="paragraph">
<p>The configuration will get updated automatically, as soon as the file changes.
This means that changes which can be integrated in a running corosync will take
effect immediately. Thus, you should always make a copy and edit that instead,
to avoid triggering unintended changes when saving the file while editing.</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>cp /etc/pve/corosync<span style="color: #990000">.</span>conf /etc/pve/corosync<span style="color: #990000">.</span>conf<span style="color: #990000">.</span>new</tt></pre></div></div>
<div class="paragraph">
<p>Then, open the config file with your favorite editor, such as <span class="monospaced">nano</span> or
<span class="monospaced">vim.tiny</span>, which come pre-installed on every Proxmox VE node.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Always increment the <em>config_version</em> number after configuration changes;
omitting this can lead to problems.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>After making the necessary changes, create another copy of the current working
configuration file. This serves as a backup if the new configuration fails to
apply or causes other issues.</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>cp /etc/pve/corosync<span style="color: #990000">.</span>conf /etc/pve/corosync<span style="color: #990000">.</span>conf<span style="color: #990000">.</span>bak</tt></pre></div></div>
<div class="paragraph">
<p>Then replace the old configuration file with the new one:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>mv /etc/pve/corosync<span style="color: #990000">.</span>conf<span style="color: #990000">.</span>new /etc/pve/corosync<span style="color: #990000">.</span>conf</tt></pre></div></div>
<div class="paragraph">
<p>You can check if the changes could be applied automatically, using the following
commands:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>systemctl status corosync
journalctl -b -u corosync</tt></pre></div></div>
<div class="paragraph">
<p>If the changes could not be applied automatically, you may have to restart the
corosync service via:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>systemctl restart corosync</tt></pre></div></div>
<div class="paragraph">
<p>On errors, check the troubleshooting section below.</p></div>
</div>
<div class="sect3">
<h4 id="_troubleshooting">5.11.2. Troubleshooting
 <a class="headerlink" href="#_troubleshooting" title="Permalink to this heading"></a>
</h4>
<div class="sect4">
<h5 id="_issue_em_quorum_expected_votes_must_be_configured_em">Issue: <em>quorum.expected_votes must be configured</em>
 <a class="headerlink" href="#_issue_em_quorum_expected_votes_must_be_configured_em" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>When corosync starts to fail and you get the following message in the system log:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>[...]
corosync[1647]:  [QUORUM] Quorum provider: corosync_votequorum failed to initialize.
corosync[1647]:  [SERV  ] Service engine 'corosync_quorum' failed to load for reason
    'configuration error: nodelist or quorum.expected_votes must be configured!'
[...]</pre>
</div></div>
<div class="paragraph">
<p>It means that the hostname you set for a corosync <em>ringX_addr</em> in the
configuration could not be resolved.</p></div>
</div>
<div class="sect4">
<h5 id="_write_configuration_when_not_quorate">Write Configuration When Not Quorate
 <a class="headerlink" href="#_write_configuration_when_not_quorate" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>If you need to change <em>/etc/pve/corosync.conf</em> on a node with no quorum, and you
understand what you are doing, use:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>pvecm expected <span style="color: #993399">1</span></tt></pre></div></div>
<div class="paragraph">
<p>This sets the expected vote count to 1 and makes the cluster quorate. You can
then fix your configuration, or revert it back to the last working backup.</p></div>
<div class="paragraph">
<p>This is not enough if corosync cannot start anymore. In that case, it is best to
edit the local copy of the corosync configuration in
<em>/etc/corosync/corosync.conf</em>, so that corosync can start again. Ensure that on
all nodes, this configuration has the same content to avoid split-brain
situations.</p></div>
</div>
</div>
<div class="sect3">
<h4 id="pvecm_corosync_conf_glossary">5.11.3. Corosync Configuration Glossary
 <a class="headerlink" href="#pvecm_corosync_conf_glossary" title="Permalink to this heading"></a>
</h4>
<div class="dlist"><dl>
<dt class="hdlist1">
ringX_addr
</dt>
<dd>
<p>
This names the different link addresses for the Kronosnet connections between
nodes.
</p>
</dd>
</dl></div>
</div>
</div>
<div class="sect2">
<h3 id="_cluster_cold_start">
<span>5.12. Cluster Cold Start</span>
 <a class="headerlink" href="#_cluster_cold_start" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>It is obvious that a cluster is not quorate when all nodes are
offline. This is a common case after a power failure.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">It is always a good idea to use an uninterruptible power supply
(“UPS”, also called “battery backup”) to avoid this state, especially if
you want HA.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>On node startup, the <span class="monospaced">pve-guests</span> service is started and waits for
quorum. Once quorate, it starts all guests which have the <span class="monospaced">onboot</span>
flag set.</p></div>
<div class="paragraph">
<p>When you turn on nodes, or when power comes back after power failure,
it is likely that some nodes will boot faster than others. Please keep in
mind that guest startup is delayed until you reach quorum.</p></div>
</div>
<div class="sect2">
<h3 id="pvecm_next_id_range">
<span>5.13. Guest VMID Auto-Selection</span>
 <a class="headerlink" href="#pvecm_next_id_range" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>When creating new guests the web interface will ask the backend for a free VMID
automatically. The default range for searching is <span class="monospaced">100</span> to <span class="monospaced">1000000</span> (lower
than the maximal allowed VMID enforced by the schema).</p></div>
<div class="paragraph">
<p>Sometimes admins either want to allocate new VMIDs in a separate range, for
example to easily separate temporary VMs with ones that choose a VMID manually.
Other times its just desired to provided a stable length VMID, for which
setting the lower boundary to, for example, <span class="monospaced">100000</span> gives much more room for.</p></div>
<div class="paragraph">
<p>To accommodate this use case one can set either lower, upper or both boundaries
via the <span class="monospaced">datacenter.cfg</span> configuration file, which can be edited in the web
interface under <em>Datacenter</em> → <em>Options</em>.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">The range is only used for the next-id API call, so it isn’t a hard
limit.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect2">
<h3 id="_guest_migration">
<span>5.14. Guest Migration</span>
 <a class="headerlink" href="#_guest_migration" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Migrating virtual guests to other nodes is a useful feature in a
cluster. There are settings to control the behavior of such
migrations. This can be done via the configuration file
<span class="monospaced">datacenter.cfg</span> or for a specific migration via API or command-line
parameters.</p></div>
<div class="paragraph">
<p>It makes a difference if a guest is online or offline, or if it has
local resources (like a local disk).</p></div>
<div class="paragraph">
<p>For details about virtual machine migration, see the
<a href="#qm_migration">QEMU/KVM Migration Chapter</a>.</p></div>
<div class="paragraph">
<p>For details about container migration, see the
<a href="#pct_migration">Container Migration Chapter</a>.</p></div>
<div class="sect3">
<h4 id="_migration_type">5.14.1. Migration Type
 <a class="headerlink" href="#_migration_type" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The migration type defines if the migration data should be sent over an
encrypted (<span class="monospaced">secure</span>) channel or an unencrypted (<span class="monospaced">insecure</span>) one.
Setting the migration type to <span class="monospaced">insecure</span> means that the RAM content of a
virtual guest is also transferred unencrypted, which can lead to
information disclosure of critical data from inside the guest (for
example, passwords or encryption keys).</p></div>
<div class="paragraph">
<p>Therefore, we strongly recommend using the secure channel if you do
not have full control over the network and can not guarantee that no
one is eavesdropping on it.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Storage migration does not follow this setting. Currently, it
always sends the storage content over a secure channel.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>Encryption requires a lot of computing power, so this setting is often
changed to <span class="monospaced">insecure</span> to achieve better performance. The impact on
modern systems is lower because they implement AES encryption in
hardware. The performance impact is particularly evident in fast
networks, where you can transfer 10 Gbps or more.</p></div>
</div>
<div class="sect3">
<h4 id="_migration_network">5.14.2. Migration Network
 <a class="headerlink" href="#_migration_network" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>By default, Proxmox VE uses the network in which cluster communication
takes place to send the migration traffic. This is not optimal both because
sensitive cluster traffic can be disrupted and this network may not
have the best bandwidth available on the node.</p></div>
<div class="paragraph">
<p>Setting the migration network parameter allows the use of a dedicated
network for all migration traffic. In addition to the memory,
this also affects the storage traffic for offline migrations.</p></div>
<div class="paragraph">
<p>The migration network is set as a network using CIDR notation. This
has the advantage that you don’t have to set individual IP addresses
for each node. Proxmox VE can determine the real address on the
destination node from the network specified in the CIDR form. To
enable this, the network must be specified so that each node has exactly one
IP in the respective network.</p></div>
<div class="sect4">
<h5 id="_example">Example
 <a class="headerlink" href="#_example" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>We assume that we have a three-node setup, with three separate
networks. One for public communication with the Internet, one for
cluster communication, and a very fast one, which we want to use as a
dedicated network for migration.</p></div>
<div class="paragraph">
<p>A network configuration for such a setup might look as follows:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>iface eno1 inet manual

# public network
auto vmbr0
iface vmbr0 inet static
    address 192.X.Y.57/24
    gateway 192.X.Y.1
    bridge-ports eno1
    bridge-stp off
    bridge-fd 0

# cluster network
auto eno2
iface eno2 inet static
    address  10.1.1.1/24

# fast network
auto eno3
iface eno3 inet static
    address  10.1.2.1/24</pre>
</div></div>
<div class="paragraph">
<p>Here, we will use the network 10.1.2.0/24 as a migration network. For
a single migration, you can do this using the <span class="monospaced">migration_network</span>
parameter of the command-line tool:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># qm migrate 106 tre --online --migration_network 10.1.2.0/24</pre>
</div></div>
<div class="paragraph">
<p>To configure this as the default network for all migrations in the
cluster, set the <span class="monospaced">migration</span> property of the <span class="monospaced">/etc/pve/datacenter.cfg</span>
file:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># use dedicated migration network
migration: secure,network=10.1.2.0/24</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">The migration type must always be set when the migration network
is set in <span class="monospaced">/etc/pve/datacenter.cfg</span>.</td>
</tr></tbody></table>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="chapter_pmxcfs">
6. Proxmox Cluster File System (pmxcfs)
 <a class="headerlink" href="#chapter_pmxcfs" title="Permalink to this heading"></a>
</h2>
<div class="sectionbody">
<div class="paragraph">
<p>The Proxmox Cluster file system (“pmxcfs”) is a database-driven file
system for storing configuration files, replicated in real time to all
cluster nodes using <span class="monospaced">corosync</span>. We use this to store all Proxmox VE related
configuration files.</p></div>
<div class="paragraph">
<p>Although the file system stores all data inside a persistent database on disk,
a copy of the data resides in RAM. This imposes restrictions on the maximum
size, which is currently 128 MiB. This is still enough to store the
configuration of several thousand virtual machines.</p></div>
<div class="paragraph">
<p>This system provides the following advantages:</p></div>
<div class="ulist"><ul>
<li>
<p>
Seamless replication of all configuration to all nodes in real time
</p>
</li>
<li>
<p>
Provides strong consistency checks to avoid duplicate VM IDs
</p>
</li>
<li>
<p>
Read-only when a node loses quorum
</p>
</li>
<li>
<p>
Automatic updates of the corosync cluster configuration to all nodes
</p>
</li>
<li>
<p>
Includes a distributed locking mechanism
</p>
</li>
</ul></div>
<div class="sect2">
<h3 id="_posix_compatibility">
<span>6.1. POSIX Compatibility</span>
 <a class="headerlink" href="#_posix_compatibility" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>The file system is based on FUSE, so the behavior is POSIX like. But
some feature are simply not implemented, because we do not need them:</p></div>
<div class="ulist"><ul>
<li>
<p>
You can just generate normal files and directories, but no symbolic
  links, …
</p>
</li>
<li>
<p>
You can’t rename non-empty directories (because this makes it easier
  to guarantee that VMIDs are unique).
</p>
</li>
<li>
<p>
You can’t change file permissions (permissions are based on paths)
</p>
</li>
<li>
<p>
<span class="monospaced">O_EXCL</span> creates were not atomic (like old NFS)
</p>
</li>
<li>
<p>
<span class="monospaced">O_TRUNC</span> creates are not atomic (FUSE restriction)
</p>
</li>
</ul></div>
</div>
<div class="sect2">
<h3 id="_file_access_rights">
<span>6.2. File Access Rights</span>
 <a class="headerlink" href="#_file_access_rights" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>All files and directories are owned by user <span class="monospaced">root</span> and have group
<span class="monospaced">www-data</span>. Only root has write permissions, but group <span class="monospaced">www-data</span> can
read most files. Files below the following paths are only accessible by root:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>/etc/pve/priv/
/etc/pve/nodes/${NAME}/priv/</pre>
</div></div>
</div>
<div class="sect2">
<h3 id="_technology">
<span>6.3. Technology</span>
 <a class="headerlink" href="#_technology" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>We use the <a href="https://www.corosync.org">Corosync Cluster Engine</a> for
cluster communication, and <a href="https://www.sqlite.org">SQlite</a> for the
database file. The file system is implemented in user space using
<a href="https://github.com/libfuse/libfuse">FUSE</a>.</p></div>
</div>
<div class="sect2">
<h3 id="_file_system_layout">
<span>6.4. File System Layout</span>
 <a class="headerlink" href="#_file_system_layout" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>The file system is mounted at:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>/etc/pve</pre>
</div></div>
<div class="sect3">
<h4 id="_files">6.4.1. Files
 <a class="headerlink" href="#_files" title="Permalink to this heading"></a>
</h4>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:50%;">
<col style="width:50%;">
</colgroup><tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">authkey.pub</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Public key used by the ticket system</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">ceph.conf</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Ceph configuration file (note: /etc/ceph/ceph.conf is a symbolic link to this)</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">corosync.conf</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Corosync cluster configuration file (prior to Proxmox VE 4.x, this file was called cluster.conf)</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">datacenter.cfg</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Proxmox VE datacenter-wide configuration (keyboard layout, proxy, …)</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">domains.cfg</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Proxmox VE authentication domains</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">firewall/cluster.fw</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Firewall configuration applied to all nodes</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">firewall/&lt;NAME&gt;.fw</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Firewall configuration for individual nodes</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">firewall/&lt;VMID&gt;.fw</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Firewall configuration for VMs and containers</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">ha/crm_commands</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Displays HA operations that are currently being carried out by the CRM</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">ha/manager_status</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">JSON-formatted information regarding HA services on the cluster</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">ha/resources.cfg</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Resources managed by high availability, and their current state</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">nodes/&lt;NAME&gt;/config</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Node-specific configuration</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">nodes/&lt;NAME&gt;/lxc/&lt;VMID&gt;.conf</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">VM configuration data for LXC containers</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">nodes/&lt;NAME&gt;/openvz/</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Prior to Proxmox VE 4.0, used for container configuration data (deprecated, removed soon)</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">nodes/&lt;NAME&gt;/pve-ssl.key</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Private SSL key for <span class="monospaced">pve-ssl.pem</span></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">nodes/&lt;NAME&gt;/pve-ssl.pem</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Public SSL certificate for web server (signed by cluster CA)</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">nodes/&lt;NAME&gt;/pveproxy-ssl.key</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Private SSL key for <span class="monospaced">pveproxy-ssl.pem</span> (optional)</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">nodes/&lt;NAME&gt;/pveproxy-ssl.pem</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Public SSL certificate (chain) for web server (optional override for <span class="monospaced">pve-ssl.pem</span>)</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">nodes/&lt;NAME&gt;/qemu-server/&lt;VMID&gt;.conf</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">VM configuration data for KVM VMs</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">priv/authkey.key</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Private key used by ticket system</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">priv/authorized_keys</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">SSH keys of cluster members for authentication</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">priv/ceph*</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Ceph authentication keys and associated capabilities</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">priv/known_hosts</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">SSH keys of the cluster members for verification</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">priv/lock/*</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Lock files used by various services to ensure safe cluster-wide operations</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">priv/pve-root-ca.key</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Private key of cluster CA</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">priv/shadow.cfg</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Shadow password file for PVE Realm users</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">priv/storage/&lt;STORAGE-ID&gt;.pw</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Contains the password of a storage in plain text</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">priv/tfa.cfg</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Base64-encoded two-factor authentication configuration</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">priv/token.cfg</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">API token secrets of all tokens</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">pve-root-ca.pem</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Public certificate of cluster CA</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">pve-www.key</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Private key used for generating CSRF tokens</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">sdn/*</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Shared configuration files for Software Defined Networking (SDN)</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">status.cfg</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Proxmox VE external metrics server configuration</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">storage.cfg</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Proxmox VE storage configuration</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">user.cfg</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Proxmox VE access control configuration (users/groups/…)</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">virtual-guest/cpu-models.conf</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">For storing custom CPU models</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">vzdump.cron</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Cluster-wide vzdump backup-job schedule</p></td>
</tr>
</tbody>
</table>
</div>
<div class="sect3">
<h4 id="_symbolic_links">6.4.2. Symbolic links
 <a class="headerlink" href="#_symbolic_links" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Certain directories within the cluster file system use symbolic links, in order
to point to a node’s own configuration files. Thus, the files pointed to in the
table below refer to different files on each node of the cluster.</p></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:50%;">
<col style="width:50%;">
</colgroup><tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">local</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">nodes/&lt;LOCAL_HOST_NAME&gt;</span></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">lxc</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">nodes/&lt;LOCAL_HOST_NAME&gt;/lxc/</span></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">openvz</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">nodes/&lt;LOCAL_HOST_NAME&gt;/openvz/</span> (deprecated, removed soon)</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">qemu-server</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">nodes/&lt;LOCAL_HOST_NAME&gt;/qemu-server/</span></p></td>
</tr>
</tbody>
</table>
</div>
<div class="sect3">
<h4 id="_special_status_files_for_debugging_json">6.4.3. Special status files for debugging (JSON)
 <a class="headerlink" href="#_special_status_files_for_debugging_json" title="Permalink to this heading"></a>
</h4>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:50%;">
<col style="width:50%;">
</colgroup><tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">.version</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">File versions (to detect file modifications)</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">.members</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Info about cluster members</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">.vmlist</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">List of all VMs</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">.clusterlog</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Cluster log (last 50 entries)</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">.rrd</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">RRD data (most recent entries)</p></td>
</tr>
</tbody>
</table>
</div>
<div class="sect3">
<h4 id="_enable_disable_debugging">6.4.4. Enable/Disable debugging
 <a class="headerlink" href="#_enable_disable_debugging" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>You can enable verbose syslog messages with:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>echo "1" &gt;/etc/pve/.debug</pre>
</div></div>
<div class="paragraph">
<p>And disable verbose syslog messages with:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>echo "0" &gt;/etc/pve/.debug</pre>
</div></div>
</div>
</div>
<div class="sect2">
<h3 id="_recovery">
<span>6.5. Recovery</span>
 <a class="headerlink" href="#_recovery" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>If you have major problems with your Proxmox VE host, for example hardware
issues, it could be helpful to copy the pmxcfs database file
<span class="monospaced">/var/lib/pve-cluster/config.db</span>, and move it to a new Proxmox VE
host. On the new host (with nothing running), you need to stop the
<span class="monospaced">pve-cluster</span> service and replace the <span class="monospaced">config.db</span> file (required permissions
<span class="monospaced">0600</span>). Following this, adapt <span class="monospaced">/etc/hostname</span> and <span class="monospaced">/etc/hosts</span> according to the
lost Proxmox VE host, then reboot and check (and don’t forget your
VM/CT data).</p></div>
<div class="sect3">
<h4 id="_remove_cluster_configuration">6.5.1. Remove Cluster Configuration
 <a class="headerlink" href="#_remove_cluster_configuration" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The recommended way is to reinstall the node after you remove it from
your cluster. This ensures that all secret cluster/ssh keys and any
shared configuration data is destroyed.</p></div>
<div class="paragraph">
<p>In some cases, you might prefer to put a node back to local mode without
reinstalling, which is described in
<a href="#pvecm_separate_node_without_reinstall">Separate A Node Without Reinstalling</a></p></div>
</div>
<div class="sect3">
<h4 id="_recovering_moving_guests_from_failed_nodes">6.5.2. Recovering/Moving Guests from Failed Nodes
 <a class="headerlink" href="#_recovering_moving_guests_from_failed_nodes" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>For the guest configuration files in <span class="monospaced">nodes/&lt;NAME&gt;/qemu-server/</span> (VMs) and
<span class="monospaced">nodes/&lt;NAME&gt;/lxc/</span> (containers), Proxmox VE sees the containing node <span class="monospaced">&lt;NAME&gt;</span> as the
owner of the respective guest. This concept enables the usage of local locks
instead of expensive cluster-wide locks for preventing concurrent guest
configuration changes.</p></div>
<div class="paragraph">
<p>As a consequence, if the owning node of a guest fails (for example, due to a power
outage, fencing event, etc.), a regular migration is not possible (even if all
the disks are located on shared storage), because such a local lock on the
(offline) owning node is unobtainable. This is not a problem for HA-managed
guests, as Proxmox VE’s High Availability stack includes the necessary
(cluster-wide) locking and watchdog functionality to ensure correct and
automatic recovery of guests from fenced nodes.</p></div>
<div class="paragraph">
<p>If a non-HA-managed guest has only shared disks (and no other local resources
which are only available on the failed node), a manual recovery
is possible by simply moving the guest configuration file from the failed
node’s directory in <span class="monospaced">/etc/pve/</span> to an online node’s directory (which changes the
logical owner or location of the guest).</p></div>
<div class="paragraph">
<p>For example, recovering the VM with ID <span class="monospaced">100</span> from an offline <span class="monospaced">node1</span> to another
node <span class="monospaced">node2</span> works by running the following command as root on any member node
of the cluster:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>mv /etc/pve/nodes/node1/qemu-server/100.conf /etc/pve/nodes/node2/qemu-server/</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">Before manually recovering a guest like this, make absolutely sure
that the failed source node is really powered off/fenced. Otherwise Proxmox VE’s
locking principles are violated by the <span class="monospaced">mv</span> command, which can have unexpected
consequences.</td>
</tr></tbody></table>
</div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">Guests with local disks (or other local resources which are only
available on the offline node) are not recoverable like this. Either wait for the
failed node to rejoin the cluster or restore such guests from backups.</td>
</tr></tbody></table>
</div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="chapter_storage">
7. Proxmox VE Storage
 <a class="headerlink" href="#chapter_storage" title="Permalink to this heading"></a>
</h2>
<div class="sectionbody">
<div class="paragraph">
<p>The Proxmox VE storage model is very flexible. Virtual machine images
can either be stored on one or several local storages, or on shared
storage like NFS or iSCSI (NAS, SAN). There are no limits, and you may
configure as many storage pools as you like. You can use all
storage technologies available for Debian Linux.</p></div>
<div class="paragraph">
<p>One major benefit of storing VMs on shared storage is the ability to
live-migrate running machines without any downtime, as all nodes in
the cluster have direct access to VM disk images. There is no need to
copy VM image data, so live migration is very fast in that case.</p></div>
<div class="paragraph">
<p>The storage library (package <span class="monospaced">libpve-storage-perl</span>) uses a flexible
plugin system to provide a common interface to all storage types. This
can be easily adopted to include further storage types in the future.</p></div>
<div class="sect2">
<h3 id="_storage_types">
<span>7.1. Storage Types</span>
 <a class="headerlink" href="#_storage_types" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>There are basically two different classes of storage types:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
File level storage
</dt>
<dd>
<p>
File level based storage technologies allow access to a fully featured (POSIX)
file system.  They are in general more flexible than any Block level storage
(see below), and allow you to store content of any type. ZFS is probably the
most advanced system, and it has full support for snapshots and clones.
</p>
</dd>
<dt class="hdlist1">
Block level storage
</dt>
<dd>
<p>
Allows to store large <em>raw</em> images. It is usually not possible to store
other files (ISO, backups, ..) on such storage types. Most modern
block level storage implementations support snapshots and clones.
RADOS and GlusterFS are distributed systems, replicating storage
data to different nodes.
</p>
</dd>
</dl></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<caption class="title">Table 2. Available storage types</caption>
<colgroup><col style="width:28%;">
<col style="width:14%;">
<col style="width:14%;">
<col style="width:14%;">
<col style="width:14%;">
<col style="width:14%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Description    </th>
<th class="tableblock halign-left valign-top">Plugin type </th>
<th class="tableblock halign-left valign-top">Level  </th>
<th class="tableblock halign-left valign-top">Shared</th>
<th class="tableblock halign-left valign-top">Snapshots</th>
<th class="tableblock halign-left valign-top">Stable</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">ZFS (local)</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">zfspool</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">both<sup>1</sup></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">no</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">Directory</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">dir</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">file</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">no</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">no<sup>2</sup></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">BTRFS</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">btrfs</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">file</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">no</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">technology preview</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">NFS</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">nfs</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">file</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">no<sup>2</sup></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">CIFS</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">cifs</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">file</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">no<sup>2</sup></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">Proxmox Backup</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">pbs</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">both</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">n/a</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">GlusterFS</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">glusterfs</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">file</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">no<sup>2</sup></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">CephFS</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">cephfs</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">file</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">LVM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">lvm</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">block</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">no<sup>3</sup></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">no</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">LVM-thin</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">lvmthin</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">block</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">no</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">iSCSI/kernel</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">iscsi</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">block</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">no</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">iSCSI/libiscsi</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">iscsidirect</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">block</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">no</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">Ceph/RBD</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">rbd</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">block</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">ZFS over iSCSI</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">zfs</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">block</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
</tr>
</tbody>
</table>
<div class="paragraph">
<p><sup>1</sup>: Disk images for VMs are stored in ZFS volume (zvol) datasets, which provide
block device functionality.</p></div>
<div class="paragraph">
<p><sup>2</sup>: On file based storages, snapshots are possible with the <em>qcow2</em> format.</p></div>
<div class="paragraph">
<p><sup>3</sup>: It is possible to use LVM on top of an iSCSI or FC-based storage.
That way you get a <span class="monospaced">shared</span> LVM storage</p></div>
<div class="sect3">
<h4 id="_thin_provisioning">7.1.1. Thin Provisioning
 <a class="headerlink" href="#_thin_provisioning" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>A number of storages, and the QEMU image format <span class="monospaced">qcow2</span>, support <em>thin
provisioning</em>.  With thin provisioning activated, only the blocks that
the guest system actually use will be written to the storage.</p></div>
<div class="paragraph">
<p>Say for instance you create a VM with a 32GB hard disk, and after
installing the guest system OS, the root file system of the VM contains
3 GB of data.  In that case only 3GB are written to the storage, even
if the guest VM sees a 32GB hard drive. In this way thin provisioning
allows you to create disk images which are larger than the currently
available storage blocks. You can create large disk images for your
VMs, and when the need arises, add more disks to your storage without
resizing the VMs' file systems.</p></div>
<div class="paragraph">
<p>All storage types which have the “Snapshots” feature also support thin
provisioning.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Caution" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKdUlEQVRoge1Ze1AV1x3+zt279wEi
DWCYGzRVktqa1MEmmtbWR22ncXxUrTDWV/5IqG2wUAUfmUwSkk6V+EAJCEQC6figwZBqZtRxqukf
tebRZBpFG1S0hiRErwiaKK977+6eX//YPXt37+UiIJlMZnpmdnb33PP4vt97z2VEhG9yc3zdAO60
/Z/A192+8QScX8Wifr+fWltbzffU1FT4fD72Vew15ASampqovr4eBw8eNPvGjRuHzMxMmj9//tCT
IKIhu958801yuVwEoNdr48aNNJT7EdHQEdi/fz/JshwTvLiKioqGlMSQLFJfX0+MMRvQsWNSqXLz
H2jez78fRWLTpk1DRuKOF6irqyOn02kDOP8XGdTz+VFSP91Hatu79NRvZ0SR2LJly5CQuKPJtbW1
JEmSDdivHp1AwctHSblYSsqFElLOF5PavIvWZU+LIrF169Y7JjHoibt3744CnzXrIQpeOUbKhVJS
mkpIPV9MyrnNpJwtIuVSNa15/MdRJLZt23ZHJBjRwIu5N954gxYvXgzOudm3aO5E1L5SBNZ1Hoxr
INLAyH6HJxVP/WkXtu9+37ZeSUkJVq9ePagQO+BMXFNTEwV+ybxHsPeVIjg6zwOkgSWNhuOuUUBC
KhxJY0AggDSgpwWbn1mO1csftq1ZUFCA0tLSQZXFA0pkVVVVlJOTA6vWsmY9hF07N0AS4OU48LgR
WLO+FPFxboxMS8G5hlPYmPMI4jwM1P0Jtj67DJyrKHvtNADdjPPz8+FwOCgvL29gmuivrVVUVESF
yqxZD1PoyjFSL+7QHbapmJTml+n9EzVRtl676ZekNKwl5VQ+KSfzKHR2E+UutodYxhiVl5cPyCf6
ZUJlZWWUm5trk/wTi6bgtVdf1M2Gq2DEwUgDNBWhUDBqjfSRwwGoIOEP3c3Y/uxirFw0zibMvLw8
VFZW9tucbkugoqKCVq1aZQO/YslUVJU+D9bVBIDroLgK4hygEDweybYGY8AD6d8yftfASL+j+2O8
9NwiPJk51kYiNzcXO3fu7BeJPgkUFxdTbm6ure/JZdNRub0QrOMcGFcBrgLgAHEAGqD2ID7OZZsz
Ji0RcR6HAV4DkaqPJxXouoSywkz8buF9NhIrV65EVVXVbUnEJHDixAlat26drS9n+U+xo/hZMMNh
yZA+uAbAIKOGMCJlmG3eA/clAYaJEamGBlRdC6QBXZewo3AhVswfbSORk5ODDz/8sE8SvRLw+/1U
XV0dBb5MgOe6LUNIEqpOwgCUlOCESw4HuAfG3KX7B2kAGaTJ0Jro77yI8ucX4DfzRtlIHD58GH6/
PyaJXgk0NjZi79695vuUiffrkr91Vt8cHAwc4FZJGiZEKqDcwsi0ZHP+d749POy8MS7iGtBxARWF
8/CDsQnm3BdeeAHWj6N+EYhspSVbgI5zYYkZGmAwzIEbwA0tUM9NfO+7aeb89HuGRQBWLcSNZxj3
zvOYMzk5NpjBEHi99mXD5lXdjqGDJh42h7CEOXhnK3448X5z/shUb1jSpIEM8yFuzOUcjAsNcnR0
KXdG4MEHH8TEiRPN9z0H3sfRE41m+GNcs9i0LjkGTTcrUoGem5g7/V5z/ohElw4O+jxGYQ3qJqmB
jN/+cbINtcfCJvPYY4+hr3qtVwI+n49lZmaa71fbbuH3hX/FycbPQKSCoIIMQEQcehjVo5Gw8/F3
3cSvH03H2HuHY1icQzcvYWKGIMiITMwQwjtnruPpqk9w/ZZq7p2RkYEJEyaAYrCIWY36/X7KyMhA
W1ub2XfP3Qmo3ZKFaQ+P0gmYIdSoOkU4lWSwxGRAdqHL/xm86k27w0blAw3/PncD6yo/xrsfdZj7
paSkoLGxESNGjAgDZsxWK8X0AZ/Px44fP460tLAzXrnWgUX5r+PYuxf1mG/Gf80sofWy2YuWbg/+
uONtPLHhPRw+0WIHL8hbwG95rcUGPiEhAYcOHUJSUhI456YZRWqiz+8BIqLm5mZMmjQJN27cMPsT
E9zYtWE25k4bYzh3mABIBRwOHPgPx+LH/6yDiXPi6pHZcEpk+I5qmtGpCzdQecCPPUevmevLsowj
R45gypQpcDgc5mVowKaJmBoQTEePHo2GhgaMGxcuum52BLHsqUOoP3oWgLB9Sz7QgpgxJoThw/SS
IutnaXBKZEpcgD998Uvs+3ubDbzT6URNTQ0mTZoEVVWhaRo459A0TeCy4etVA+JH8RvnHK2trZgz
Zw5Onz5tjnPJDrxSOAPLZt1nSVIqwDmIVJw6dx1nLt3A3J/cjZThkhE29cvf3o2KA59ja91lcz3G
GDZv3oylS5fC7XbD6XTC6XRCkiRIktS7Jm5HQLDXNA3Xr19HVlYWPvjgA4vEGLav+RHmTxsJX4rb
8AsO4kbOIG6GzTD4HvzlmB/PVH9q2zc/Px/Z2dnwer1wu92QZRkulwuSJEGWZZMEY8wkEGVCVvCC
AOcciqIgPj4e+/btw9SpU83xqkpYteU9HHm7Bf62rnC2RjikisTFSMPV9m4ceucannvVDn7p0qVY
uHAhgsEggsEgQqFQlAlZnVm0PjOxIKFpGogIqqrC5XKhuroa06dPt4wDcl78F+qPNeNKe6cRoTjs
ZYOKK+3dON7wBQrKm2H5pMbMmTOxZMkSBAIB9PT0oKenB4FAAIqiQFEUU4hRRyq3I2AlIZ4553A6
nSgvL8fs2bNtY9eVNaDub58YmhDZWpe+v70bJ5tu4clt/4WihqU4efJkrFixAkRkAg4Gg1AUBaqq
muDF/pGtX7WQNXeIZ0mSUFRUhAULFtjGPl3ZiJdev4QrbZ16zUQqrrZ342JLN5ZvaEJ3ICz68ePH
IycnB5qmmZIW4EOhUJTZWEKo+Rx1KsEYY9Zk4XA4IEmSKXnOuRkVJElCYWEhvF4v6urqzDVK9n2M
zu4Qnl5+L9q/7MHltgBWbr9kA5+eno7c3FzIsoyI5GrTeCzgosU8VmGMmSGLiEwSsixH2WNBQQEA
2EhUH/wc7330BaaOH479/2zHtS/CFabP58OaNWvg9Xp1EE4nZFmGLMtwu93wer1wuVy2ECpJko2I
iTNWJrZGI+HEwi6FisUVCAQQCASwZ88eVFZW9lk9JicnY/369fD5fPB4PCbo+Ph4k4TH44HL5YLX
64XX6zXH9EKE9aUBRkTEGLNJP1Kd4nI4HMjOzkZ8fDyKi4ttJ3eiJSYmYu3atRg1apQJzuv1wuPx
wOPxwOFwmEBFHnA6nXC5XGYSiywlbns2SnoDANN0hMNZnS0UCpkaOnPmDN566y20tLSY2ktLS8OM
GTOQkpICt9uNuLg4M2EJ8CJZSZJkZmFBwkrAWpH2+3BXEBEkRIhTVdVMOCL5qKqKy5cvQ1EUaJoG
VVWRnKx/Jlpt3eVy2QCKu9C68AFr9o0spwd0Oh1ZYggCIlNaiQktiXdhZgKQABcp3ci7LMtM7B0J
fsAErJoAYItG1mdr+hfvVlACtHDIXkploYXbHvQOikDEe6/ZmjFmAhf9ZvJxOk2Qol88W4kYz32S
GNQfHLG+TyMTkAVEr88xQfVnkBg7GAKixSIykDYQsL21/wFkW/B5QqT9lwAAAABJRU5ErkJggg==">
</td>
<td class="content">If a storage runs full, all guests using volumes on that
storage receive IO errors. This can cause file system inconsistencies
and may corrupt your data. So it is advisable to avoid
over-provisioning of your storage resources, or carefully observe
free space to avoid such conditions.</td>
</tr></tbody></table>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_storage_configuration">
<span>7.2. Storage Configuration</span>
 <a class="headerlink" href="#_storage_configuration" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>All Proxmox VE related storage configuration is stored within a single text
file at <span class="monospaced">/etc/pve/storage.cfg</span>. As this file is within <span class="monospaced">/etc/pve/</span>, it
gets automatically distributed to all cluster nodes. So all nodes
share the same storage configuration.</p></div>
<div class="paragraph">
<p>Sharing storage configuration makes perfect sense for shared storage,
because the same “shared” storage is accessible from all nodes. But it is
also useful for local storage types. In this case such local storage
is available on all nodes, but it is physically different and can have
totally different content.</p></div>
<div class="sect3">
<h4 id="_storage_pools">7.2.1. Storage Pools
 <a class="headerlink" href="#_storage_pools" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Each storage pool has a <span class="monospaced">&lt;type&gt;</span>, and is uniquely identified by its
<span class="monospaced">&lt;STORAGE_ID&gt;</span>. A pool configuration looks like this:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>&lt;type&gt;: &lt;STORAGE_ID&gt;
        &lt;property&gt; &lt;value&gt;
        &lt;property&gt; &lt;value&gt;
        &lt;property&gt;
        ...</pre>
</div></div>
<div class="paragraph">
<p>The <span class="monospaced">&lt;type&gt;: &lt;STORAGE_ID&gt;</span> line starts the pool definition, which is then
followed by a list of properties. Most properties require a value. Some have
reasonable defaults, in which case you can omit the value.</p></div>
<div class="paragraph">
<p>To be more specific, take a look at the default storage configuration
after installation. It contains one special local storage pool named
<span class="monospaced">local</span>, which refers to the directory <span class="monospaced">/var/lib/vz</span> and is always
available. The Proxmox VE installer creates additional storage entries
depending on the storage type chosen at installation time.</p></div>
<div class="listingblock">
<div class="title">Default storage configuration (<span class="monospaced">/etc/pve/storage.cfg</span>)</div>
<div class="content monospaced">
<pre>dir: local
        path /var/lib/vz
        content iso,vztmpl,backup

# default image store on LVM based installation
lvmthin: local-lvm
        thinpool data
        vgname pve
        content rootdir,images

# default image store on ZFS based installation
zfspool: local-zfs
        pool rpool/data
        sparse
        content images,rootdir</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Caution" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKdUlEQVRoge1Ze1AV1x3+zt279wEi
DWCYGzRVktqa1MEmmtbWR22ncXxUrTDWV/5IqG2wUAUfmUwSkk6V+EAJCEQC6figwZBqZtRxqukf
tebRZBpFG1S0hiRErwiaKK977+6eX//YPXt37+UiIJlMZnpmdnb33PP4vt97z2VEhG9yc3zdAO60
/Z/A192+8QScX8Wifr+fWltbzffU1FT4fD72Vew15ASampqovr4eBw8eNPvGjRuHzMxMmj9//tCT
IKIhu958801yuVwEoNdr48aNNJT7EdHQEdi/fz/JshwTvLiKioqGlMSQLFJfX0+MMRvQsWNSqXLz
H2jez78fRWLTpk1DRuKOF6irqyOn02kDOP8XGdTz+VFSP91Hatu79NRvZ0SR2LJly5CQuKPJtbW1
JEmSDdivHp1AwctHSblYSsqFElLOF5PavIvWZU+LIrF169Y7JjHoibt3744CnzXrIQpeOUbKhVJS
mkpIPV9MyrnNpJwtIuVSNa15/MdRJLZt23ZHJBjRwIu5N954gxYvXgzOudm3aO5E1L5SBNZ1Hoxr
INLAyH6HJxVP/WkXtu9+37ZeSUkJVq9ePagQO+BMXFNTEwV+ybxHsPeVIjg6zwOkgSWNhuOuUUBC
KhxJY0AggDSgpwWbn1mO1csftq1ZUFCA0tLSQZXFA0pkVVVVlJOTA6vWsmY9hF07N0AS4OU48LgR
WLO+FPFxboxMS8G5hlPYmPMI4jwM1P0Jtj67DJyrKHvtNADdjPPz8+FwOCgvL29gmuivrVVUVESF
yqxZD1PoyjFSL+7QHbapmJTml+n9EzVRtl676ZekNKwl5VQ+KSfzKHR2E+UutodYxhiVl5cPyCf6
ZUJlZWWUm5trk/wTi6bgtVdf1M2Gq2DEwUgDNBWhUDBqjfSRwwGoIOEP3c3Y/uxirFw0zibMvLw8
VFZW9tucbkugoqKCVq1aZQO/YslUVJU+D9bVBIDroLgK4hygEDweybYGY8AD6d8yftfASL+j+2O8
9NwiPJk51kYiNzcXO3fu7BeJPgkUFxdTbm6ure/JZdNRub0QrOMcGFcBrgLgAHEAGqD2ID7OZZsz
Ji0RcR6HAV4DkaqPJxXouoSywkz8buF9NhIrV65EVVXVbUnEJHDixAlat26drS9n+U+xo/hZMMNh
yZA+uAbAIKOGMCJlmG3eA/clAYaJEamGBlRdC6QBXZewo3AhVswfbSORk5ODDz/8sE8SvRLw+/1U
XV0dBb5MgOe6LUNIEqpOwgCUlOCESw4HuAfG3KX7B2kAGaTJ0Jro77yI8ucX4DfzRtlIHD58GH6/
PyaJXgk0NjZi79695vuUiffrkr91Vt8cHAwc4FZJGiZEKqDcwsi0ZHP+d749POy8MS7iGtBxARWF
8/CDsQnm3BdeeAHWj6N+EYhspSVbgI5zYYkZGmAwzIEbwA0tUM9NfO+7aeb89HuGRQBWLcSNZxj3
zvOYMzk5NpjBEHi99mXD5lXdjqGDJh42h7CEOXhnK3448X5z/shUb1jSpIEM8yFuzOUcjAsNcnR0
KXdG4MEHH8TEiRPN9z0H3sfRE41m+GNcs9i0LjkGTTcrUoGem5g7/V5z/ohElw4O+jxGYQ3qJqmB
jN/+cbINtcfCJvPYY4+hr3qtVwI+n49lZmaa71fbbuH3hX/FycbPQKSCoIIMQEQcehjVo5Gw8/F3
3cSvH03H2HuHY1icQzcvYWKGIMiITMwQwjtnruPpqk9w/ZZq7p2RkYEJEyaAYrCIWY36/X7KyMhA
W1ub2XfP3Qmo3ZKFaQ+P0gmYIdSoOkU4lWSwxGRAdqHL/xm86k27w0blAw3/PncD6yo/xrsfdZj7
paSkoLGxESNGjAgDZsxWK8X0AZ/Px44fP460tLAzXrnWgUX5r+PYuxf1mG/Gf80sofWy2YuWbg/+
uONtPLHhPRw+0WIHL8hbwG95rcUGPiEhAYcOHUJSUhI456YZRWqiz+8BIqLm5mZMmjQJN27cMPsT
E9zYtWE25k4bYzh3mABIBRwOHPgPx+LH/6yDiXPi6pHZcEpk+I5qmtGpCzdQecCPPUevmevLsowj
R45gypQpcDgc5mVowKaJmBoQTEePHo2GhgaMGxcuum52BLHsqUOoP3oWgLB9Sz7QgpgxJoThw/SS
IutnaXBKZEpcgD998Uvs+3ubDbzT6URNTQ0mTZoEVVWhaRo459A0TeCy4etVA+JH8RvnHK2trZgz
Zw5Onz5tjnPJDrxSOAPLZt1nSVIqwDmIVJw6dx1nLt3A3J/cjZThkhE29cvf3o2KA59ja91lcz3G
GDZv3oylS5fC7XbD6XTC6XRCkiRIktS7Jm5HQLDXNA3Xr19HVlYWPvjgA4vEGLav+RHmTxsJX4rb
8AsO4kbOIG6GzTD4HvzlmB/PVH9q2zc/Px/Z2dnwer1wu92QZRkulwuSJEGWZZMEY8wkEGVCVvCC
AOcciqIgPj4e+/btw9SpU83xqkpYteU9HHm7Bf62rnC2RjikisTFSMPV9m4ceucannvVDn7p0qVY
uHAhgsEggsEgQqFQlAlZnVm0PjOxIKFpGogIqqrC5XKhuroa06dPt4wDcl78F+qPNeNKe6cRoTjs
ZYOKK+3dON7wBQrKm2H5pMbMmTOxZMkSBAIB9PT0oKenB4FAAIqiQFEUU4hRRyq3I2AlIZ4553A6
nSgvL8fs2bNtY9eVNaDub58YmhDZWpe+v70bJ5tu4clt/4WihqU4efJkrFixAkRkAg4Gg1AUBaqq
muDF/pGtX7WQNXeIZ0mSUFRUhAULFtjGPl3ZiJdev4QrbZ16zUQqrrZ342JLN5ZvaEJ3ICz68ePH
IycnB5qmmZIW4EOhUJTZWEKo+Rx1KsEYY9Zk4XA4IEmSKXnOuRkVJElCYWEhvF4v6urqzDVK9n2M
zu4Qnl5+L9q/7MHltgBWbr9kA5+eno7c3FzIsoyI5GrTeCzgosU8VmGMmSGLiEwSsixH2WNBQQEA
2EhUH/wc7330BaaOH479/2zHtS/CFabP58OaNWvg9Xp1EE4nZFmGLMtwu93wer1wuVy2ECpJko2I
iTNWJrZGI+HEwi6FisUVCAQQCASwZ88eVFZW9lk9JicnY/369fD5fPB4PCbo+Ph4k4TH44HL5YLX
64XX6zXH9EKE9aUBRkTEGLNJP1Kd4nI4HMjOzkZ8fDyKi4ttJ3eiJSYmYu3atRg1apQJzuv1wuPx
wOPxwOFwmEBFHnA6nXC5XGYSiywlbns2SnoDANN0hMNZnS0UCpkaOnPmDN566y20tLSY2ktLS8OM
GTOQkpICt9uNuLg4M2EJ8CJZSZJkZmFBwkrAWpH2+3BXEBEkRIhTVdVMOCL5qKqKy5cvQ1EUaJoG
VVWRnKx/Jlpt3eVy2QCKu9C68AFr9o0spwd0Oh1ZYggCIlNaiQktiXdhZgKQABcp3ci7LMtM7B0J
fsAErJoAYItG1mdr+hfvVlACtHDIXkploYXbHvQOikDEe6/ZmjFmAhf9ZvJxOk2Qol88W4kYz32S
GNQfHLG+TyMTkAVEr88xQfVnkBg7GAKixSIykDYQsL21/wFkW/B5QqT9lwAAAABJRU5ErkJggg==">
</td>
<td class="content">It is problematic to have multiple storage configurations pointing to
the exact same underlying storage. Such an <em>aliased</em> storage configuration can
lead to two different volume IDs (<em>volid</em>) pointing to the exact same disk
image. Proxmox VE expects that the images' volume IDs point to, are unique. Choosing
different content types for <em>aliased</em> storage configurations can be fine, but
is not recommended.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="_common_storage_properties">7.2.2. Common Storage Properties
 <a class="headerlink" href="#_common_storage_properties" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>A few storage properties are common among different storage types.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
nodes
</dt>
<dd>
<p>
List of cluster node names where this storage is
usable/accessible. One can use this property to restrict storage
access to a limited set of nodes.
</p>
</dd>
<dt class="hdlist1">
content
</dt>
<dd>
<p>
A storage can support several content types, for example virtual disk
images, cdrom iso images, container templates or container root
directories. Not all storage types support all content types. One can set
this property to select what this storage is used for.
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
images
</dt>
<dd>
<p>
QEMU/KVM VM images.
</p>
</dd>
<dt class="hdlist1">
rootdir
</dt>
<dd>
<p>
Allow to store container data.
</p>
</dd>
<dt class="hdlist1">
vztmpl
</dt>
<dd>
<p>
Container templates.
</p>
</dd>
<dt class="hdlist1">
backup
</dt>
<dd>
<p>
Backup files (<span class="monospaced">vzdump</span>).
</p>
</dd>
<dt class="hdlist1">
iso
</dt>
<dd>
<p>
ISO images
</p>
</dd>
<dt class="hdlist1">
snippets
</dt>
<dd>
<p>
Snippet files, for example guest hook scripts
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
shared
</dt>
<dd>
<p>
Indicate that this is a single storage with the same contents on all nodes (or
all listed in the <em>nodes</em> option). It will not make the contents of a local
storage automatically accessible to other nodes, it just marks an already shared
storage as such!
</p>
</dd>
<dt class="hdlist1">
disable
</dt>
<dd>
<p>
You can use this flag to disable the storage completely.
</p>
</dd>
<dt class="hdlist1">
maxfiles
</dt>
<dd>
<p>
Deprecated, please use <span class="monospaced">prune-backups</span> instead. Maximum number of backup files
per VM. Use <span class="monospaced">0</span> for unlimited.
</p>
</dd>
<dt class="hdlist1">
prune-backups
</dt>
<dd>
<p>
Retention options for backups. For details, see
<a href="#vzdump_retention">Backup Retention</a>.
</p>
</dd>
<dt class="hdlist1">
format
</dt>
<dd>
<p>
Default image format (<span class="monospaced">raw|qcow2|vmdk</span>)
</p>
</dd>
<dt class="hdlist1">
preallocation
</dt>
<dd>
<p>
Preallocation mode (<span class="monospaced">off|metadata|falloc|full</span>) for <span class="monospaced">raw</span> and <span class="monospaced">qcow2</span> images on
file-based storages. The default is <span class="monospaced">metadata</span>, which is treated like <span class="monospaced">off</span> for
<span class="monospaced">raw</span> images. When using network storages in combination with large <span class="monospaced">qcow2</span>
images, using <span class="monospaced">off</span> can help to avoid timeouts.
</p>
</dd>
</dl></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">It is not advisable to use the same storage pool on different
Proxmox VE clusters. Some storage operation need exclusive access to the
storage, so proper locking is required. While this is implemented
within a cluster, it does not work between different clusters.</td>
</tr></tbody></table>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_volumes">
<span>7.3. Volumes</span>
 <a class="headerlink" href="#_volumes" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>We use a special notation to address storage data. When you allocate
data from a storage pool, it returns such a volume identifier. A volume
is identified by the <span class="monospaced">&lt;STORAGE_ID&gt;</span>, followed by a storage type
dependent volume name, separated by colon. A valid <span class="monospaced">&lt;VOLUME_ID&gt;</span> looks
like:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>local:230/example-image.raw</pre>
</div></div>
<div class="literalblock">
<div class="content monospaced">
<pre>local:iso/debian-501-amd64-netinst.iso</pre>
</div></div>
<div class="literalblock">
<div class="content monospaced">
<pre>local:vztmpl/debian-5.0-joomla_1.5.9-1_i386.tar.gz</pre>
</div></div>
<div class="literalblock">
<div class="content monospaced">
<pre>iscsi-storage:0.0.2.scsi-14f504e46494c4500494b5042546d2d646744372d31616d61</pre>
</div></div>
<div class="paragraph">
<p>To get the file system path for a <span class="monospaced">&lt;VOLUME_ID&gt;</span> use:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>pvesm path &lt;VOLUME_ID&gt;</pre>
</div></div>
<div class="sect3">
<h4 id="_volume_ownership">7.3.1. Volume Ownership
 <a class="headerlink" href="#_volume_ownership" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>There exists an ownership relation for <span class="monospaced">image</span> type volumes. Each such
volume is owned by a VM or Container. For example volume
<span class="monospaced">local:230/example-image.raw</span> is owned by VM 230. Most storage
backends encodes this ownership information into the volume name.</p></div>
<div class="paragraph">
<p>When you remove a VM or Container, the system also removes all
associated volumes which are owned by that VM or Container.</p></div>
</div>
</div>
<div class="sect2">
<h3 id="_using_the_command_line_interface">
<span>7.4. Using the Command-line Interface</span>
 <a class="headerlink" href="#_using_the_command_line_interface" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>It is recommended to familiarize yourself with the concept behind storage
pools and volume identifiers, but in real life, you are not forced to do any
of those low level operations on the command line. Normally,
allocation and removal of volumes is done by the VM and Container
management tools.</p></div>
<div class="paragraph">
<p>Nevertheless, there is a command-line tool called <span class="monospaced">pvesm</span> (“Proxmox VE
Storage Manager”), which is able to perform common storage management
tasks.</p></div>
<div class="sect3">
<h4 id="_examples">7.4.1. Examples
 <a class="headerlink" href="#_examples" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Add storage pools</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>pvesm add &lt;TYPE&gt; &lt;STORAGE_ID&gt; &lt;OPTIONS&gt;
pvesm add dir &lt;STORAGE_ID&gt; --path &lt;PATH&gt;
pvesm add nfs &lt;STORAGE_ID&gt; --path &lt;PATH&gt; --server &lt;SERVER&gt; --export &lt;EXPORT&gt;
pvesm add lvm &lt;STORAGE_ID&gt; --vgname &lt;VGNAME&gt;
pvesm add iscsi &lt;STORAGE_ID&gt; --portal &lt;HOST[:PORT]&gt; --target &lt;TARGET&gt;</pre>
</div></div>
<div class="paragraph">
<p>Disable storage pools</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>pvesm set &lt;STORAGE_ID&gt; --disable 1</pre>
</div></div>
<div class="paragraph">
<p>Enable storage pools</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>pvesm set &lt;STORAGE_ID&gt; --disable 0</pre>
</div></div>
<div class="paragraph">
<p>Change/set storage options</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>pvesm set &lt;STORAGE_ID&gt; &lt;OPTIONS&gt;
pvesm set &lt;STORAGE_ID&gt; --shared 1
pvesm set local --format qcow2
pvesm set &lt;STORAGE_ID&gt; --content iso</pre>
</div></div>
<div class="paragraph">
<p>Remove storage pools. This does not delete any data, and does not
disconnect or unmount anything. It just removes the storage
configuration.</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>pvesm remove &lt;STORAGE_ID&gt;</pre>
</div></div>
<div class="paragraph">
<p>Allocate volumes</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>pvesm alloc &lt;STORAGE_ID&gt; &lt;VMID&gt; &lt;name&gt; &lt;size&gt; [--format &lt;raw|qcow2&gt;]</pre>
</div></div>
<div class="paragraph">
<p>Allocate a 4G volume in local storage. The name is auto-generated if
you pass an empty string as <span class="monospaced">&lt;name&gt;</span></p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>pvesm alloc local &lt;VMID&gt; '' 4G</pre>
</div></div>
<div class="paragraph">
<p>Free volumes</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>pvesm free &lt;VOLUME_ID&gt;</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">This really destroys all volume data.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>List storage status</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>pvesm status</pre>
</div></div>
<div class="paragraph">
<p>List storage contents</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>pvesm list &lt;STORAGE_ID&gt; [--vmid &lt;VMID&gt;]</pre>
</div></div>
<div class="paragraph">
<p>List volumes allocated by VMID</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>pvesm list &lt;STORAGE_ID&gt; --vmid &lt;VMID&gt;</pre>
</div></div>
<div class="paragraph">
<p>List iso images</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>pvesm list &lt;STORAGE_ID&gt; --content iso</pre>
</div></div>
<div class="paragraph">
<p>List container templates</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>pvesm list &lt;STORAGE_ID&gt; --content vztmpl</pre>
</div></div>
<div class="paragraph">
<p>Show file system path for a volume</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>pvesm path &lt;VOLUME_ID&gt;</pre>
</div></div>
<div class="paragraph">
<p>Exporting the volume <span class="monospaced">local:103/vm-103-disk-0.qcow2</span> to the file <span class="monospaced">target</span>.
This is mostly used internally with <span class="monospaced">pvesm import</span>.
The stream format qcow2+size is different to the qcow2 format.
Consequently, the exported file cannot simply be attached to a VM.
This also holds for the other formats.</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>pvesm export local:103/vm-103-disk-0.qcow2 qcow2+size target --with-snapshots 1</pre>
</div></div>
</div>
</div>
<div class="sect2">
<h3 id="storage_directory">
<span>7.5. Directory Backend</span>
 <a class="headerlink" href="#storage_directory" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Storage pool type: <span class="monospaced">dir</span></p></div>
<div class="paragraph">
<p>Proxmox VE can use local directories or locally mounted shares for
storage. A directory is a file level storage, so you can store any
content type like virtual disk images, containers, templates, ISO images
or backup files.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">You can mount additional storages via standard linux <span class="monospaced">/etc/fstab</span>,
and then define a directory storage for that mount point. This way you
can use any file system supported by Linux.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>This backend assumes that the underlying directory is POSIX
compatible, but nothing else. This implies that you cannot create
snapshots at the storage level. But there exists a workaround for VM
images using the <span class="monospaced">qcow2</span> file format, because that format supports
snapshots internally.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">Some storage types do not support <span class="monospaced">O_DIRECT</span>, so you can’t use
cache mode <span class="monospaced">none</span> with such storages. Simply use cache mode
<span class="monospaced">writeback</span> instead.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>We use a predefined directory layout to store different content types
into different sub-directories. This layout is used by all file level
storage backends.</p></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<caption class="title">Table 3. Directory layout</caption>
<colgroup><col style="width:50%;">
<col style="width:50%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Content type        </th>
<th class="tableblock halign-left valign-top">Subdir</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">VM images</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">images/&lt;VMID&gt;/</span></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">ISO images</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">template/iso/</span></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">Container templates</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">template/cache/</span></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">Backup files</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">dump/</span></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">Snippets</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced"><span class="monospaced">snippets/</span></p></td>
</tr>
</tbody>
</table>
<div class="sect3">
<h4 id="_configuration_2">7.5.1. Configuration
 <a class="headerlink" href="#_configuration_2" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>This backend supports all common storage properties, and adds two
additional properties. The <span class="monospaced">path</span> property is used to specify the
directory. This needs to be an absolute file system path.</p></div>
<div class="paragraph">
<p>The optional <span class="monospaced">content-dirs</span> property allows for the default layout
to be changed. It consists of a comma-separated list of identifiers
in the following format:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>vtype=path</pre>
</div></div>
<div class="paragraph">
<p>Where <span class="monospaced">vtype</span> is one of the allowed content types for the storage, and
<span class="monospaced">path</span> is a path relative to the mountpoint of the storage.</p></div>
<div class="listingblock">
<div class="title">Configuration Example (<span class="monospaced">/etc/pve/storage.cfg</span>)</div>
<div class="content monospaced">
<pre>dir: backup
        path /mnt/backup
        content backup
        prune-backups keep-last=7
        max-protected-backups 3
        content-dirs backup=custom/backup/dir</pre>
</div></div>
<div class="paragraph">
<p>The above configuration defines a storage pool called <span class="monospaced">backup</span>. That pool can be
used to store up to 7 regular backups (<span class="monospaced">keep-last=7</span>) and 3 protected backups
per VM. The real path for the backup files is <span class="monospaced">/mnt/backup/custom/backup/dir/...</span>.</p></div>
</div>
<div class="sect3">
<h4 id="_file_naming_conventions">7.5.2. File naming conventions
 <a class="headerlink" href="#_file_naming_conventions" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>This backend uses a well defined naming scheme for VM images:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>vm-&lt;VMID&gt;-&lt;NAME&gt;.&lt;FORMAT&gt;</pre>
</div></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;VMID&gt;</span>
</dt>
<dd>
<p>
This specifies the owner VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;NAME&gt;</span>
</dt>
<dd>
<p>
This can be an arbitrary name (<span class="monospaced">ascii</span>) without white space. The
backend uses <span class="monospaced">disk-[N]</span> as default, where <span class="monospaced">[N]</span> is replaced by an
integer to make the name unique.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;FORMAT&gt;</span>
</dt>
<dd>
<p>
Specifies the image format (<span class="monospaced">raw|qcow2|vmdk</span>).
</p>
</dd>
</dl></div>
<div class="paragraph">
<p>When you create a VM template, all VM images are renamed to indicate
that they are now read-only, and can be used as a base image for clones:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>base-&lt;VMID&gt;-&lt;NAME&gt;.&lt;FORMAT&gt;</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Such base images are used to generate cloned images. So it is
important that those files are read-only, and never get modified. The
backend changes the access mode to <span class="monospaced">0444</span>, and sets the immutable flag
(<span class="monospaced">chattr +i</span>) if the storage supports that.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="_storage_features">7.5.3. Storage Features
 <a class="headerlink" href="#_storage_features" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>As mentioned above, most file systems do not support snapshots out
of the box. To workaround that problem, this backend is able to use
<span class="monospaced">qcow2</span> internal snapshot capabilities.</p></div>
<div class="paragraph">
<p>Same applies to clones. The backend uses the <span class="monospaced">qcow2</span> base image
feature to create clones.</p></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<caption class="title">Table 4. Storage features for backend <span class="monospaced">dir</span></caption>
<colgroup><col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Content types                              </th>
<th class="tableblock halign-left valign-top">Image formats         </th>
<th class="tableblock halign-left valign-top">Shared </th>
<th class="tableblock halign-left valign-top">Snapshots </th>
<th class="tableblock halign-left valign-top">Clones</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">images rootdir vztmpl iso backup snippets</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">raw qcow2 vmdk subvol</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">no</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">qcow2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">qcow2</p></td>
</tr>
</tbody>
</table>
</div>
<div class="sect3">
<h4 id="_examples_2">7.5.4. Examples
 <a class="headerlink" href="#_examples_2" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Please use the following command to allocate a 4GB image on storage <span class="monospaced">local</span>:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># pvesm alloc local 100 vm-100-disk10.raw 4G
Formatting '/var/lib/vz/images/100/vm-100-disk10.raw', fmt=raw size=4294967296
successfully created 'local:100/vm-100-disk10.raw'</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">The image name must conform to above naming conventions.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>The real file system path is shown with:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># pvesm path local:100/vm-100-disk10.raw
/var/lib/vz/images/100/vm-100-disk10.raw</pre>
</div></div>
<div class="paragraph">
<p>And you can remove the image with:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># pvesm free local:100/vm-100-disk10.raw</pre>
</div></div>
</div>
</div>
<div class="sect2">
<h3 id="storage_nfs">
<span>7.6. NFS Backend</span>
 <a class="headerlink" href="#storage_nfs" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Storage pool type: <span class="monospaced">nfs</span></p></div>
<div class="paragraph">
<p>The NFS backend is based on the directory backend, so it shares most
properties. The directory layout and the file naming conventions are
the same. The main advantage is that you can directly configure the
NFS server properties, so the backend can mount the share
automatically. There is no need to modify <span class="monospaced">/etc/fstab</span>. The backend
can also test if the server is online, and provides a method to query
the server for exported shares.</p></div>
<div class="sect3">
<h4 id="_configuration_3">7.6.1. Configuration
 <a class="headerlink" href="#_configuration_3" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The backend supports all common storage properties, except the shared
flag, which is always set. Additionally, the following properties are
used to configure the NFS server:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
server
</dt>
<dd>
<p>
Server IP or DNS name. To avoid DNS lookup delays, it is usually
preferable to use an IP address instead of a DNS name - unless you
have a very reliable DNS server, or list the server in the local
<span class="monospaced">/etc/hosts</span> file.
</p>
</dd>
<dt class="hdlist1">
export
</dt>
<dd>
<p>
NFS export path (as listed by <span class="monospaced">pvesm nfsscan</span>).
</p>
</dd>
</dl></div>
<div class="paragraph">
<p>You can also set NFS mount options:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
path
</dt>
<dd>
<p>
The local mount point (defaults to <span class="monospaced">/mnt/pve/&lt;STORAGE_ID&gt;/</span>).
</p>
</dd>
<dt class="hdlist1">
content-dirs
</dt>
<dd>
<p>
Overrides for the default directory layout. Optional.
</p>
</dd>
<dt class="hdlist1">
options
</dt>
<dd>
<p>
NFS mount options (see <span class="monospaced">man nfs</span>).
</p>
</dd>
</dl></div>
<div class="listingblock">
<div class="title">Configuration Example (<span class="monospaced">/etc/pve/storage.cfg</span>)</div>
<div class="content monospaced">
<pre>nfs: iso-templates
        path /mnt/pve/iso-templates
        server 10.0.0.10
        export /space/iso-templates
        options vers=3,soft
        content iso,vztmpl</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">After an NFS request times out, NFS request are retried
indefinitely by default. This can lead to unexpected hangs on the
client side. For read-only content, it is worth to consider the NFS
<span class="monospaced">soft</span> option, which limits the number of retries to three.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="_storage_features_2">7.6.2. Storage Features
 <a class="headerlink" href="#_storage_features_2" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>NFS does not support snapshots, but the backend uses <span class="monospaced">qcow2</span> features
to implement snapshots and cloning.</p></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<caption class="title">Table 5. Storage features for backend <span class="monospaced">nfs</span></caption>
<colgroup><col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Content types                              </th>
<th class="tableblock halign-left valign-top">Image formats  </th>
<th class="tableblock halign-left valign-top">Shared </th>
<th class="tableblock halign-left valign-top">Snapshots </th>
<th class="tableblock halign-left valign-top">Clones</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">images rootdir vztmpl iso backup snippets</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">raw qcow2 vmdk</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">qcow2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">qcow2</p></td>
</tr>
</tbody>
</table>
</div>
<div class="sect3">
<h4 id="_examples_3">7.6.3. Examples
 <a class="headerlink" href="#_examples_3" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>You can get a list of exported NFS shares with:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># pvesm nfsscan &lt;server&gt;</pre>
</div></div>
</div>
</div>
<div class="sect2">
<h3 id="storage_cifs">
<span>7.7. CIFS Backend</span>
 <a class="headerlink" href="#storage_cifs" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Storage pool type: <span class="monospaced">cifs</span></p></div>
<div class="paragraph">
<p>The CIFS backend extends the directory backend, so that no manual
setup of a CIFS mount is needed. Such a storage can be added directly
through the Proxmox VE API or the web UI, with all our backend advantages,
like server heartbeat check or comfortable selection of exported
shares.</p></div>
<div class="sect3">
<h4 id="_configuration_4">7.7.1. Configuration
 <a class="headerlink" href="#_configuration_4" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The backend supports all common storage properties, except the shared
flag, which is always set. Additionally, the following CIFS special
properties are available:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
server
</dt>
<dd>
<p>
Server IP or DNS name. Required.
</p>
</dd>
</dl></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">To avoid DNS lookup delays, it is usually preferable to use an IP
address instead of a DNS name - unless you have a very reliable DNS
server, or list the server in the local <span class="monospaced">/etc/hosts</span> file.</td>
</tr></tbody></table>
</div>
<div class="dlist"><dl>
<dt class="hdlist1">
share
</dt>
<dd>
<p>
CIFS share to use (get available ones with <span class="monospaced">pvesm scan cifs &lt;address&gt;</span> or the
web UI). Required.
</p>
</dd>
<dt class="hdlist1">
username
</dt>
<dd>
<p>
The username for the CIFS storage. Optional, defaults to ‘guest’.
</p>
</dd>
<dt class="hdlist1">
password
</dt>
<dd>
<p>
The user password. Optional.
It will be saved in a file only readable by root
(<span class="monospaced">/etc/pve/priv/storage/&lt;STORAGE-ID&gt;.pw</span>).
</p>
</dd>
<dt class="hdlist1">
domain
</dt>
<dd>
<p>
Sets the user domain (workgroup) for this storage. Optional.
</p>
</dd>
<dt class="hdlist1">
smbversion
</dt>
<dd>
<p>
SMB protocol Version. Optional, default is <span class="monospaced">3</span>.
SMB1 is not supported due to security issues.
</p>
</dd>
<dt class="hdlist1">
path
</dt>
<dd>
<p>
The local mount point. Optional, defaults to <span class="monospaced">/mnt/pve/&lt;STORAGE_ID&gt;/</span>.
</p>
</dd>
<dt class="hdlist1">
content-dirs
</dt>
<dd>
<p>
Overrides for the default directory layout. Optional.
</p>
</dd>
<dt class="hdlist1">
options
</dt>
<dd>
<p>
Additional CIFS mount options (see <span class="monospaced">man mount.cifs</span>). Some options are set
automatically and shouldn’t be set here. Proxmox VE will always set the option
<span class="monospaced">soft</span>. Depending on the configuration, these options are set automatically:
<span class="monospaced">username</span>, <span class="monospaced">credentials</span>, <span class="monospaced">guest</span>, <span class="monospaced">domain</span>, <span class="monospaced">vers</span>.
</p>
</dd>
<dt class="hdlist1">
subdir
</dt>
<dd>
<p>
The subdirectory of the share to mount. Optional, defaults to the root directory
of the share.
</p>
</dd>
</dl></div>
<div class="listingblock">
<div class="title">Configuration Example (<span class="monospaced">/etc/pve/storage.cfg</span>)</div>
<div class="content monospaced">
<pre>cifs: backup
        path /mnt/pve/backup
        server 10.0.0.11
        share VMData
        content backup
        options noserverino,echo_interval=30
        username anna
        smbversion 3
        subdir /data</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_storage_features_3">7.7.2. Storage Features
 <a class="headerlink" href="#_storage_features_3" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>CIFS does not support snapshots on a storage level. But you may use
<span class="monospaced">qcow2</span> backing files if you still want to have snapshots and cloning
features available.</p></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<caption class="title">Table 6. Storage features for backend <span class="monospaced">cifs</span></caption>
<colgroup><col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Content types                             </th>
<th class="tableblock halign-left valign-top">Image formats   </th>
<th class="tableblock halign-left valign-top">Shared </th>
<th class="tableblock halign-left valign-top">Snapshots </th>
<th class="tableblock halign-left valign-top">Clones</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">images rootdir vztmpl iso backup snippets</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">raw qcow2 vmdk</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">qcow2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">qcow2</p></td>
</tr>
</tbody>
</table>
</div>
<div class="sect3">
<h4 id="_examples_4">7.7.3. Examples
 <a class="headerlink" href="#_examples_4" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>You can get a list of exported CIFS shares with:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># pvesm scan cifs &lt;server&gt; [--username &lt;username&gt;] [--password]</pre>
</div></div>
<div class="paragraph">
<p>Then you could add this share as a storage to the whole Proxmox VE cluster
with:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># pvesm add cifs &lt;storagename&gt; --server &lt;server&gt; --share &lt;share&gt; [--username &lt;username&gt;] [--password]</pre>
</div></div>
</div>
</div>
<div class="sect2">
<h3 id="storage_pbs">
<span>7.8. Proxmox Backup Server</span>
 <a class="headerlink" href="#storage_pbs" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Storage pool type: <span class="monospaced">pbs</span></p></div>
<div class="paragraph">
<p>This backend allows direct integration of a Proxmox Backup Server into Proxmox VE
like any other storage.
A Proxmox Backup storage can be added directly through the Proxmox VE API, CLI or
the web interface.</p></div>
<div class="sect3">
<h4 id="_configuration_5">7.8.1. Configuration
 <a class="headerlink" href="#_configuration_5" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The backend supports all common storage properties, except the shared flag,
which is always set. Additionally, the following special properties to Proxmox
Backup Server are available:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
server
</dt>
<dd>
<p>
Server IP or DNS name. Required.
</p>
</dd>
<dt class="hdlist1">
port
</dt>
<dd>
<p>
Use this port instead of the default one, i.e. <span class="monospaced">8007</span>. Optional.
</p>
</dd>
<dt class="hdlist1">
username
</dt>
<dd>
<p>
The username for the Proxmox Backup Server storage. Required.
</p>
</dd>
</dl></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">Do not forget to add the realm to the username. For example, <span class="monospaced">root@pam</span> or
<span class="monospaced">archiver@pbs</span>.</td>
</tr></tbody></table>
</div>
<div class="dlist"><dl>
<dt class="hdlist1">
password
</dt>
<dd>
<p>
The user password. The value will be saved in a file under
<span class="monospaced">/etc/pve/priv/storage/&lt;STORAGE-ID&gt;.pw</span> with access restricted to the root
user. Required.
</p>
</dd>
<dt class="hdlist1">
datastore
</dt>
<dd>
<p>
The ID of the Proxmox Backup Server datastore to use. Required.
</p>
</dd>
<dt class="hdlist1">
fingerprint
</dt>
<dd>
<p>
The fingerprint of the Proxmox Backup Server API TLS certificate. You can get
it in the Servers Dashboard or using the <span class="monospaced">proxmox-backup-manager cert info</span>
command. Required for self-signed certificates or any other one where the host
does not trusts the servers CA.
</p>
</dd>
<dt class="hdlist1">
encryption-key
</dt>
<dd>
<p>
A key to encrypt the backup data from the client side. Currently only
non-password protected (no key derive function (kdf)) are supported. Will be
saved in a file under <span class="monospaced">/etc/pve/priv/storage/&lt;STORAGE-ID&gt;.enc</span> with access
restricted to the root user.  Use the magic value <span class="monospaced">autogen</span> to automatically
generate a new one using <span class="monospaced">proxmox-backup-client key create --kdf none &lt;path&gt;</span>.
Optional.
</p>
</dd>
<dt class="hdlist1">
master-pubkey
</dt>
<dd>
<p>
A public RSA key used to encrypt the backup encryption key as part of the
backup task. Will be saved in a file under
<span class="monospaced">/etc/pve/priv/storage/&lt;STORAGE-ID&gt;.master.pem</span> with access restricted to the
root user.
The encrypted copy of the backup encryption key will be appended to each backup
and stored on the Proxmox Backup Server instance for recovery purposes.
Optional, requires <span class="monospaced">encryption-key</span>.
</p>
</dd>
</dl></div>
<div class="listingblock">
<div class="title">Configuration Example (<span class="monospaced">/etc/pve/storage.cfg</span>)</div>
<div class="content monospaced">
<pre>pbs: backup
        datastore main
        server enya.proxmox.com
        content backup
        fingerprint 09:54:ef:..snip..:88:af:47:fe:4c:3b:cf:8b:26:88:0b:4e:3c:b2
        prune-backups keep-all=1
        username archiver@pbs
        encryption-key a9:ee:c8:02:13:..snip..:2d:53:2c:98
        master-pubkey 1</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_storage_features_4">7.8.2. Storage Features
 <a class="headerlink" href="#_storage_features_4" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Proxmox Backup Server only supports backups, they can be block-level or
file-level based. Proxmox VE uses block-level for virtual machines and file-level for
container.</p></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<caption class="title">Table 7. Storage features for backend <span class="monospaced">pbs</span></caption>
<colgroup><col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Content types </th>
<th class="tableblock halign-left valign-top">Image formats </th>
<th class="tableblock halign-left valign-top">Shared </th>
<th class="tableblock halign-left valign-top">Snapshots </th>
<th class="tableblock halign-left valign-top">Clones</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">backup</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">n/a</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">n/a</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">n/a</p></td>
</tr>
</tbody>
</table>
</div>
<div class="sect3">
<h4 id="storage_pbs_encryption">7.8.3. Encryption
 <a class="headerlink" href="#storage_pbs_encryption" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<a class="image" href="images/screenshot/storage-pbs-encryption-with-key.png">
<img src="images/screenshot/storage-pbs-encryption-with-key.png" alt="screenshot/storage-pbs-encryption-with-key.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>Optionally, you can configure client-side encryption with AES-256 in GCM mode.
Encryption can be configured either via the web interface, or on the CLI with
the <span class="monospaced">encryption-key</span> option (see above). The key will be saved in the file
<span class="monospaced">/etc/pve/priv/storage/&lt;STORAGE-ID&gt;.enc</span>, which is only accessible by the root
user.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">Without their key, backups will be inaccessible. Thus, you should
keep keys ordered and in a place that is separate from the contents being
backed up. It can happen, for example, that you back up an entire system, using
a key on that system. If the system then becomes inaccessible for any reason
and needs to be restored, this will not be possible as the encryption key will be
lost along with the broken system.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>It is recommended that you keep your key safe, but easily accessible, in
order for quick disaster recovery. For this reason, the best place to store it
is in your password manager, where it is immediately recoverable. As a backup to
this, you should also save the key to a USB flash drive and store that in a secure
place. This way, it is detached from any system, but is still easy to recover
from, in case of emergency. Finally, in preparation for the worst case scenario,
you should also consider keeping a paper copy of your key locked away in a safe
place. The <span class="monospaced">paperkey</span> subcommand can be used to create a QR encoded version of
your key. The following command sends the output of the <span class="monospaced">paperkey</span> command to
a text file, for easy printing.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># proxmox-backup-client key paperkey /etc/pve/priv/storage/&lt;STORAGE-ID&gt;.enc --output-format text &gt; qrkey.txt</pre>
</div></div>
<div class="paragraph">
<p>Additionally, it is possible to use a single RSA master key pair for key
recovery purposes: configure all clients doing encrypted backups to use a
single public master key, and all subsequent encrypted backups will contain a
RSA-encrypted copy of the used AES encryption key. The corresponding private
master key allows recovering the AES key and decrypting the backup even if the
client system is no longer available.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">The same safe-keeping rules apply to the master key pair as to the
regular encryption keys. Without a copy of the private key recovery is not
possible! The <span class="monospaced">paperkey</span> command supports generating paper copies of private
master keys for storage in a safe, physical location.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>Because the encryption is managed on the client side, you can use the same
datastore on the server for unencrypted backups and encrypted backups, even
if they are encrypted with different keys. However, deduplication between
backups with different keys is not possible, so it is often better to create
separate datastores.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Do not use encryption if there is no benefit from it, for example, when
you are running the server locally in a trusted network. It is always easier to
recover from unencrypted backups.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="_example_add_storage_over_cli">7.8.4. Example: Add Storage over CLI
 <a class="headerlink" href="#_example_add_storage_over_cli" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Then you could add this share as a storage to the whole Proxmox VE cluster
with:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># pvesm add pbs &lt;id&gt; --server &lt;server&gt; --datastore &lt;datastore&gt; --username &lt;username&gt; --fingerprint 00:B4:... --password</pre>
</div></div>
</div>
</div>
<div class="sect2">
<h3 id="storage_glusterfs">
<span>7.9. GlusterFS Backend</span>
 <a class="headerlink" href="#storage_glusterfs" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Storage pool type: <span class="monospaced">glusterfs</span></p></div>
<div class="paragraph">
<p>GlusterFS is a scalable network file system. The system uses a modular
design, runs on commodity hardware, and can provide a highly available
enterprise storage at low costs. Such system is capable of scaling to
several petabytes, and can handle thousands of clients.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">After a node/brick crash, GlusterFS does a full <span class="monospaced">rsync</span> to make
sure data is consistent. This can take a very long time with large
files, so this backend is not suitable to store large VM images.</td>
</tr></tbody></table>
</div>
<div class="sect3">
<h4 id="_configuration_6">7.9.1. Configuration
 <a class="headerlink" href="#_configuration_6" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The backend supports all common storage properties, and adds the
following GlusterFS specific options:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">server</span>
</dt>
<dd>
<p>
GlusterFS volfile server IP or DNS name.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">server2</span>
</dt>
<dd>
<p>
Backup volfile server IP or DNS name.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">volume</span>
</dt>
<dd>
<p>
GlusterFS Volume.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">transport</span>
</dt>
<dd>
<p>
GlusterFS transport: <span class="monospaced">tcp</span>, <span class="monospaced">unix</span> or <span class="monospaced">rdma</span>
</p>
</dd>
</dl></div>
<div class="listingblock">
<div class="title">Configuration Example (<span class="monospaced">/etc/pve/storage.cfg</span>)</div>
<div class="content monospaced">
<pre>glusterfs: Gluster
        server 10.2.3.4
        server2 10.2.3.5
        volume glustervol
        content images,iso</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_file_naming_conventions_2">7.9.2. File naming conventions
 <a class="headerlink" href="#_file_naming_conventions_2" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The directory layout and the file naming conventions are inherited
from the <span class="monospaced">dir</span> backend.</p></div>
</div>
<div class="sect3">
<h4 id="_storage_features_5">7.9.3. Storage Features
 <a class="headerlink" href="#_storage_features_5" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The storage provides a file level interface, but no native
snapshot/clone implementation.</p></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<caption class="title">Table 8. Storage features for backend <span class="monospaced">glusterfs</span></caption>
<colgroup><col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Content types                      </th>
<th class="tableblock halign-left valign-top">Image formats   </th>
<th class="tableblock halign-left valign-top">Shared </th>
<th class="tableblock halign-left valign-top">Snapshots </th>
<th class="tableblock halign-left valign-top">Clones</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">images vztmpl iso backup snippets</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">raw qcow2 vmdk</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">qcow2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">qcow2</p></td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="sect2">
<h3 id="storage_zfspool">
<span>7.10. Local ZFS Pool Backend</span>
 <a class="headerlink" href="#storage_zfspool" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Storage pool type: <span class="monospaced">zfspool</span></p></div>
<div class="paragraph">
<p>This backend allows you to access local ZFS pools (or ZFS file systems
inside such pools).</p></div>
<div class="sect3">
<h4 id="_configuration_7">7.10.1. Configuration
 <a class="headerlink" href="#_configuration_7" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The backend supports the common storage properties <span class="monospaced">content</span>, <span class="monospaced">nodes</span>,
<span class="monospaced">disable</span>, and the following ZFS specific properties:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
pool
</dt>
<dd>
<p>
Select the ZFS pool/filesystem. All allocations are done within that
pool.
</p>
</dd>
<dt class="hdlist1">
blocksize
</dt>
<dd>
<p>
Set ZFS blocksize parameter.
</p>
</dd>
<dt class="hdlist1">
sparse
</dt>
<dd>
<p>
Use ZFS thin-provisioning. A sparse volume is a volume whose
reservation is not equal to the volume size.
</p>
</dd>
<dt class="hdlist1">
mountpoint
</dt>
<dd>
<p>
The mount point of the ZFS pool/filesystem. Changing this does not
affect the <span class="monospaced">mountpoint</span> property of the dataset seen by <span class="monospaced">zfs</span>.
Defaults to <span class="monospaced">/&lt;pool&gt;</span>.
</p>
</dd>
</dl></div>
<div class="listingblock">
<div class="title">Configuration Example (<span class="monospaced">/etc/pve/storage.cfg</span>)</div>
<div class="content monospaced">
<pre>zfspool: vmdata
        pool tank/vmdata
        content rootdir,images
        sparse</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_file_naming_conventions_3">7.10.2. File naming conventions
 <a class="headerlink" href="#_file_naming_conventions_3" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The backend uses the following naming scheme for VM images:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>vm-&lt;VMID&gt;-&lt;NAME&gt;      // normal VM images
base-&lt;VMID&gt;-&lt;NAME&gt;    // template VM image (read-only)
subvol-&lt;VMID&gt;-&lt;NAME&gt;  // subvolumes (ZFS filesystem for containers)</pre>
</div></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;VMID&gt;</span>
</dt>
<dd>
<p>
This specifies the owner VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;NAME&gt;</span>
</dt>
<dd>
<p>
This can be an arbitrary name (<span class="monospaced">ascii</span>) without white space. The
backend uses <span class="monospaced">disk[N]</span> as default, where <span class="monospaced">[N]</span> is replaced by an
integer to make the name unique.
</p>
</dd>
</dl></div>
</div>
<div class="sect3">
<h4 id="_storage_features_6">7.10.3. Storage Features
 <a class="headerlink" href="#_storage_features_6" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>ZFS is probably the most advanced storage type regarding snapshot and
cloning. The backend uses ZFS datasets for both VM images (format
<span class="monospaced">raw</span>) and container data (format <span class="monospaced">subvol</span>). ZFS properties are
inherited from the parent dataset, so you can simply set defaults
on the parent dataset.</p></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<caption class="title">Table 9. Storage features for backend <span class="monospaced">zfs</span></caption>
<colgroup><col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Content types  </th>
<th class="tableblock halign-left valign-top">Image formats  </th>
<th class="tableblock halign-left valign-top">Shared </th>
<th class="tableblock halign-left valign-top">Snapshots </th>
<th class="tableblock halign-left valign-top">Clones</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">images rootdir</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">raw subvol</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">no</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
</tr>
</tbody>
</table>
</div>
<div class="sect3">
<h4 id="_examples_5">7.10.4. Examples
 <a class="headerlink" href="#_examples_5" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>It is recommended to create an extra ZFS file system to store your VM images:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># zfs create tank/vmdata</pre>
</div></div>
<div class="paragraph">
<p>To enable compression on that newly allocated file system:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># zfs set compression=on tank/vmdata</pre>
</div></div>
<div class="paragraph">
<p>You can get a list of available ZFS filesystems with:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># pvesm zfsscan</pre>
</div></div>
</div>
</div>
<div class="sect2">
<h3 id="storage_lvm">
<span>7.11. LVM Backend</span>
 <a class="headerlink" href="#storage_lvm" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Storage pool type: <span class="monospaced">lvm</span></p></div>
<div class="paragraph">
<p>LVM is a light software layer on top of hard disks and partitions. It
can be used to split available disk space into smaller logical
volumes. LVM is widely used on Linux and makes managing hard drives
easier.</p></div>
<div class="paragraph">
<p>Another use case is to put LVM on top of a big iSCSI LUN. That way you
can easily manage space on that iSCSI LUN, which would not be possible
otherwise, because the iSCSI specification does not define a
management interface for space allocation.</p></div>
<div class="sect3">
<h4 id="_configuration_8">7.11.1. Configuration
 <a class="headerlink" href="#_configuration_8" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The LVM backend supports the common storage properties <span class="monospaced">content</span>, <span class="monospaced">nodes</span>,
<span class="monospaced">disable</span>, and the following LVM specific properties:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">vgname</span>
</dt>
<dd>
<p>
LVM volume group name. This must point to an existing volume group.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">base</span>
</dt>
<dd>
<p>
Base volume. This volume is automatically activated before accessing
the storage. This is mostly useful when the LVM volume group resides
on a remote iSCSI server.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">saferemove</span>
</dt>
<dd>
<p>
Called "Wipe Removed Volumes" in the web UI. Zero-out data when removing LVs.
When removing a volume, this makes sure that all data gets erased and cannot be
accessed by other LVs created later (which happen to be assigned the same
physical extents). This is a costly operation, but may be required as a security
measure in certain environments.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">saferemove_throughput</span>
</dt>
<dd>
<p>
Wipe throughput (<span class="monospaced">cstream -t</span> parameter value).
</p>
</dd>
</dl></div>
<div class="listingblock">
<div class="title">Configuration Example (<span class="monospaced">/etc/pve/storage.cfg</span>)</div>
<div class="content monospaced">
<pre>lvm: myspace
        vgname myspace
        content rootdir,images</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_file_naming_conventions_4">7.11.2. File naming conventions
 <a class="headerlink" href="#_file_naming_conventions_4" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The backend use basically the same naming conventions as the ZFS pool
backend.</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>vm-&lt;VMID&gt;-&lt;NAME&gt;      // normal VM images</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_storage_features_7">7.11.3. Storage Features
 <a class="headerlink" href="#_storage_features_7" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>LVM is a typical block storage, but this backend does not support
snapshots and clones. Unfortunately, normal LVM snapshots are quite
inefficient, because they interfere with all writes on the entire volume
group during snapshot time.</p></div>
<div class="paragraph">
<p>One big advantage is that you can use it on top of a shared storage,
for example, an iSCSI LUN. The backend itself implements proper cluster-wide
locking.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">The newer LVM-thin backend allows snapshots and clones, but does
not support shared storage.</td>
</tr></tbody></table>
</div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<caption class="title">Table 10. Storage features for backend <span class="monospaced">lvm</span></caption>
<colgroup><col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Content types  </th>
<th class="tableblock halign-left valign-top">Image formats  </th>
<th class="tableblock halign-left valign-top">Shared   </th>
<th class="tableblock halign-left valign-top">Snapshots </th>
<th class="tableblock halign-left valign-top">Clones</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">images rootdir</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">raw</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">possible</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">no</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">no</p></td>
</tr>
</tbody>
</table>
</div>
<div class="sect3">
<h4 id="_examples_6">7.11.4. Examples
 <a class="headerlink" href="#_examples_6" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>List available volume groups:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># pvesm lvmscan</pre>
</div></div>
</div>
</div>
<div class="sect2">
<h3 id="storage_lvmthin">
<span>7.12. LVM thin Backend</span>
 <a class="headerlink" href="#storage_lvmthin" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Storage pool type: <span class="monospaced">lvmthin</span></p></div>
<div class="paragraph">
<p>LVM normally allocates blocks when you create a volume. LVM thin pools
instead allocates blocks when they are written. This behaviour is
called thin-provisioning, because volumes can be much larger than
physically available space.</p></div>
<div class="paragraph">
<p>You can use the normal LVM command-line tools to manage and create LVM
thin pools (see <span class="monospaced">man lvmthin</span> for details). Assuming you already have
a LVM volume group called <span class="monospaced">pve</span>, the following commands create a new
LVM thin pool (size 100G) called <span class="monospaced">data</span>:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>lvcreate -L 100G -n data pve
lvconvert --type thin-pool pve/data</pre>
</div></div>
<div class="sect3">
<h4 id="_configuration_9">7.12.1. Configuration
 <a class="headerlink" href="#_configuration_9" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The LVM thin backend supports the common storage properties <span class="monospaced">content</span>, <span class="monospaced">nodes</span>,
<span class="monospaced">disable</span>, and the following LVM specific properties:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">vgname</span>
</dt>
<dd>
<p>
LVM volume group name. This must point to an existing volume group.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">thinpool</span>
</dt>
<dd>
<p>
The name of the LVM thin pool.
</p>
</dd>
</dl></div>
<div class="listingblock">
<div class="title">Configuration Example (<span class="monospaced">/etc/pve/storage.cfg</span>)</div>
<div class="content monospaced">
<pre>lvmthin: local-lvm
        thinpool data
        vgname pve
        content rootdir,images</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_file_naming_conventions_5">7.12.2. File naming conventions
 <a class="headerlink" href="#_file_naming_conventions_5" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The backend use basically the same naming conventions as the ZFS pool
backend.</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>vm-&lt;VMID&gt;-&lt;NAME&gt;      // normal VM images</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_storage_features_8">7.12.3. Storage Features
 <a class="headerlink" href="#_storage_features_8" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>LVM thin is a block storage, but fully supports snapshots and clones
efficiently. New volumes are automatically initialized with zero.</p></div>
<div class="paragraph">
<p>It must be mentioned that LVM thin pools cannot be shared across
multiple nodes, so you can only use them as local storage.</p></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<caption class="title">Table 11. Storage features for backend <span class="monospaced">lvmthin</span></caption>
<colgroup><col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Content types  </th>
<th class="tableblock halign-left valign-top">Image formats  </th>
<th class="tableblock halign-left valign-top">Shared   </th>
<th class="tableblock halign-left valign-top">Snapshots </th>
<th class="tableblock halign-left valign-top">Clones</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">images rootdir</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">raw</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">no</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
</tr>
</tbody>
</table>
</div>
<div class="sect3">
<h4 id="_examples_7">7.12.4. Examples
 <a class="headerlink" href="#_examples_7" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>List available LVM thin pools on volume group <span class="monospaced">pve</span>:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># pvesm lvmthinscan pve</pre>
</div></div>
</div>
</div>
<div class="sect2">
<h3 id="storage_open_iscsi">
<span>7.13. Open-iSCSI initiator</span>
 <a class="headerlink" href="#storage_open_iscsi" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Storage pool type: <span class="monospaced">iscsi</span></p></div>
<div class="paragraph">
<p>iSCSI is a widely employed technology used to connect to storage
servers. Almost all storage vendors support iSCSI. There are also open
source iSCSI target solutions available,
e.g. <a href="https://www.openmediavault.org/">OpenMediaVault</a>, which is based on
Debian.</p></div>
<div class="paragraph">
<p>To use this backend, you need to install the
<a href="https://www.open-iscsi.com/">Open-iSCSI</a> (<span class="monospaced">open-iscsi</span>) package. This is a
standard Debian package, but it is not installed by default to save
resources.</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># apt-get install open-iscsi</pre>
</div></div>
<div class="paragraph">
<p>Low-level iscsi management task can be done using the <span class="monospaced">iscsiadm</span> tool.</p></div>
<div class="sect3">
<h4 id="_configuration_10">7.13.1. Configuration
 <a class="headerlink" href="#_configuration_10" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The backend supports the common storage properties <span class="monospaced">content</span>, <span class="monospaced">nodes</span>,
<span class="monospaced">disable</span>, and the following iSCSI specific properties:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
portal
</dt>
<dd>
<p>
iSCSI portal (IP or DNS name with optional port).
</p>
</dd>
<dt class="hdlist1">
target
</dt>
<dd>
<p>
iSCSI target.
</p>
</dd>
</dl></div>
<div class="listingblock">
<div class="title">Configuration Example (<span class="monospaced">/etc/pve/storage.cfg</span>)</div>
<div class="content monospaced">
<pre>iscsi: mynas
     portal 10.10.10.1
     target iqn.2006-01.openfiler.com:tsn.dcb5aaaddd
     content none</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">If you want to use LVM on top of iSCSI, it make sense to set
<span class="monospaced">content none</span>. That way it is not possible to create VMs using iSCSI
LUNs directly.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="_file_naming_conventions_6">7.13.2. File naming conventions
 <a class="headerlink" href="#_file_naming_conventions_6" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The iSCSI protocol does not define an interface to allocate or delete
data. Instead, that needs to be done on the target side and is vendor
specific. The target simply exports them as numbered LUNs. So Proxmox VE
iSCSI volume names just encodes some information about the LUN as seen
by the linux kernel.</p></div>
</div>
<div class="sect3">
<h4 id="_storage_features_9">7.13.3. Storage Features
 <a class="headerlink" href="#_storage_features_9" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>iSCSI is a block level type storage, and provides no management
interface.  So it is usually best to export one big LUN, and setup LVM
on top of that LUN. You can then use the LVM plugin to manage the
storage on that iSCSI LUN.</p></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<caption class="title">Table 12. Storage features for backend <span class="monospaced">iscsi</span></caption>
<colgroup><col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Content types  </th>
<th class="tableblock halign-left valign-top">Image formats  </th>
<th class="tableblock halign-left valign-top">Shared </th>
<th class="tableblock halign-left valign-top">Snapshots </th>
<th class="tableblock halign-left valign-top">Clones</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">images none</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">raw</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">no</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">no</p></td>
</tr>
</tbody>
</table>
</div>
<div class="sect3">
<h4 id="_examples_8">7.13.4. Examples
 <a class="headerlink" href="#_examples_8" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Scan a remote iSCSI portal, and returns a list of possible targets:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>pvesm scan iscsi &lt;HOST[:PORT]&gt;</pre>
</div></div>
</div>
</div>
<div class="sect2">
<h3 id="storage_iscsidirect">
<span>7.14. User Mode iSCSI Backend</span>
 <a class="headerlink" href="#storage_iscsidirect" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Storage pool type: <span class="monospaced">iscsidirect</span></p></div>
<div class="paragraph">
<p>This backend provides basically the same functionality as the Open-iSCSI backed,
but uses a user-level library to implement it. You need to install the
<span class="monospaced">libiscsi-bin</span> package in order to use this backend.</p></div>
<div class="paragraph">
<p>It should be noted that there are no kernel drivers involved, so this
can be viewed as performance optimization. But this comes with the
drawback that you cannot use LVM on top of such iSCSI LUN. So you need
to manage all space allocations at the storage server side.</p></div>
<div class="sect3">
<h4 id="_configuration_11">7.14.1. Configuration
 <a class="headerlink" href="#_configuration_11" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The user mode iSCSI backend uses the same configuration options as the
Open-iSCSI backed.</p></div>
<div class="listingblock">
<div class="title">Configuration Example (<span class="monospaced">/etc/pve/storage.cfg</span>)</div>
<div class="content monospaced">
<pre>iscsidirect: faststore
     portal 10.10.10.1
     target iqn.2006-01.openfiler.com:tsn.dcb5aaaddd</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_storage_features_10">7.14.2. Storage Features
 <a class="headerlink" href="#_storage_features_10" title="Permalink to this heading"></a>
</h4>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">This backend works with VMs only. Containers cannot use this
driver.</td>
</tr></tbody></table>
</div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<caption class="title">Table 13. Storage features for backend <span class="monospaced">iscsidirect</span></caption>
<colgroup><col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Content types  </th>
<th class="tableblock halign-left valign-top">Image formats  </th>
<th class="tableblock halign-left valign-top">Shared </th>
<th class="tableblock halign-left valign-top">Snapshots </th>
<th class="tableblock halign-left valign-top">Clones</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">images</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">raw</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">no</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">no</p></td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="sect2">
<h3 id="ceph_rados_block_devices">
<span>7.15. Ceph RADOS Block Devices (RBD)</span>
 <a class="headerlink" href="#ceph_rados_block_devices" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Storage pool type: <span class="monospaced">rbd</span></p></div>
<div class="paragraph">
<p><a href="https://ceph.com">Ceph</a> is a distributed object store and file system
designed to provide excellent performance, reliability and
scalability. RADOS block devices implement a feature rich block level
storage, and you get the following advantages:</p></div>
<div class="ulist"><ul>
<li>
<p>
thin provisioning
</p>
</li>
<li>
<p>
resizable volumes
</p>
</li>
<li>
<p>
distributed and redundant (striped over multiple OSDs)
</p>
</li>
<li>
<p>
full snapshot and clone capabilities
</p>
</li>
<li>
<p>
self healing
</p>
</li>
<li>
<p>
no single point of failure
</p>
</li>
<li>
<p>
scalable to the exabyte level
</p>
</li>
<li>
<p>
kernel and user space implementation available
</p>
</li>
</ul></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">For smaller deployments, it is also possible to run Ceph
services directly on your Proxmox VE nodes. Recent hardware has plenty
of CPU power and RAM, so running storage services and VMs on same node
is possible.</td>
</tr></tbody></table>
</div>
<div class="sect3">
<h4 id="storage_rbd_config">7.15.1. Configuration
 <a class="headerlink" href="#storage_rbd_config" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>This backend supports the common storage properties <span class="monospaced">nodes</span>,
<span class="monospaced">disable</span>, <span class="monospaced">content</span>, and the following <span class="monospaced">rbd</span> specific properties:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
monhost
</dt>
<dd>
<p>
List of monitor daemon IPs. Optional, only needed if Ceph is not running on the
Proxmox VE cluster.
</p>
</dd>
<dt class="hdlist1">
pool
</dt>
<dd>
<p>
Ceph pool name.
</p>
</dd>
<dt class="hdlist1">
username
</dt>
<dd>
<p>
RBD user ID. Optional, only needed if Ceph is not running on the Proxmox VE cluster.
Note that only the user ID should be used. The "client." type prefix must be
left out.
</p>
</dd>
<dt class="hdlist1">
krbd
</dt>
<dd>
<p>
Enforce access to rados block devices through the krbd kernel module. Optional.
</p>
</dd>
</dl></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Containers will use <span class="monospaced">krbd</span> independent of the option value.</td>
</tr></tbody></table>
</div>
<div class="listingblock">
<div class="title">Configuration Example for a external Ceph cluster (<span class="monospaced">/etc/pve/storage.cfg</span>)</div>
<div class="content monospaced">
<pre>rbd: ceph-external
        monhost 10.1.1.20 10.1.1.21 10.1.1.22
        pool ceph-external
        content images
        username admin</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">You can use the <span class="monospaced">rbd</span> utility to do low-level management tasks.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="_authentication">7.15.2. Authentication
 <a class="headerlink" href="#_authentication" title="Permalink to this heading"></a>
</h4>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">If Ceph is installed locally on the Proxmox VE cluster, the following is done
automatically when adding the storage.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>If you use <span class="monospaced">cephx</span> authentication, which is enabled by default, you need to
provide the keyring from the external Ceph cluster.</p></div>
<div class="paragraph">
<p>To configure the storage via the CLI, you first need to make the file
containing the keyring available. One way is to copy the file from the external
Ceph cluster directly to one of the Proxmox VE nodes. The following example will
copy it to the <span class="monospaced">/root</span> directory of the node on which we run it:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># scp &lt;external cephserver&gt;:/etc/ceph/ceph.client.admin.keyring /root/rbd.keyring</pre>
</div></div>
<div class="paragraph">
<p>Then use the <span class="monospaced">pvesm</span> CLI tool to configure the external RBD storage, use the
<span class="monospaced">--keyring</span> parameter, which needs to be a path to the keyring file that you
copied.  For example:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># pvesm add rbd &lt;name&gt; --monhost "10.1.1.20 10.1.1.21 10.1.1.22" --content images --keyring /root/rbd.keyring</pre>
</div></div>
<div class="paragraph">
<p>When configuring an external RBD storage via the GUI, you can copy and paste
the keyring into the appropriate field.</p></div>
<div class="paragraph">
<p>The keyring will be stored at</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># /etc/pve/priv/ceph/&lt;STORAGE_ID&gt;.keyring</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">Creating a keyring with only the needed capabilities is recommend when
connecting to an external cluster. For further information on Ceph user
management, see the Ceph docs.<span class="footnote" id="_footnote_cephusermgmt" data-note="<a href=&quot;https://docs.ceph.com/en/quincy/rados/operations/user-management/&quot;>Ceph User Management</a>">[<a id="_footnoteref_12" href="#_footnote_12" title="View footnote" class="footnote">12</a>]</span></td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="_ceph_client_configuration_optional">7.15.3. Ceph client configuration (optional)
 <a class="headerlink" href="#_ceph_client_configuration_optional" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Connecting to an external Ceph storage doesn’t always allow setting
client-specific options in the config DB on the external cluster. You can add a
<span class="monospaced">ceph.conf</span> beside the Ceph keyring to change the Ceph client configuration for
the storage.</p></div>
<div class="paragraph">
<p>The ceph.conf needs to have the same name as the storage.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># /etc/pve/priv/ceph/&lt;STORAGE_ID&gt;.conf</pre>
</div></div>
<div class="paragraph">
<p>See the RBD configuration reference <span class="footnote" data-note="RBD configuration reference
<a href=&quot;https://docs.ceph.com/en/quincy/rbd/rbd-config-ref/&quot;>https://docs.ceph.com/en/quincy/rbd/rbd-config-ref/</a>">[<a id="_footnoteref_13" href="#_footnote_13" title="View footnote" class="footnote">13</a>]</span> for possible settings.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Do not change these settings lightly. Proxmox VE is merging the
&lt;STORAGE_ID&gt;.conf with the storage configuration.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="_storage_features_11">7.15.4. Storage Features
 <a class="headerlink" href="#_storage_features_11" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The <span class="monospaced">rbd</span> backend is a block level storage, and implements full
snapshot and clone functionality.</p></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<caption class="title">Table 14. Storage features for backend <span class="monospaced">rbd</span></caption>
<colgroup><col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Content types  </th>
<th class="tableblock halign-left valign-top">Image formats  </th>
<th class="tableblock halign-left valign-top">Shared </th>
<th class="tableblock halign-left valign-top">Snapshots </th>
<th class="tableblock halign-left valign-top">Clones</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">images rootdir</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">raw</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="sect2">
<h3 id="storage_cephfs">
<span>7.16. Ceph Filesystem (CephFS)</span>
 <a class="headerlink" href="#storage_cephfs" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Storage pool type: <span class="monospaced">cephfs</span></p></div>
<div class="paragraph">
<p>CephFS implements a POSIX-compliant filesystem, using a <a href="https://ceph.com">Ceph</a>
storage cluster to store its data. As CephFS builds upon Ceph, it shares most of
its properties. This includes redundancy, scalability, self-healing, and high
availability.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">Proxmox VE can <a href="#chapter_pveceph">manage Ceph setups</a>, which makes
configuring a CephFS storage easier. As modern hardware offers a lot of
processing power and RAM, running storage services and VMs on same node is
possible without a significant performance impact.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>To use the CephFS storage plugin, you must replace the stock Debian Ceph client,
by adding our <a href="#sysadmin_package_repositories_ceph">Ceph repository</a>.
Once added, run <span class="monospaced">apt update</span>, followed by <span class="monospaced">apt dist-upgrade</span>, in order to get
the newest packages.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">Please ensure that there are no other Ceph repositories configured.
Otherwise the installation will fail or there will be mixed package versions on
the node, leading to unexpected behavior.</td>
</tr></tbody></table>
</div>
<div class="sect3">
<h4 id="storage_cephfs_config">7.16.1. Configuration
 <a class="headerlink" href="#storage_cephfs_config" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>This backend supports the common storage properties <span class="monospaced">nodes</span>,
<span class="monospaced">disable</span>, <span class="monospaced">content</span>, as well as the following <span class="monospaced">cephfs</span> specific properties:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
fs-name
</dt>
<dd>
<p>
Name of the Ceph FS.
</p>
</dd>
<dt class="hdlist1">
monhost
</dt>
<dd>
<p>
List of monitor daemon addresses. Optional, only needed if Ceph is not running
on the Proxmox VE cluster.
</p>
</dd>
<dt class="hdlist1">
path
</dt>
<dd>
<p>
The local mount point. Optional, defaults to <span class="monospaced">/mnt/pve/&lt;STORAGE_ID&gt;/</span>.
</p>
</dd>
<dt class="hdlist1">
username
</dt>
<dd>
<p>
Ceph user id. Optional, only needed if Ceph is not running on the Proxmox VE cluster,
where it defaults to <span class="monospaced">admin</span>.
</p>
</dd>
<dt class="hdlist1">
subdir
</dt>
<dd>
<p>
CephFS subdirectory to mount. Optional, defaults to <span class="monospaced">/</span>.
</p>
</dd>
<dt class="hdlist1">
fuse
</dt>
<dd>
<p>
Access CephFS through FUSE, instead of the kernel client. Optional, defaults
to <span class="monospaced">0</span>.
</p>
</dd>
</dl></div>
<div class="listingblock">
<div class="title">Configuration example for an external Ceph cluster (<span class="monospaced">/etc/pve/storage.cfg</span>)</div>
<div class="content monospaced">
<pre>cephfs: cephfs-external
        monhost 10.1.1.20 10.1.1.21 10.1.1.22
        path /mnt/pve/cephfs-external
        content backup
        username admin
        fs-name cephfs</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Don’t forget to set up the client’s secret key file, if cephx was not
disabled.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="_authentication_2">7.16.2. Authentication
 <a class="headerlink" href="#_authentication_2" title="Permalink to this heading"></a>
</h4>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">If Ceph is installed locally on the Proxmox VE cluster, the following is done
automatically when adding the storage.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>If you use <span class="monospaced">cephx</span> authentication, which is enabled by default, you need to
provide the secret from the external Ceph cluster.</p></div>
<div class="paragraph">
<p>To configure the storage via the CLI, you first need to make the file
containing the secret available. One way is to copy the file from the external
Ceph cluster directly to one of the Proxmox VE nodes. The following example will
copy it to the <span class="monospaced">/root</span> directory of the node on which we run it:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># scp &lt;external cephserver&gt;:/etc/ceph/cephfs.secret /root/cephfs.secret</pre>
</div></div>
<div class="paragraph">
<p>Then use the <span class="monospaced">pvesm</span> CLI tool to configure the external RBD storage, use the
<span class="monospaced">--keyring</span> parameter, which needs to be a path to the secret file that you
copied.  For example:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># pvesm add cephfs &lt;name&gt; --monhost "10.1.1.20 10.1.1.21 10.1.1.22" --content backup --keyring /root/cephfs.secret</pre>
</div></div>
<div class="paragraph">
<p>When configuring an external RBD storage via the GUI, you can copy and paste
the secret into the appropriate field.</p></div>
<div class="paragraph">
<p>The secret is only the key itself, as opposed to the <span class="monospaced">rbd</span> backend which also
contains a <span class="monospaced">[client.userid]</span> section.</p></div>
<div class="paragraph">
<p>The secret will be stored at</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># /etc/pve/priv/ceph/&lt;STORAGE_ID&gt;.secret</pre>
</div></div>
<div class="paragraph">
<p>A secret can be received from the Ceph cluster (as Ceph admin) by issuing the
command below, where <span class="monospaced">userid</span> is the client ID that has been configured to
access the cluster. For further information on Ceph user management, see the
Ceph docs.<span class="footnoteref">[<a href="#_footnote_12" title="View footnote" class="footnote">12</a>]</span></p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># ceph auth get-key client.userid &gt; cephfs.secret</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_storage_features_12">7.16.3. Storage Features
 <a class="headerlink" href="#_storage_features_12" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The <span class="monospaced">cephfs</span> backend is a POSIX-compliant filesystem, on top of a Ceph cluster.</p></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<caption class="title">Table 15. Storage features for backend <span class="monospaced">cephfs</span></caption>
<colgroup><col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Content types              </th>
<th class="tableblock halign-left valign-top">Image formats  </th>
<th class="tableblock halign-left valign-top">Shared </th>
<th class="tableblock halign-left valign-top">Snapshots </th>
<th class="tableblock halign-left valign-top">Clones</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">vztmpl iso backup snippets</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">none</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes<sup>[1]</sup></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">no</p></td>
</tr>
</tbody>
</table>
<div class="paragraph">
<p><sup>[1]</sup> While no known bugs exist, snapshots are not yet guaranteed to be stable,
as they lack sufficient testing.</p></div>
</div>
</div>
<div class="sect2">
<h3 id="storage_btrfs">
<span>7.17. BTRFS Backend</span>
 <a class="headerlink" href="#storage_btrfs" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Storage pool type: <span class="monospaced">btrfs</span></p></div>
<div class="paragraph">
<p>On the surface, this storage type is very similar to the directory storage type,
so see the directory backend section for a general overview.</p></div>
<div class="paragraph">
<p>The main difference is that with this storage type <span class="monospaced">raw</span> formatted disks will be
placed in a subvolume, in order to allow taking snapshots and supporting offline
storage migration with snapshots being preserved.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">BTRFS will honor the <span class="monospaced">O_DIRECT</span> flag when opening files, meaning VMs
should not use cache mode <span class="monospaced">none</span>, otherwise there will be checksum errors.</td>
</tr></tbody></table>
</div>
<div class="sect3">
<h4 id="_configuration_12">7.17.1. Configuration
 <a class="headerlink" href="#_configuration_12" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>This backend is configured similarly to the directory storage. Note that when
adding a directory as a BTRFS storage, which is not itself also the mount point,
it is highly recommended to specify the actual mount point via the
<span class="monospaced">is_mountpoint</span> option.</p></div>
<div class="paragraph">
<p>For example, if a BTRFS file system is mounted at <span class="monospaced">/mnt/data2</span> and its
<span class="monospaced">pve-storage/</span> subdirectory (which may be a snapshot, which is recommended)
should be added as a storage pool called <span class="monospaced">data2</span>, you can use the following
entry:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>btrfs: data2
        path /mnt/data2/pve-storage
        content rootdir,images
        is_mountpoint /mnt/data2</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_snapshots">7.17.2. Snapshots
 <a class="headerlink" href="#_snapshots" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>When taking a snapshot of a subvolume or <span class="monospaced">raw</span> file, the snapshot will be
created as a read-only subvolume with the same path followed by an <span class="monospaced">@</span> and the
snapshot’s name.</p></div>
</div>
</div>
<div class="sect2">
<h3 id="storage_zfs">
<span>7.18. ZFS over ISCSI Backend</span>
 <a class="headerlink" href="#storage_zfs" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Storage pool type: <span class="monospaced">zfs</span></p></div>
<div class="paragraph">
<p>This backend accesses a remote machine having a ZFS pool as storage and an iSCSI
target implementation via <span class="monospaced">ssh</span>. For each guest disk it creates a ZVOL and,
exports it as iSCSI LUN. This LUN is used by Proxmox VE for the guest disk.</p></div>
<div class="paragraph">
<p>The following iSCSI target implementations are supported:</p></div>
<div class="ulist"><ul>
<li>
<p>
LIO (Linux)
</p>
</li>
<li>
<p>
IET (Linux)
</p>
</li>
<li>
<p>
ISTGT (FreeBSD)
</p>
</li>
<li>
<p>
Comstar (Solaris)
</p>
</li>
</ul></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">This plugin needs a ZFS capable remote storage appliance, you cannot use
it to create a ZFS Pool on a regular Storage Appliance/SAN</td>
</tr></tbody></table>
</div>
<div class="sect3">
<h4 id="_configuration_13">7.18.1. Configuration
 <a class="headerlink" href="#_configuration_13" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>In order to use the ZFS over iSCSI plugin you need to configure the remote
machine (target) to accept <span class="monospaced">ssh</span> connections from the Proxmox VE node. Proxmox VE connects to the target for creating the ZVOLs and exporting them via iSCSI.
Authentication is done through a ssh-key (without password protection) stored in
<span class="monospaced">/etc/pve/priv/zfs/&lt;target_ip&gt;_id_rsa</span></p></div>
<div class="paragraph">
<p>The following steps create a ssh-key and distribute it to the storage machine
with IP 192.0.2.1:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>mkdir /etc/pve/priv/zfs
ssh-keygen -f /etc/pve/priv/zfs/192.0.2.1_id_rsa
ssh-copy-id -i /etc/pve/priv/zfs/192.0.2.1_id_rsa.pub [email protected]
ssh -i /etc/pve/priv/zfs/192.0.2.1_id_rsa [email protected]</pre>
</div></div>
<div class="paragraph">
<p>The backend supports the common storage properties <span class="monospaced">content</span>, <span class="monospaced">nodes</span>,
<span class="monospaced">disable</span>, and the following ZFS over ISCSI specific properties:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
pool
</dt>
<dd>
<p>
The ZFS pool/filesystem on the iSCSI target. All allocations are done within that
pool.
</p>
</dd>
<dt class="hdlist1">
portal
</dt>
<dd>
<p>
iSCSI portal (IP or DNS name with optional port).
</p>
</dd>
<dt class="hdlist1">
target
</dt>
<dd>
<p>
iSCSI target.
</p>
</dd>
<dt class="hdlist1">
iscsiprovider
</dt>
<dd>
<p>
The iSCSI target implementation used on the remote machine
</p>
</dd>
<dt class="hdlist1">
comstar_tg
</dt>
<dd>
<p>
target group for comstar views.
</p>
</dd>
<dt class="hdlist1">
comstar_hg
</dt>
<dd>
<p>
host group for comstar views.
</p>
</dd>
<dt class="hdlist1">
lio_tpg
</dt>
<dd>
<p>
target portal group for Linux LIO targets
</p>
</dd>
<dt class="hdlist1">
nowritecache
</dt>
<dd>
<p>
disable write caching on the target
</p>
</dd>
<dt class="hdlist1">
blocksize
</dt>
<dd>
<p>
Set ZFS blocksize parameter.
</p>
</dd>
<dt class="hdlist1">
sparse
</dt>
<dd>
<p>
Use ZFS thin-provisioning. A sparse volume is a volume whose
reservation is not equal to the volume size.
</p>
</dd>
</dl></div>
<div class="listingblock">
<div class="title">Configuration Examples (<span class="monospaced">/etc/pve/storage.cfg</span>)</div>
<div class="content monospaced">
<pre>zfs: lio
   blocksize 4k
   iscsiprovider LIO
   pool tank
   portal 192.0.2.111
   target iqn.2003-01.org.linux-iscsi.lio.x8664:sn.xxxxxxxxxxxx
   content images
   lio_tpg tpg1
   sparse 1

zfs: solaris
   blocksize 4k
   target iqn.2010-08.org.illumos:02:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx:tank1
   pool tank
   iscsiprovider comstar
   portal 192.0.2.112
   content images

zfs: freebsd
   blocksize 4k
   target iqn.2007-09.jp.ne.peach.istgt:tank1
   pool tank
   iscsiprovider istgt
   portal 192.0.2.113
   content images

zfs: iet
   blocksize 4k
   target iqn.2001-04.com.example:tank1
   pool tank
   iscsiprovider iet
   portal 192.0.2.114
   content images</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_storage_features_13">7.18.2. Storage Features
 <a class="headerlink" href="#_storage_features_13" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The ZFS over iSCSI plugin provides a shared storage, which is capable of
snapshots. You need to make sure that the ZFS appliance does not become a single
point of failure in your deployment.</p></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<caption class="title">Table 16. Storage features for backend <span class="monospaced">iscsi</span></caption>
<colgroup><col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Content types  </th>
<th class="tableblock halign-left valign-top">Image formats  </th>
<th class="tableblock halign-left valign-top">Shared </th>
<th class="tableblock halign-left valign-top">Snapshots </th>
<th class="tableblock halign-left valign-top">Clones</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">images</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">raw</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">no</p></td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="chapter_pveceph">
8. Deploy Hyper-Converged Ceph Cluster
 <a class="headerlink" href="#chapter_pveceph" title="Permalink to this heading"></a>
</h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="_introduction_2">
<span>8.1. Introduction</span>
 <a class="headerlink" href="#_introduction_2" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-ceph-status-dashboard.png">
<img src="images/screenshot/gui-ceph-status-dashboard.png" alt="screenshot/gui-ceph-status-dashboard.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>Proxmox VE unifies your compute and storage systems, that is, you can use the same
physical nodes within a cluster for both computing (processing VMs and
containers) and replicated storage. The traditional silos of compute and
storage resources can be wrapped up into a single hyper-converged appliance.
Separate storage networks (SANs) and connections via network attached storage
(NAS) disappear. With the integration of Ceph, an open source software-defined
storage platform, Proxmox VE has the ability to run and manage Ceph storage directly
on the hypervisor nodes.</p></div>
<div class="paragraph">
<p>Ceph is a distributed object store and file system designed to provide
excellent performance, reliability and scalability.</p></div>
<div class="ulist"><div class="title">Some advantages of Ceph on Proxmox VE are:</div><ul>
<li>
<p>
Easy setup and management via CLI and GUI
</p>
</li>
<li>
<p>
Thin provisioning
</p>
</li>
<li>
<p>
Snapshot support
</p>
</li>
<li>
<p>
Self healing
</p>
</li>
<li>
<p>
Scalable to the exabyte level
</p>
</li>
<li>
<p>
Provides block, file system, and object storage
</p>
</li>
<li>
<p>
Setup pools with different performance and redundancy characteristics
</p>
</li>
<li>
<p>
Data is replicated, making it fault tolerant
</p>
</li>
<li>
<p>
Runs on commodity hardware
</p>
</li>
<li>
<p>
No need for hardware RAID controllers
</p>
</li>
<li>
<p>
Open source
</p>
</li>
</ul></div>
<div class="paragraph">
<p>For small to medium-sized deployments, it is possible to install a Ceph server
for using RADOS Block Devices (RBD) or CephFS directly on your Proxmox VE cluster
nodes (see <a href="#ceph_rados_block_devices">Ceph RADOS Block Devices (RBD)</a>).
Recent hardware has a lot of CPU power and RAM, so running storage services and
virtual guests on the same node is possible.</p></div>
<div class="paragraph">
<p>To simplify management, Proxmox VE provides you native integration to install and
manage <a href="http://ceph.com">Ceph</a> services on Proxmox VE nodes either via the built-in web interface, or
using the <em>pveceph</em> command line tool.</p></div>
</div>
<div class="sect2">
<h3 id="_terminology">
<span>8.2. Terminology</span>
 <a class="headerlink" href="#_terminology" title="Permalink to this heading"></a>
</h3>
<div class="ulist"><div class="title">Ceph consists of multiple Daemons, for use as an RBD storage:</div><ul>
<li>
<p>
Ceph Monitor (ceph-mon, or MON)
</p>
</li>
<li>
<p>
Ceph Manager (ceph-mgr, or MGS)
</p>
</li>
<li>
<p>
Ceph Metadata Service (ceph-mds, or MDS)
</p>
</li>
<li>
<p>
Ceph Object Storage Daemon (ceph-osd, or OSD)
</p>
</li>
</ul></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">We highly recommend to get familiar with Ceph
<span class="footnote" data-note="Ceph intro <a href=&quot;https://docs.ceph.com/en/quincy/start/intro/&quot;>https://docs.ceph.com/en/quincy/start/intro/</a>">[<a id="_footnoteref_14" href="#_footnote_14" title="View footnote" class="footnote">14</a>]</span>,
its architecture
<span class="footnote" data-note="Ceph architecture <a href=&quot;https://docs.ceph.com/en/quincy/architecture/&quot;>https://docs.ceph.com/en/quincy/architecture/</a>">[<a id="_footnoteref_15" href="#_footnote_15" title="View footnote" class="footnote">15</a>]</span>
and vocabulary
<span class="footnote" data-note="Ceph glossary <a href=&quot;https://docs.ceph.com/en/quincy/glossary&quot;>https://docs.ceph.com/en/quincy/glossary</a>">[<a id="_footnoteref_16" href="#_footnote_16" title="View footnote" class="footnote">16</a>]</span>.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect2">
<h3 id="_recommendations_for_a_healthy_ceph_cluster">
<span>8.3. Recommendations for a Healthy Ceph Cluster</span>
 <a class="headerlink" href="#_recommendations_for_a_healthy_ceph_cluster" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>To build a hyper-converged Proxmox + Ceph Cluster, you must use at least three
(preferably) identical servers for the setup.</p></div>
<div class="paragraph">
<p>Check also the recommendations from
<a href="https://docs.ceph.com/en/quincy/start/hardware-recommendations/">Ceph’s website</a>.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">The recommendations below should be seen as a rough guidance for choosing
hardware. Therefore, it is still essential to adapt it to your specific needs.
You should test your setup and monitor health and performance continuously.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<div class="title">CPU</div><p>Ceph services can be classified into two categories:
* Intensive CPU usage, benefiting from high CPU base frequencies and multiple
  cores. Members of that category are:
<strong> Object Storage Daemon (OSD) services
</strong> Meta Data Service (MDS) used for CephFS
* Moderate CPU usage, not needing multiple CPU cores. These are:
<strong> Monitor (MON) services
</strong> Manager (MGR) services</p></div>
<div class="paragraph">
<p>As a simple rule of thumb, you should assign at least one CPU core (or thread)
to each Ceph service to provide the minimum resources required for stable and
durable Ceph performance.</p></div>
<div class="paragraph">
<p>For example, if you plan to run a Ceph monitor, a Ceph manager and 6 Ceph OSDs
services on a node you should reserve 8 CPU cores purely for Ceph when targeting
basic and stable performance.</p></div>
<div class="paragraph">
<p>Note that OSDs CPU usage depend mostly from the disks performance. The higher
the possible IOPS (<strong>IO</strong> <strong>O</strong>perations per <strong>S</strong>econd) of a disk, the more CPU
can be utilized by a OSD service.
For modern enterprise SSD disks, like NVMe’s that can permanently sustain a high
IOPS load over 100’000 with sub millisecond latency, each OSD can use multiple
CPU threads, e.g., four to six CPU threads utilized per NVMe backed OSD is
likely for very high performance disks.</p></div>
<div class="paragraph">
<div class="title">Memory</div><p>Especially in a hyper-converged setup, the memory consumption needs to be
carefully planned out and monitored. In addition to the predicted memory usage
of virtual machines and containers, you must also account for having enough
memory available for Ceph to provide excellent and stable performance.</p></div>
<div class="paragraph">
<p>As a rule of thumb, for roughly <strong>1 TiB of data, 1 GiB of memory</strong> will be used
by an OSD. While the usage might be less under normal conditions, it will use
most during critical operations like recovery, re-balancing or backfilling.
That means that you should avoid maxing out your available memory already on
normal operation, but rather leave some headroom to cope with outages.</p></div>
<div class="paragraph">
<p>The OSD service itself will use additional memory. The Ceph BlueStore backend of
the daemon requires by default <strong>3-5 GiB of memory</strong> (adjustable).</p></div>
<div class="paragraph">
<div class="title">Network</div><p>We recommend a network bandwidth of at least 10 Gbps, or more, to be used
exclusively for Ceph traffic. A meshed network setup
<span class="footnote" data-note="Full Mesh Network for Ceph <a href=&quot;https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server&quot;>https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server</a>">[<a id="_footnoteref_17" href="#_footnote_17" title="View footnote" class="footnote">17</a>]</span>
is also an option for three to five node clusters, if there are no 10+ Gbps
switches available.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Important" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAALa0lEQVRogdWZa2wc1RXHfzM7O/te
P9e1vXHSmEdjx3YeDkkaF6REKRRCEDSEFNkRjdSWSsgC2iqoRWqLQKiqIBg1NOQDiMeHtkQIQkRR
S9S4aWwgCQoUgl232KkT28J21l6vd3d2dx79sDuTXdtre03Uqlc62tl53Pv/n3PuOffcKxiGwf9z
E//XAL5sk65WR0a6AaDrOjMtKwgCoigiCAKCIAhXa9wvTcAEHhoc5KMjR7jw3nucf+MN67k/GCS4
YQOr77yTpl27kGUZm81mXC0ywlLnQDbwEwcO0P2b3yz4jV5RwZq2Nm5/5BH8fj+SJGGz2b4UiSUR
MAzD0DSNEwcO8MdHHln8d0AcsAUCfOe551h7yy04HA7sdvuSrVEwAV3XjdDgIC/v2cOl06dznqWA
JCAAMvn9MwlEgF1PP8037rsPt9uN3W5fkjUKIqDrujF+4QLP3XQTU0NDOc8UIAa4y8sJtrQQqK/H
7XQSfv99Pn/nHQRRxNB16/0EEAUatm9n76uv4vP5cDgcBZNYNIF84I0MEK20lIZvf5sNe/dSWlqK
1+vFxBEZHubT3/+evqNHmR4dJRWP55BevW0bra+8gt/vx+l0IknSokksikA+8ElgClixYwc3799P
ZWUlTqczbz+x0VFOPvEEQ6dPE/niCwxdt0jUbd1K68svU1xcXBCJBQnMBz5eWkrL/v2s++Y3KSsr
QxQXlxc7f/5zBjo7CQ8NoadSKKQn943t7dyyf79FYjHuNC8BwzAMVVV56Z57+OzNN3PAJ8vK+FZH
B2s2b55X6/nahRMnOP7TnxLOKEXJyO2/+hVf37uXoqIinE6nmfzyksirMjNUvvGjH+WAVwG1vJxb
Dhxg1bp1uFwuMwQWJCu3bWPN3r24SkoAcGbA/OXXv2bws8+IRqOkUqlZGX1RBAzDMHRd51+nTvHe
wYNX7gPTQMO+faxav56SkpIlgTdlU3s7TXv2IGUs6AGioRDH2tuZmJggFouhqirGPCzyEUBVVX7X
1pZzPwI0fP/7bNmzh7KysrzADh8+zP3338++ffs4ePBg3vdEUWTDD3/IsuZmRLsdW4bEpd5eOp99
lnA4jKIoaJqWl8QsAqb2Txw4kDNp40BxXR0bWlspLy/PC+qZZ55hdHQUSZIYHByko6ODxx57bBZw
cy3kq6xk6y9/ib+qCgQBmbQ7nT50iIs9PUSjUZLJJIZhzEliLgKMDQzwp5/9zLqnkQ51W3/xCwKB
AE6nMy+B3t5exsfH6ezspLe3l3g8zuHDh2cBz5bKpibWtrXhLi0FwA3owNsPP8zExATxeBxVVecy
QC4Bc+L+taNjluvU33031dddN6/2BUGgpKSE/v5+JicncweaA3i2tDz8MIHrr0eU0gsQBzDW10ff
yZNEIhESicScrpRDQNd1xgYG+OC3v7XuJQBXIMDWn/yEioqKeUGMjY0hCAKKoszS1MjIyLzfAmx/
4gmKli0DQcBJOmicfOopJicnicfjJoG5LWBq/+Szz+a8oABNe/bgdrvndR1BEKiqqiIejxOLxWYR
CAaDC0alqjVrqGxowFNWhkDalUKff07f3/5mWSFTLFksLAK6rjN+4UKO9lXAUV7O2tbWeaNOthZl
WZ4FHlh0aN3y4IPIHk+6L8AGvNfRQTgctuZCthVEU/u6rnPutddyBk0A9bt3W4us+QY2fbyurg6v
15vTzw033LBoAsHmZoLNzdgcDgBcQKi/n391dTE9PU0ymcyxgkmAVCrF+4cO5WjfFgiwfoGwOTOy
lJWVUVtba1mkUAKCILClvR1veTkA9owVPnzllZy8YFnA1H5/dzfTw8PWgziwcutWfD4fsiwvCNyU
xsZG3G43lZWVVl933XVXQQSq161j+ebNlhXcwBdnzxIOh60lhmkFEUDTNP7++uvWgBrp6mrNHBk3
H/BsAtFoFEdm8GAwyMaNGwsiIAgCjbt346uosKwgAp8cPWolNj1THInmsqHn6FGLQApY1thIxcqV
+Hy+WX6+0ETesWMHLpcLWZZ54IEHCgYvCAKrbrsNyem0JrQEXMzkhKzlBaKu61z+97+ZHhnJmbwr
tm2zJu5CWp8pra2tNDU1sXPnTtrb25dEQBAEvrplC57MXJCB8XPniEQixONxa6Uq6brOQHd3jvvo
QP2ttxZUpMxsL7zwwpK+AyxLXrt9Oxe6uxFEEXvGZc4fO0bVD35AMplM1wuapjGYtbugAp5AgKLq
aquu/W/JTEs37dqFqihWzSADk8PD1jzQNC3tQtmrThUINDbm+H4h0tfXx6OPPsodd9xBW1sbPT09
BQPPluu2b7fWRxIQ6usjFotZBCRN06wZbbZAfT1FRUWWKQtpzz//PMPDw/T09DA1NcXo6CjHjx+f
11Xma95AwHJjEdB0HUVRrhCYuRGrAiXBoFUqFtq6u7sZHx9nJBMUurq65uxnsX0Hm5v56A9/ANIJ
LXrpEolEwgqlUqZQyPlo6OOPlzx5ly1bxnBWQmxpackBW6hSBEC02axrpqasNZGmaek8kN0kYPDj
j1FVdUlz4Mknn6Surs4Cb5aUhYZiUy59+GGOgm3V1WiadiUPCILAV9autV5wAaM9PXQeOWJprBBp
aGjgxIkTpFIpOjs7aWxsXBJwUwZOnUKZmgLS2zl4vUjSlV1XURRFqtavv3IDKAVe/O53efPxx69a
SCxUBs+c4dV77+Vyfz+xy5eBdG3iX7ECm81m9S2JokiwuZmiujrCPT1AuqgOAqcef5zzL77Imt27
uemhhyipqVnYZ5cw8bPbR0eOcOall7g8MEB4aIhEJAJk9qOAwKZNyLJsHpQgRKNRIxQK0XPmDH/+
3vdQQ6G8na/YtIk199zDypYWVmzceFWAT1y8yMWzZzn/9tv0vfsudqeTlKIQHRvL2QSOAqU7d/K1
W2/l2muvpba2Nl3iJhIJIxKJMDw8TO/Zs3zw4x+jTUzkHVCw2fCUleHw+QiuXYu/upqqhgb8VVUY
hkFNczPFweCs7z556y3r+vyxYwiiyD+PHwdBwNB1krEYhqYRC4UwdB2D9LJmmnRtXHz77VTfeCPL
ly+ntraWmpqa9MaaqqqGoiiEQiGGhoYYHBzk00OHiHR2LkqDNlnG7nJhd7nQswqN+ZogiuipFMlY
DDWRgBl5KEF6woqAUFFByc03U756NZWVlSxfvpyamhoCgQAejwdB13VDVVVisRihUIiRkRGGh4cZ
+sc/GO3qIt7VhZGJAle7mVpWuVKDCBng8jXX4N68Gc8111BUVER5eTnV1dVUV1cTCASuFFqGYaDr
upFKpYjH40xMTDA+Ps7Y2Bjj4+OEw2EmenqYPneO1OAgZCWpxTY9S9QMcLhyHGUDRI8HqbISZ309
zoYGnE4nHo8Hv99PaWkpgUCAiooK6/DE4XCkI1HW2a6hqiqKojA9Pc3k5KQl4XCY6elpotFoetuk
v59UKIR66RLToRBJTcMYHUWIRvMSMPO6rbY2/SsIOFatQpYk5NWrkSQJWZZxOBy43W68Xi9+v5+i
oiJKSkooLi7G5/PhdruRZdnads85HzD3hpLJJIqiEIvFiEajRCIRIpEI0WiUWCxGPB631iOKopBK
pazUbi4Oc7Y+snKCzWbDZrMhSRKSJGG3262w6HK5cLlceDwevF4vPp8Pr9eL2+3G5XLhcDjMkxuE
TNibdcBhnv9qmkYqlSKZTJJIJFAUhXg8jqIoKIpCIpGwSCSTSVRVtWSuk3qTiAnebrfngHc4HDid
TpxOJy6Xy7qWZdk6wZzrxCbvCY1JRNd1NE3LAWgSM69ngp+PgCiK1gF3NhFTzHvmO+Y3wCzw8xLI
JpL5tYCZpEyw2f/nWt1ag2UtM0x3Mq1iAp1j+bH0M7KFCGWDnfmbd8AMnuxfUzL/C0rp/wFnFd4n
EQn3XQAAAABJRU5ErkJggg==">
</td>
<td class="content">The volume of traffic, especially during recovery, will interfere
with other services on the same network, especially the latency sensitive Proxmox VE
corosync cluster stack can be affected, resulting in possible loss of cluster
quorum.  Moving the Ceph traffic to dedicated and physical separated networks
will avoid such interference, not only for corosync, but also for the networking
services provided by any virtual guests.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>For estimating your bandwidth needs, you need to take the performance of your
disks into account.. While a single HDD might not saturate a 1 Gb link, multiple
HDD OSDs per node can already saturate 10 Gbps too.
If modern NVMe-attached SSDs are used, a single one can already saturate 10 Gbps
of bandwidth, or more. For such high-performance setups we recommend at least
a 25 Gpbs, while even 40 Gbps or 100+ Gbps might be required to utilize the full
performance potential of the underlying disks.</p></div>
<div class="paragraph">
<p>If unsure, we recommend using three (physical) separate networks for
high-performance setups:
* one very high bandwidth (25+ Gbps) network for Ceph (internal) cluster
  traffic.
* one high bandwidth (10+ Gpbs) network for Ceph (public) traffic between the
  ceph server and ceph client storage traffic. Depending on your needs this can
  also be used to host the virtual guest traffic and the VM live-migration
  traffic.
* one medium bandwidth (1 Gbps) exclusive for the latency sensitive corosync
  cluster communication.</p></div>
<div class="paragraph">
<div class="title">Disks</div><p>When planning the size of your Ceph cluster, it is important to take the
recovery time into consideration. Especially with small clusters, recovery
might take long. It is recommended that you use SSDs instead of HDDs in small
setups to reduce recovery time, minimizing the likelihood of a subsequent
failure event during recovery.</p></div>
<div class="paragraph">
<p>In general, SSDs will provide more IOPS than spinning disks. With this in mind,
in addition to the higher cost, it may make sense to implement a
<a href="#pve_ceph_device_classes">class based</a> separation of pools. Another way to
speed up OSDs is to use a faster disk as a journal or
DB/<strong>W</strong>rite-<strong>A</strong>head-<strong>L</strong>og device, see
<a href="#pve_ceph_osds">creating Ceph OSDs</a>.
If a faster disk is used for multiple OSDs, a proper balance between OSD
and WAL / DB (or journal) disk must be selected, otherwise the faster disk
becomes the bottleneck for all linked OSDs.</p></div>
<div class="paragraph">
<p>Aside from the disk type, Ceph performs best with an evenly sized, and an evenly
distributed amount of disks per node. For example, 4 x 500 GB disks within each
node is better than a mixed setup with a single 1 TB and three 250 GB disk.</p></div>
<div class="paragraph">
<p>You also need to balance OSD count and single OSD capacity. More capacity
allows you to increase storage density, but it also means that a single OSD
failure forces Ceph to recover more data at once.</p></div>
<div class="paragraph">
<div class="title">Avoid RAID</div><p>As Ceph handles data object redundancy and multiple parallel writes to disks
(OSDs) on its own, using a RAID controller normally doesn’t improve
performance or availability. On the contrary, Ceph is designed to handle whole
disks on it’s own, without any abstraction in between. RAID controllers are not
designed for the Ceph workload and may complicate things and sometimes even
reduce performance, as their write and caching algorithms may interfere with
the ones from Ceph.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">Avoid RAID controllers. Use host bus adapter (HBA) instead.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect2">
<h3 id="pve_ceph_install_wizard">
<span>8.4. Initial Ceph Installation &amp; Configuration</span>
 <a class="headerlink" href="#pve_ceph_install_wizard" title="Permalink to this heading"></a>
</h3>
<div class="sect3">
<h4 id="_using_the_web_based_wizard">8.4.1. Using the Web-based Wizard
 <a class="headerlink" href="#_using_the_web_based_wizard" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-node-ceph-install.png">
<img src="images/screenshot/gui-node-ceph-install.png" alt="screenshot/gui-node-ceph-install.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>With Proxmox VE you have the benefit of an easy to use installation wizard
for Ceph. Click on one of your cluster nodes and navigate to the Ceph
section in the menu tree. If Ceph is not already installed, you will see a
prompt offering to do so.</p></div>
<div class="paragraph">
<p>The wizard is divided into multiple sections, where each needs to
finish successfully, in order to use Ceph.</p></div>
<div class="paragraph">
<p>First you need to chose which Ceph version you want to install. Prefer the one
from your other nodes, or the newest if this is the first node you install
Ceph.</p></div>
<div class="paragraph">
<p>After starting the installation, the wizard will download and install all the
required packages from Proxmox VE’s Ceph repository.</p></div>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-node-ceph-install-wizard-step0.png">
<img src="images/screenshot/gui-node-ceph-install-wizard-step0.png" alt="screenshot/gui-node-ceph-install-wizard-step0.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>After finishing the installation step, you will need to create a configuration.
This step is only needed once per cluster, as this configuration is distributed
automatically to all remaining cluster members through Proxmox VE’s clustered
<a href="#chapter_pmxcfs">configuration file system (pmxcfs)</a>.</p></div>
<div class="paragraph">
<p>The configuration step includes the following settings:</p></div>
<div class="ulist" id="pve_ceph_wizard_networks"><ul>
<li>
<p>
<strong>Public Network:</strong> This network will be used for public storage communication
  (e.g., for virtual machines using a Ceph RBD backed disk, or a CephFS mount),
  and communication between the different Ceph services. This setting is
  required.
 <br>
  Separating your Ceph traffic from the Proxmox VE cluster communication (corosync),
  and possible the front-facing (public) networks of your virtual guests, is
  highly recommended. Otherwise, Ceph’s high-bandwidth IO-traffic could cause
  interference with other low-latency dependent services.
</p>
</li>
<li>
<p>
<strong>Cluster Network:</strong> Specify to separate the <a href="#pve_ceph_osds">OSD</a> replication
  and heartbeat traffic as well. This setting is optional.
 <br>
  Using a physically separated network is recommended, as it will relieve the
  Ceph public and the virtual guests network, while also providing a significant
  Ceph performance improvements.
 <br>
  The Ceph cluster network can be configured and moved to another physically
  separated network at a later time.
</p>
</li>
</ul></div>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-node-ceph-install-wizard-step2.png">
<img src="images/screenshot/gui-node-ceph-install-wizard-step2.png" alt="screenshot/gui-node-ceph-install-wizard-step2.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>You have two more options which are considered advanced and therefore should
only changed if you know what you are doing.</p></div>
<div class="ulist"><ul>
<li>
<p>
<strong>Number of replicas</strong>: Defines how often an object is replicated.
</p>
</li>
<li>
<p>
<strong>Minimum replicas</strong>: Defines the minimum number of required replicas for I/O to
  be marked as complete.
</p>
</li>
</ul></div>
<div class="paragraph">
<p>Additionally, you need to choose your first monitor node. This step is required.</p></div>
<div class="paragraph">
<p>That’s it. You should now see a success page as the last step, with further
instructions on how to proceed. Your system is now ready to start using Ceph.
To get started, you will need to create some additional <a href="#pve_ceph_monitors">monitors</a>,
<a href="#pve_ceph_osds">OSDs</a> and at least one <a href="#pve_ceph_pools">pool</a>.</p></div>
<div class="paragraph">
<p>The rest of this chapter will guide you through getting the most out of
your Proxmox VE based Ceph setup. This includes the aforementioned tips and
more, such as <a href="#pveceph_fs">CephFS</a>, which is a helpful addition to your
new Ceph cluster.</p></div>
</div>
<div class="sect3">
<h4 id="pve_ceph_install">8.4.2. CLI Installation of Ceph Packages
 <a class="headerlink" href="#pve_ceph_install" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Alternatively to the the recommended Proxmox VE  Ceph installation wizard available
in the web interface, you can use the following CLI command on each node:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>pveceph install</tt></pre></div></div>
<div class="paragraph">
<p>This sets up an <span class="monospaced">apt</span> package repository in
<span class="monospaced">/etc/apt/sources.list.d/ceph.list</span> and installs the required software.</p></div>
</div>
<div class="sect3">
<h4 id="_initial_ceph_configuration_via_cli">8.4.3. Initial Ceph configuration via CLI
 <a class="headerlink" href="#_initial_ceph_configuration_via_cli" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Use the Proxmox VE Ceph installation wizard (recommended) or run the
following command on one node:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>pveceph init --network <span style="color: #993399">10.10</span><span style="color: #990000">.</span><span style="color: #993399">10.0</span><span style="color: #990000">/</span><span style="color: #993399">24</span></tt></pre></div></div>
<div class="paragraph">
<p>This creates an initial configuration at <span class="monospaced">/etc/pve/ceph.conf</span> with a
dedicated network for Ceph. This file is automatically distributed to
all Proxmox VE nodes, using <a href="#chapter_pmxcfs">pmxcfs</a>. The command also
creates a symbolic link at <span class="monospaced">/etc/ceph/ceph.conf</span>, which points to that file.
Thus, you can simply run Ceph commands without the need to specify a
configuration file.</p></div>
</div>
</div>
<div class="sect2">
<h3 id="pve_ceph_monitors">
<span>8.5. Ceph Monitor</span>
 <a class="headerlink" href="#pve_ceph_monitors" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-ceph-monitor.png">
<img src="images/screenshot/gui-ceph-monitor.png" alt="screenshot/gui-ceph-monitor.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>The Ceph Monitor (MON)
<span class="footnote" data-note="Ceph Monitor <a href=&quot;https://docs.ceph.com/en/quincy/start/intro/&quot;>https://docs.ceph.com/en/quincy/start/intro/</a>">[<a id="_footnoteref_18" href="#_footnote_18" title="View footnote" class="footnote">18</a>]</span>
maintains a master copy of the cluster map. For high availability, you need at
least 3 monitors. One monitor will already be installed if you
used the installation wizard. You won’t need more than 3 monitors, as long
as your cluster is small to medium-sized. Only really large clusters will
require more than this.</p></div>
<div class="sect3">
<h4 id="pveceph_create_mon">8.5.1. Create Monitors
 <a class="headerlink" href="#pveceph_create_mon" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>On each node where you want to place a monitor (three monitors are recommended),
create one by using the <em>Ceph → Monitor</em> tab in the GUI or run:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>pveceph mon create</tt></pre></div></div>
</div>
<div class="sect3">
<h4 id="pveceph_destroy_mon">8.5.2. Destroy Monitors
 <a class="headerlink" href="#pveceph_destroy_mon" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>To remove a Ceph Monitor via the GUI, first select a node in the tree view and
go to the <strong>Ceph → Monitor</strong> panel. Select the MON and click the <strong>Destroy</strong>
button.</p></div>
<div class="paragraph">
<p>To remove a Ceph Monitor via the CLI, first connect to the node on which the MON
is running. Then execute the following command:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>pveceph mon destroy</tt></pre></div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">At least three Monitors are needed for quorum.</td>
</tr></tbody></table>
</div>
</div>
</div>
<div class="sect2">
<h3 id="pve_ceph_manager">
<span>8.6. Ceph Manager</span>
 <a class="headerlink" href="#pve_ceph_manager" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>The Manager daemon runs alongside the monitors. It provides an interface to
monitor the cluster. Since the release of Ceph luminous, at least one ceph-mgr
<span class="footnote" data-note="Ceph Manager <a href=&quot;https://docs.ceph.com/en/quincy/mgr/&quot;>https://docs.ceph.com/en/quincy/mgr/</a>">[<a id="_footnoteref_19" href="#_footnote_19" title="View footnote" class="footnote">19</a>]</span> daemon is
required.</p></div>
<div class="sect3">
<h4 id="pveceph_create_mgr">8.6.1. Create Manager
 <a class="headerlink" href="#pveceph_create_mgr" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Multiple Managers can be installed, but only one Manager is active at any given
time.</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>pveceph mgr create</tt></pre></div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">It is recommended to install the Ceph Manager on the monitor nodes. For
high availability install more then one manager.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="pveceph_destroy_mgr">8.6.2. Destroy Manager
 <a class="headerlink" href="#pveceph_destroy_mgr" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>To remove a Ceph Manager via the GUI, first select a node in the tree view and
go to the <strong>Ceph → Monitor</strong> panel. Select the Manager and click the
<strong>Destroy</strong> button.</p></div>
<div class="paragraph">
<p>To remove a Ceph Monitor via the CLI, first connect to the node on which the
Manager is running. Then execute the following command:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>pveceph mgr destroy</tt></pre></div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">While a manager is not a hard-dependency, it is crucial for a Ceph cluster,
as it handles important features like PG-autoscaling, device health monitoring,
telemetry and more.</td>
</tr></tbody></table>
</div>
</div>
</div>
<div class="sect2">
<h3 id="pve_ceph_osds">
<span>8.7. Ceph OSDs</span>
 <a class="headerlink" href="#pve_ceph_osds" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-ceph-osd-status.png">
<img src="images/screenshot/gui-ceph-osd-status.png" alt="screenshot/gui-ceph-osd-status.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>Ceph <strong>O</strong>bject <strong>S</strong>torage <strong>D</strong>aemons store objects for Ceph over the
network. It is recommended to use one OSD per physical disk.</p></div>
<div class="sect3">
<h4 id="pve_ceph_osd_create">8.7.1. Create OSDs
 <a class="headerlink" href="#pve_ceph_osd_create" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>You can create an OSD either via the Proxmox VE web interface or via the CLI using
<span class="monospaced">pveceph</span>. For example:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>pveceph osd create /dev/sd<span style="color: #990000">[</span>X<span style="color: #990000">]</span></tt></pre></div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">We recommend a Ceph cluster with at least three nodes and at least 12
OSDs, evenly distributed among the nodes.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>If the disk was in use before (for example, for ZFS or as an OSD) you first need
to zap all traces of that usage. To remove the partition table, boot sector and
any other OSD leftover, you can use the following command:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>ceph-volume lvm zap /dev/sd<span style="color: #990000">[</span>X<span style="color: #990000">]</span> --destroy</tt></pre></div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">The above command will destroy all data on the disk!</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<div class="title">Ceph Bluestore</div><p>Starting with the Ceph Kraken release, a new Ceph OSD storage type was
introduced called Bluestore
<span class="footnote" data-note="Ceph Bluestore <a href=&quot;https://ceph.com/community/new-luminous-bluestore/&quot;>https://ceph.com/community/new-luminous-bluestore/</a>">[<a id="_footnoteref_20" href="#_footnote_20" title="View footnote" class="footnote">20</a>]</span>.
This is the default when creating OSDs since Ceph Luminous.</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>pveceph osd create /dev/sd<span style="color: #990000">[</span>X<span style="color: #990000">]</span></tt></pre></div></div>
<div class="paragraph">
<div class="title">Block.db and block.wal</div><p>If you want to use a separate DB/WAL device for your OSDs, you can specify it
through the <em>-db_dev</em> and <em>-wal_dev</em> options. The WAL is placed with the DB, if
not specified separately.</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>pveceph osd create /dev/sd<span style="color: #990000">[</span>X<span style="color: #990000">]</span> -db_dev /dev/sd<span style="color: #990000">[</span>Y<span style="color: #990000">]</span> -wal_dev /dev/sd<span style="color: #990000">[</span>Z<span style="color: #990000">]</span></tt></pre></div></div>
<div class="paragraph">
<p>You can directly choose the size of those with the <em>-db_size</em> and <em>-wal_size</em>
parameters respectively. If they are not given, the following values (in order)
will be used:</p></div>
<div class="ulist"><ul>
<li>
<p>
bluestore_block_{db,wal}_size from Ceph configuration…
</p>
<div class="ulist"><ul>
<li>
<p>
… database, section <em>osd</em>
</p>
</li>
<li>
<p>
… database, section <em>global</em>
</p>
</li>
<li>
<p>
… file, section <em>osd</em>
</p>
</li>
<li>
<p>
… file, section <em>global</em>
</p>
</li>
</ul></div>
</li>
<li>
<p>
10% (DB)/1% (WAL) of OSD size
</p>
</li>
</ul></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">The DB stores BlueStore’s internal metadata, and the WAL is BlueStore’s
internal journal or write-ahead log. It is recommended to use a fast SSD or
NVRAM for better performance.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<div class="title">Ceph Filestore</div><p>Before Ceph Luminous, Filestore was used as the default storage type for Ceph OSDs.
Starting with Ceph Nautilus, Proxmox VE does not support creating such OSDs with
<em>pveceph</em> anymore. If you still want to create filestore OSDs, use
<em>ceph-volume</em> directly.</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>ceph-volume lvm create --filestore --data /dev/sd<span style="color: #990000">[</span>X<span style="color: #990000">]</span> --journal /dev/sd<span style="color: #990000">[</span>Y<span style="color: #990000">]</span></tt></pre></div></div>
</div>
<div class="sect3">
<h4 id="pve_ceph_osd_destroy">8.7.2. Destroy OSDs
 <a class="headerlink" href="#pve_ceph_osd_destroy" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>To remove an OSD via the GUI, first select a Proxmox VE node in the tree view and go
to the <strong>Ceph → OSD</strong> panel. Then select the OSD to destroy and click the <strong>OUT</strong>
button. Once the OSD status has changed from <span class="monospaced">in</span> to <span class="monospaced">out</span>, click the <strong>STOP</strong>
button. Finally, after the status has changed from <span class="monospaced">up</span> to <span class="monospaced">down</span>, select
<strong>Destroy</strong> from the <span class="monospaced">More</span> drop-down menu.</p></div>
<div class="paragraph">
<p>To remove an OSD via the CLI run the following commands.</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>ceph osd out <span style="color: #990000">&lt;</span>ID<span style="color: #990000">&gt;</span>
systemctl stop ceph-osd@<span style="color: #990000">&lt;</span>ID<span style="color: #990000">&gt;.</span>service</tt></pre></div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">The first command instructs Ceph not to include the OSD in the data
distribution. The second command stops the OSD service. Until this time, no
data is lost.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>The following command destroys the OSD. Specify the <em>-cleanup</em> option to
additionally destroy the partition table.</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>pveceph osd destroy <span style="color: #990000">&lt;</span>ID<span style="color: #990000">&gt;</span></tt></pre></div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">The above command will destroy all data on the disk!</td>
</tr></tbody></table>
</div>
</div>
</div>
<div class="sect2">
<h3 id="pve_ceph_pools">
<span>8.8. Ceph Pools</span>
 <a class="headerlink" href="#pve_ceph_pools" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-ceph-pools.png">
<img src="images/screenshot/gui-ceph-pools.png" alt="screenshot/gui-ceph-pools.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>A pool is a logical group for storing objects. It holds a collection of objects,
known as <strong>P</strong>lacement <strong>G</strong>roups (<span class="monospaced">PG</span>, <span class="monospaced">pg_num</span>).</p></div>
<div class="sect3">
<h4 id="_create_and_edit_pools">8.8.1. Create and Edit Pools
 <a class="headerlink" href="#_create_and_edit_pools" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>You can create and edit pools from the command line or the web interface of any
Proxmox VE host under <strong>Ceph → Pools</strong>.</p></div>
<div class="paragraph">
<p>When no options are given, we set a default of <strong>128 PGs</strong>, a <strong>size of 3
replicas</strong> and a <strong>min_size of 2 replicas</strong>, to ensure no data loss occurs if
any OSD fails.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content"><strong>Do not set a min_size of 1</strong>. A replicated pool with min_size of 1
allows I/O on an object when it has only 1 replica, which could lead to data
loss, incomplete PGs or unfound objects.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>It is advised that you either enable the PG-Autoscaler or calculate the PG
number based on your setup. You can find the formula and the PG calculator
<span class="footnote" data-note="PG calculator <a href=&quot;https://web.archive.org/web/20210301111112/http://ceph.com/pgcalc/&quot;>https://web.archive.org/web/20210301111112/http://ceph.com/pgcalc/</a>">[<a id="_footnoteref_21" href="#_footnote_21" title="View footnote" class="footnote">21</a>]</span> online. From Ceph Nautilus
onward, you can change the number of PGs
<span class="footnote" id="_footnote_placement_groups" data-note="Placement Groups <a href=&quot;https://docs.ceph.com/en/quincy/rados/operations/placement-groups/&quot;>https://docs.ceph.com/en/quincy/rados/operations/placement-groups/</a>">[<a id="_footnoteref_22" href="#_footnote_22" title="View footnote" class="footnote">22</a>]</span> after the setup.</p></div>
<div class="paragraph">
<p>The PG autoscaler <span class="footnote" id="_footnote_autoscaler" data-note="Automated Scaling <a href=&quot;https://docs.ceph.com/en/quincy/rados/operations/placement-groups/#automated-scaling&quot;>https://docs.ceph.com/en/quincy/rados/operations/placement-groups/#automated-scaling</a>">[<a id="_footnoteref_23" href="#_footnote_23" title="View footnote" class="footnote">23</a>]</span> can
automatically scale the PG count for a pool in the background. Setting the
<span class="monospaced">Target Size</span> or <span class="monospaced">Target Ratio</span> advanced parameters helps the PG-Autoscaler to
make better decisions.</p></div>
<div class="listingblock">
<div class="title">Example for creating a pool over the CLI</div>
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>pveceph pool create <span style="color: #990000">&lt;</span>pool-name<span style="color: #990000">&gt;</span> --add_storages</tt></pre></div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">If you would also like to automatically define a storage for your
pool, keep the ‘Add as Storage’ checkbox checked in the web interface, or use the
command-line option <em>--add_storages</em> at pool creation.</td>
</tr></tbody></table>
</div>
<div class="sect4">
<h5 id="_pool_options">Pool Options
 <a class="headerlink" href="#_pool_options" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-ceph-pool-create.png">
<img src="images/screenshot/gui-ceph-pool-create.png" alt="screenshot/gui-ceph-pool-create.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>The following options are available on pool creation, and partially also when
editing a pool.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
Name
</dt>
<dd>
<p>
The name of the pool. This must be unique and can’t be changed afterwards.
</p>
</dd>
<dt class="hdlist1">
Size
</dt>
<dd>
<p>
The number of replicas per object. Ceph always tries to have this many
copies of an object. Default: <span class="monospaced">3</span>.
</p>
</dd>
<dt class="hdlist1">
PG Autoscale Mode
</dt>
<dd>
<p>
The automatic PG scaling mode <span class="footnoteref">[<a href="#_footnote_23" title="View footnote" class="footnote">23</a>]</span> of
the pool. If set to <span class="monospaced">warn</span>, it produces a warning message when a pool
has a non-optimal PG count. Default: <span class="monospaced">warn</span>.
</p>
</dd>
<dt class="hdlist1">
Add as Storage
</dt>
<dd>
<p>
Configure a VM or container storage using the new pool.
Default: <span class="monospaced">true</span> (only visible on creation).
</p>
</dd>
</dl></div>
<div class="dlist"><div class="title">Advanced Options</div><dl>
<dt class="hdlist1">
Min. Size
</dt>
<dd>
<p>
The minimum number of replicas per object. Ceph will reject I/O on
the pool if a PG has less than this many replicas. Default: <span class="monospaced">2</span>.
</p>
</dd>
<dt class="hdlist1">
Crush Rule
</dt>
<dd>
<p>
The rule to use for mapping object placement in the cluster. These
rules define how data is placed within the cluster. See
<a href="#pve_ceph_device_classes">Ceph CRUSH &amp; device classes</a> for information on
device-based rules.
</p>
</dd>
<dt class="hdlist1">
# of PGs
</dt>
<dd>
<p>
The number of placement groups <span class="footnoteref">[<a href="#_footnote_22" title="View footnote" class="footnote">22</a>]</span> that
the pool should have at the beginning. Default: <span class="monospaced">128</span>.
</p>
</dd>
<dt class="hdlist1">
Target Ratio
</dt>
<dd>
<p>
The ratio of data that is expected in the pool. The PG
autoscaler uses the ratio relative to other ratio sets. It takes precedence
over the <span class="monospaced">target size</span> if both are set.
</p>
</dd>
<dt class="hdlist1">
Target Size
</dt>
<dd>
<p>
The estimated amount of data expected in the pool. The PG
autoscaler uses this size to estimate the optimal PG count.
</p>
</dd>
<dt class="hdlist1">
Min. # of PGs
</dt>
<dd>
<p>
The minimum number of placement groups. This setting is used to
fine-tune the lower bound of the PG count for that pool. The PG autoscaler
will not merge PGs below this threshold.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p>Further information on Ceph pool handling can be found in the Ceph pool
operation <span class="footnote" data-note="Ceph pool operation
<a href=&quot;https://docs.ceph.com/en/quincy/rados/operations/pools/&quot;>https://docs.ceph.com/en/quincy/rados/operations/pools/</a>">[<a id="_footnoteref_24" href="#_footnote_24" title="View footnote" class="footnote">24</a>]</span>
manual.</p></div>
</div>
</div>
<div class="sect3">
<h4 id="pve_ceph_ec_pools">8.8.2. Erasure Coded Pools
 <a class="headerlink" href="#pve_ceph_ec_pools" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Erasure coding (EC) is a form of ‘forward error correction’ codes that allows
to recover from a certain amount of data loss. Erasure coded pools can offer
more usable space compared to replicated pools, but they do that for the price
of performance.</p></div>
<div class="paragraph">
<p>For comparison: in classic, replicated pools, multiple replicas of the data
are stored (<span class="monospaced">size</span>) while in erasure coded pool, data is split into <span class="monospaced">k</span> data
chunks with additional <span class="monospaced">m</span> coding (checking) chunks. Those coding chunks can be
used to recreate data should data chunks be missing.</p></div>
<div class="paragraph">
<p>The number of coding chunks, <span class="monospaced">m</span>, defines how many OSDs can be lost without
losing any data. The total amount of objects stored is <span class="monospaced">k + m</span>.</p></div>
<div class="sect4">
<h5 id="_creating_ec_pools">Creating EC Pools
 <a class="headerlink" href="#_creating_ec_pools" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Erasure coded (EC) pools can be created with the <span class="monospaced">pveceph</span> CLI tooling.
Planning an EC pool needs to account for the fact, that they work differently
than replicated pools.</p></div>
<div class="paragraph">
<p>The default <span class="monospaced">min_size</span> of an EC pool depends on the <span class="monospaced">m</span> parameter. If <span class="monospaced">m = 1</span>,
the <span class="monospaced">min_size</span> of the EC pool will be <span class="monospaced">k</span>. The <span class="monospaced">min_size</span> will be <span class="monospaced">k + 1</span> if
<span class="monospaced">m &gt; 1</span>. The Ceph documentation recommends a conservative <span class="monospaced">min_size</span> of <span class="monospaced">k + 2</span>
<span class="footnote" data-note="Ceph Erasure Coded Pool Recovery
<a href=&quot;https://docs.ceph.com/en/quincy/rados/operations/erasure-code/#erasure-coded-pool-recovery&quot;>https://docs.ceph.com/en/quincy/rados/operations/erasure-code/#erasure-coded-pool-recovery</a>">[<a id="_footnoteref_25" href="#_footnote_25" title="View footnote" class="footnote">25</a>]</span>.</p></div>
<div class="paragraph">
<p>If there are less than <span class="monospaced">min_size</span> OSDs available, any IO to the pool will be
blocked until there are enough OSDs available again.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">When planning an erasure coded pool, keep an eye on the <span class="monospaced">min_size</span> as it
defines how many OSDs need to be available. Otherwise, IO will be blocked.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>For example, an EC pool with <span class="monospaced">k = 2</span> and <span class="monospaced">m = 1</span> will have <span class="monospaced">size = 3</span>,
<span class="monospaced">min_size = 2</span> and will stay operational if one OSD fails. If the pool is
configured with <span class="monospaced">k = 2</span>, <span class="monospaced">m = 2</span>, it will have a <span class="monospaced">size = 4</span> and <span class="monospaced">min_size = 3</span>
and stay operational if one OSD is lost.</p></div>
<div class="paragraph">
<p>To create a new EC pool, run the following command:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>pveceph pool create <span style="color: #990000">&lt;</span>pool-name<span style="color: #990000">&gt;</span> --erasure-coding <span style="color: #009900">k</span><span style="color: #990000">=</span><span style="color: #993399">2</span><span style="color: #990000">,</span><span style="color: #009900">m</span><span style="color: #990000">=</span><span style="color: #993399">1</span></tt></pre></div></div>
<div class="paragraph">
<p>Optional parameters are <span class="monospaced">failure-domain</span> and <span class="monospaced">device-class</span>. If you
need to change any EC profile settings used by the pool, you will have to
create a new pool with a new profile.</p></div>
<div class="paragraph">
<p>This will create a new EC pool plus the needed replicated pool to store the RBD
omap and other metadata. In the end, there will be a <span class="monospaced">&lt;pool name&gt;-data</span> and
<span class="monospaced">&lt;pool name&gt;-metada</span> pool. The default behavior is to create a matching storage
configuration as well. If that behavior is not wanted, you can disable it by
providing the <span class="monospaced">--add_storages 0</span> parameter.  When configuring the storage
configuration manually, keep in mind that the <span class="monospaced">data-pool</span> parameter needs to be
set. Only then will the EC pool be used to store the data objects. For example:</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">The optional parameters <span class="monospaced">--size</span>, <span class="monospaced">--min_size</span> and <span class="monospaced">--crush_rule</span> will be
used for the replicated metadata pool, but not for the erasure coded data pool.
If you need to change the <span class="monospaced">min_size</span> on the data pool, you can do it later.
The <span class="monospaced">size</span> and <span class="monospaced">crush_rule</span> parameters cannot be changed on erasure coded
pools.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>If there is a need to further customize the EC profile, you can do so by
creating it with the Ceph tools directly <span class="footnote" data-note="Ceph Erasure Code Profile
<a href=&quot;https://docs.ceph.com/en/quincy/rados/operations/erasure-code/#erasure-code-profiles&quot;>https://docs.ceph.com/en/quincy/rados/operations/erasure-code/#erasure-code-profiles</a>">[<a id="_footnoteref_26" href="#_footnote_26" title="View footnote" class="footnote">26</a>]</span>, and
specify the profile to use with the <span class="monospaced">profile</span> parameter.</p></div>
<div class="paragraph">
<p>For example:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>pveceph pool create <span style="color: #990000">&lt;</span>pool-name<span style="color: #990000">&gt;</span> --erasure-coding <span style="color: #009900">profile</span><span style="color: #990000">=&lt;</span>profile-name<span style="color: #990000">&gt;</span></tt></pre></div></div>
</div>
<div class="sect4">
<h5 id="_adding_ec_pools_as_storage">Adding EC Pools as Storage
 <a class="headerlink" href="#_adding_ec_pools_as_storage" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>You can add an already existing EC pool as storage to Proxmox VE. It works the same
way as adding an <span class="monospaced">RBD</span> pool but requires the extra <span class="monospaced">data-pool</span> option.</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>pvesm add rbd <span style="color: #990000">&lt;</span>storage-name<span style="color: #990000">&gt;</span> --pool <span style="color: #990000">&lt;</span>replicated-pool<span style="color: #990000">&gt;</span> --data-pool <span style="color: #990000">&lt;</span>ec-pool<span style="color: #990000">&gt;</span></tt></pre></div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">Do not forget to add the <span class="monospaced">keyring</span> and <span class="monospaced">monhost</span> option for any external
Ceph clusters, not managed by the local Proxmox VE cluster.</td>
</tr></tbody></table>
</div>
</div>
</div>
<div class="sect3">
<h4 id="_destroy_pools">8.8.3. Destroy Pools
 <a class="headerlink" href="#_destroy_pools" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>To destroy a pool via the GUI, select a node in the tree view and go to the
<strong>Ceph → Pools</strong> panel. Select the pool to destroy and click the <strong>Destroy</strong>
button. To confirm the destruction of the pool, you need to enter the pool name.</p></div>
<div class="paragraph">
<p>Run the following command to destroy a pool. Specify the <em>-remove_storages</em> to
also remove the associated storage.</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>pveceph pool destroy <span style="color: #990000">&lt;</span>name<span style="color: #990000">&gt;</span></tt></pre></div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Pool deletion runs in the background and can take some time.
You will notice the data usage in the cluster decreasing throughout this
process.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="_pg_autoscaler">8.8.4. PG Autoscaler
 <a class="headerlink" href="#_pg_autoscaler" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The PG autoscaler allows the cluster to consider the amount of (expected) data
stored in each pool and to choose the appropriate pg_num values automatically.
It is available since Ceph Nautilus.</p></div>
<div class="paragraph">
<p>You may need to activate the PG autoscaler module before adjustments can take
effect.</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>ceph mgr module <span style="font-weight: bold"><span style="color: #0000FF">enable</span></span> pg_autoscaler</tt></pre></div></div>
<div class="paragraph">
<p>The autoscaler is configured on a per pool basis and has the following modes:</p></div>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
warn
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
A health warning is issued if the suggested <span class="monospaced">pg_num</span> value differs too
much from the current value.
</p>
</td>
</tr>
<tr>
<td class="hdlist1">
on
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
The <span class="monospaced">pg_num</span> is adjusted automatically with no need for any manual
interaction.
</p>
</td>
</tr>
<tr>
<td class="hdlist1">
off
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
No automatic <span class="monospaced">pg_num</span> adjustments are made, and no warning will be issued
if the PG count is not optimal.
</p>
</td>
</tr>
</tbody></table></div>
<div class="paragraph">
<p>The scaling factor can be adjusted to facilitate future data storage with the
<span class="monospaced">target_size</span>, <span class="monospaced">target_size_ratio</span> and the <span class="monospaced">pg_num_min</span> options.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">By default, the autoscaler considers tuning the PG count of a pool if
it is off by a factor of 3. This will lead to a considerable shift in data
placement and might introduce a high load on the cluster.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>You can find a more in-depth introduction to the PG autoscaler on Ceph’s Blog -
<a href="https://ceph.io/rados/new-in-nautilus-pg-merging-and-autotuning/">New in
Nautilus: PG merging and autotuning</a>.</p></div>
</div>
</div>
<div class="sect2">
<h3 id="pve_ceph_device_classes">
<span>8.9. Ceph CRUSH &amp; device classes</span>
 <a class="headerlink" href="#pve_ceph_device_classes" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-ceph-config.png">
<img src="images/screenshot/gui-ceph-config.png" alt="screenshot/gui-ceph-config.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>The <span class="footnote" data-note="CRUSH
<a href=&quot;https://ceph.com/wp-content/uploads/2016/08/weil-crush-sc06.pdf&quot;>https://ceph.com/wp-content/uploads/2016/08/weil-crush-sc06.pdf</a>">[<a id="_footnoteref_27" href="#_footnote_27" title="View footnote" class="footnote">27</a>]</span> (<strong>C</strong>ontrolled
<strong>R</strong>eplication <strong>U</strong>nder <strong>S</strong>calable <strong>H</strong>ashing) algorithm is at the
foundation of Ceph.</p></div>
<div class="paragraph">
<p>CRUSH calculates where to store and retrieve data from. This has the
advantage that no central indexing service is needed. CRUSH works using a map of
OSDs, buckets (device locations) and rulesets (data replication) for pools.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Further information can be found in the Ceph documentation, under the
section CRUSH map <span class="footnote" data-note="CRUSH map <a href=&quot;https://docs.ceph.com/en/quincy/rados/operations/crush-map/&quot;>https://docs.ceph.com/en/quincy/rados/operations/crush-map/</a>">[<a id="_footnoteref_28" href="#_footnote_28" title="View footnote" class="footnote">28</a>]</span>.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>This map can be altered to reflect different replication hierarchies. The object
replicas can be separated (e.g., failure domains), while maintaining the desired
distribution.</p></div>
<div class="paragraph">
<p>A common configuration is to use different classes of disks for different Ceph
pools.  For this reason, Ceph introduced device classes with luminous, to
accommodate the need for easy ruleset generation.</p></div>
<div class="paragraph">
<p>The device classes can be seen in the <em>ceph osd tree</em> output. These classes
represent their own root bucket, which can be seen with the below command.</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>ceph osd crush tree --show-shadow</tt></pre></div></div>
<div class="paragraph">
<p>Example output form the above command:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>ID  CLASS WEIGHT  TYPE NAME
-<span style="color: #993399">16</span>  nvme <span style="color: #993399">2.18307</span> root default<span style="color: #990000">~</span>nvme
-<span style="color: #993399">13</span>  nvme <span style="color: #993399">0.72769</span>     host sumi1<span style="color: #990000">~</span>nvme
 <span style="color: #993399">12</span>  nvme <span style="color: #993399">0.72769</span>         osd<span style="color: #990000">.</span><span style="color: #993399">12</span>
-<span style="color: #993399">14</span>  nvme <span style="color: #993399">0.72769</span>     host sumi2<span style="color: #990000">~</span>nvme
 <span style="color: #993399">13</span>  nvme <span style="color: #993399">0.72769</span>         osd<span style="color: #990000">.</span><span style="color: #993399">13</span>
-<span style="color: #993399">15</span>  nvme <span style="color: #993399">0.72769</span>     host sumi3<span style="color: #990000">~</span>nvme
 <span style="color: #993399">14</span>  nvme <span style="color: #993399">0.72769</span>         osd<span style="color: #990000">.</span><span style="color: #993399">14</span>
 -<span style="color: #993399">1</span>       <span style="color: #993399">7.70544</span> root default
 -<span style="color: #993399">3</span>       <span style="color: #993399">2.56848</span>     host sumi1
 <span style="color: #993399">12</span>  nvme <span style="color: #993399">0.72769</span>         osd<span style="color: #990000">.</span><span style="color: #993399">12</span>
 -<span style="color: #993399">5</span>       <span style="color: #993399">2.56848</span>     host sumi2
 <span style="color: #993399">13</span>  nvme <span style="color: #993399">0.72769</span>         osd<span style="color: #990000">.</span><span style="color: #993399">13</span>
 -<span style="color: #993399">7</span>       <span style="color: #993399">2.56848</span>     host sumi3
 <span style="color: #993399">14</span>  nvme <span style="color: #993399">0.72769</span>         osd<span style="color: #990000">.</span><span style="color: #993399">14</span></tt></pre></div></div>
<div class="paragraph">
<p>To instruct a pool to only distribute objects on a specific device class, you
first need to create a ruleset for the device class:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>ceph osd crush rule create-replicated <span style="color: #990000">&lt;</span>rule-name<span style="color: #990000">&gt;</span> <span style="color: #990000">&lt;</span>root<span style="color: #990000">&gt;</span> <span style="color: #990000">&lt;</span>failure-domain<span style="color: #990000">&gt;</span> <span style="color: #990000">&lt;</span>class<span style="color: #990000">&gt;</span></tt></pre></div></div>
<table class="tableblock frame-none grid-none" style="
margin-left:0; margin-right:auto;
width:100%;
">
<colgroup><col style="width:30%;">
<col style="width:70%;">
</colgroup><tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">&lt;rule-name&gt;</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">name of the rule, to connect with a pool (seen in GUI &amp; CLI)</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">&lt;root&gt;</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">which crush root it should belong to (default Ceph root "default")</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">&lt;failure-domain&gt;</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">at which failure-domain the objects should be distributed (usually host)</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">&lt;class&gt;</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">what type of OSD backing store to use (e.g., nvme, ssd, hdd)</p></td>
</tr>
</tbody>
</table>
<div class="paragraph">
<p>Once the rule is in the CRUSH map, you can tell a pool to use the ruleset.</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>ceph osd pool <span style="font-weight: bold"><span style="color: #0000FF">set</span></span> <span style="color: #990000">&lt;</span>pool-name<span style="color: #990000">&gt;</span> crush_rule <span style="color: #990000">&lt;</span>rule-name<span style="color: #990000">&gt;</span></tt></pre></div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">If the pool already contains objects, these must be moved accordingly.
Depending on your setup, this may introduce a big performance impact on your
cluster. As an alternative, you can create a new pool and move disks separately.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect2">
<h3 id="_ceph_client">
<span>8.10. Ceph Client</span>
 <a class="headerlink" href="#_ceph_client" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-ceph-log.png">
<img src="images/screenshot/gui-ceph-log.png" alt="screenshot/gui-ceph-log.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>Following the setup from the previous sections, you can configure Proxmox VE to use
such pools to store VM and Container images. Simply use the GUI to add a new
<span class="monospaced">RBD</span> storage (see section
<a href="#ceph_rados_block_devices">Ceph RADOS Block Devices (RBD)</a>).</p></div>
<div class="paragraph">
<p>You also need to copy the keyring to a predefined location for an external Ceph
cluster. If Ceph is installed on the Proxmox nodes itself, then this will be
done automatically.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">The filename needs to be <span class="monospaced">&lt;storage_id&gt; + `.keyring</span>, where <span class="monospaced">&lt;storage_id&gt;</span> is
the expression after <em>rbd:</em> in <span class="monospaced">/etc/pve/storage.cfg</span>. In the following example,
<span class="monospaced">my-ceph-storage</span> is the <span class="monospaced">&lt;storage_id&gt;</span>:</td>
</tr></tbody></table>
</div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>mkdir /etc/pve/priv/ceph
cp /etc/ceph/ceph<span style="color: #990000">.</span>client<span style="color: #990000">.</span>admin<span style="color: #990000">.</span>keyring /etc/pve/priv/ceph/my-ceph-storage<span style="color: #990000">.</span>keyring</tt></pre></div></div>
</div>
<div class="sect2">
<h3 id="pveceph_fs">
<span>8.11. CephFS</span>
 <a class="headerlink" href="#pveceph_fs" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Ceph also provides a filesystem, which runs on top of the same object storage as
RADOS block devices do. A <strong>M</strong>eta<strong>d</strong>ata <strong>S</strong>erver (<span class="monospaced">MDS</span>) is used to map the
RADOS backed objects to files and directories, allowing Ceph to provide a
POSIX-compliant, replicated filesystem. This allows you to easily configure a
clustered, highly available, shared filesystem. Ceph’s Metadata Servers
guarantee that files are evenly distributed over the entire Ceph cluster. As a
result, even cases of high load will not overwhelm a single host, which can be
an issue with traditional shared filesystem approaches, for example <span class="monospaced">NFS</span>.</p></div>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-node-ceph-cephfs-panel.png">
<img src="images/screenshot/gui-node-ceph-cephfs-panel.png" alt="screenshot/gui-node-ceph-cephfs-panel.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>Proxmox VE supports both creating a hyper-converged CephFS and using an existing
<a href="#storage_cephfs">CephFS as storage</a> to save backups, ISO files, and container
templates.</p></div>
<div class="sect3">
<h4 id="pveceph_fs_mds">8.11.1. Metadata Server (MDS)
 <a class="headerlink" href="#pveceph_fs_mds" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>CephFS needs at least one Metadata Server to be configured and running, in order
to function. You can create an MDS through the Proxmox VE web GUI’s <span class="monospaced">Node
-&gt; CephFS</span> panel or from the command line with:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>pveceph mds create</pre>
</div></div>
<div class="paragraph">
<p>Multiple metadata servers can be created in a cluster, but with the default
settings, only one can be active at a time. If an MDS or its node becomes
unresponsive (or crashes), another <span class="monospaced">standby</span> MDS will get promoted to <span class="monospaced">active</span>.
You can speed up the handover between the active and standby MDS by using
the <em>hotstandby</em> parameter option on creation, or if you have already created it
you may set/add:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>mds standby replay = true</pre>
</div></div>
<div class="paragraph">
<p>in the respective MDS section of <span class="monospaced">/etc/pve/ceph.conf</span>. With this enabled, the
specified MDS will remain in a <span class="monospaced">warm</span> state, polling the active one, so that it
can take over faster in case of any issues.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">This active polling will have an additional performance impact on your
system and the active <span class="monospaced">MDS</span>.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<div class="title">Multiple Active MDS</div><p>Since Luminous (12.2.x) you can have multiple active metadata servers
running at once, but this is normally only useful if you have a high amount of
clients running in parallel. Otherwise the <span class="monospaced">MDS</span> is rarely the bottleneck in a
system. If you want to set this up, please refer to the Ceph documentation.
<span class="footnote" data-note="Configuring multiple active MDS daemons
<a href=&quot;https://docs.ceph.com/en/quincy/cephfs/multimds/&quot;>https://docs.ceph.com/en/quincy/cephfs/multimds/</a>">[<a id="_footnoteref_29" href="#_footnote_29" title="View footnote" class="footnote">29</a>]</span></p></div>
</div>
<div class="sect3">
<h4 id="pveceph_fs_create">8.11.2. Create CephFS
 <a class="headerlink" href="#pveceph_fs_create" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>With Proxmox VE’s integration of CephFS, you can easily create a CephFS using the
web interface, CLI or an external API interface. Some prerequisites are required
for this to work:</p></div>
<div class="ulist"><div class="title">Prerequisites for a successful CephFS setup:</div><ul>
<li>
<p>
<a href="#pve_ceph_install">Install Ceph packages</a> - if this was already done some
time ago, you may want to rerun it on an up-to-date system to
ensure that all CephFS related packages get installed.
</p>
</li>
<li>
<p>
<a href="#pve_ceph_monitors">Setup Monitors</a>
</p>
</li>
<li>
<p>
<a href="#pve_ceph_monitors">Setup your OSDs</a>
</p>
</li>
<li>
<p>
<a href="#pveceph_fs_mds">Setup at least one MDS</a>
</p>
</li>
</ul></div>
<div class="paragraph">
<p>After this is complete, you can simply create a CephFS through
either the Web GUI’s <span class="monospaced">Node -&gt; CephFS</span> panel or the command-line tool <span class="monospaced">pveceph</span>,
for example:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>pveceph fs create --pg_num 128 --add-storage</pre>
</div></div>
<div class="paragraph">
<p>This creates a CephFS named <em>cephfs</em>, using a pool for its data named
<em>cephfs_data</em> with <em>128</em> placement groups and a pool for its metadata named
<em>cephfs_metadata</em> with one quarter of the data pool’s placement groups (<span class="monospaced">32</span>).
Check the <a href="#pve_ceph_pools">Proxmox VE managed Ceph pool chapter</a> or visit the
Ceph documentation for more information regarding an appropriate placement group
number (<span class="monospaced">pg_num</span>) for your setup <span class="footnoteref">[<a href="#_footnote_22" title="View footnote" class="footnote">22</a>]</span>.
Additionally, the <em>--add-storage</em> parameter will add the CephFS to the Proxmox VE
storage configuration after it has been created successfully.</p></div>
</div>
<div class="sect3">
<h4 id="_destroy_cephfs">8.11.3. Destroy CephFS
 <a class="headerlink" href="#_destroy_cephfs" title="Permalink to this heading"></a>
</h4>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">Destroying a CephFS will render all of its data unusable. This cannot be
undone!</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>To completely and gracefully remove a CephFS, the following steps are
necessary:</p></div>
<div class="ulist"><ul>
<li>
<p>
Disconnect every non-Proxmox VE client (e.g. unmount the CephFS in guests).
</p>
</li>
<li>
<p>
Disable all related CephFS Proxmox VE storage entries (to prevent it from being
  automatically mounted).
</p>
</li>
<li>
<p>
Remove all used resources from guests (e.g. ISOs) that are on the CephFS you
  want to destroy.
</p>
</li>
<li>
<p>
Unmount the CephFS storages on all cluster nodes manually with
</p>
<div class="listingblock">
<div class="content monospaced">
<pre>umount /mnt/pve/&lt;STORAGE-NAME&gt;</pre>
</div></div>
<div class="paragraph">
<p>Where <span class="monospaced">&lt;STORAGE-NAME&gt;</span> is the name of the CephFS storage in your Proxmox VE.</p></div>
</li>
<li>
<p>
Now make sure that no metadata server (<span class="monospaced">MDS</span>) is running for that CephFS,
  either by stopping or destroying them. This can be done through the web
  interface or via the command-line interface, for the latter you would issue
  the following command:
</p>
<div class="listingblock">
<div class="content monospaced">
<pre>pveceph stop --service mds.NAME</pre>
</div></div>
<div class="paragraph">
<p>to stop them, or</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>pveceph mds destroy NAME</pre>
</div></div>
<div class="paragraph">
<p>to destroy them.</p></div>
<div class="paragraph">
<p>Note that standby servers will automatically be promoted to active when an
active <span class="monospaced">MDS</span> is stopped or removed, so it is best to first stop all standby
servers.</p></div>
</li>
<li>
<p>
Now you can destroy the CephFS with
</p>
<div class="listingblock">
<div class="content monospaced">
<pre>pveceph fs destroy NAME --remove-storages --remove-pools</pre>
</div></div>
<div class="paragraph">
<p>This will automatically destroy the underlying Ceph pools as well as remove
the storages from pve config.</p></div>
</li>
</ul></div>
<div class="paragraph">
<p>After these steps, the CephFS should be completely removed and if you have
other CephFS instances, the stopped metadata servers can be started again
to act as standbys.</p></div>
</div>
</div>
<div class="sect2">
<h3 id="_ceph_maintenance">
<span>8.12. Ceph maintenance</span>
 <a class="headerlink" href="#_ceph_maintenance" title="Permalink to this heading"></a>
</h3>
<div class="sect3">
<h4 id="_replace_osds">8.12.1. Replace OSDs
 <a class="headerlink" href="#_replace_osds" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>One of the most common maintenance tasks in Ceph is to replace the disk of an
OSD. If a disk is already in a failed state, then you can go ahead and run
through the steps in <a href="#pve_ceph_osd_destroy">Destroy OSDs</a>. Ceph will recreate
those copies on the remaining OSDs if possible. This rebalancing will start as
soon as an OSD failure is detected or an OSD was actively stopped.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">With the default size/min_size (3/2) of a pool, recovery only starts when
‘size + 1` nodes are available. The reason for this is that the Ceph object
balancer <a href="#pve_ceph_device_classes">CRUSH</a> defaults to a full node as
`failure domain’.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>To replace a functioning disk from the GUI, go through the steps in
<a href="#pve_ceph_osd_destroy">Destroy OSDs</a>. The only addition is to wait until
the cluster shows <em>HEALTH_OK</em> before stopping the OSD to destroy it.</p></div>
<div class="paragraph">
<p>On the command line, use the following commands:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>ceph osd out osd.&lt;id&gt;</pre>
</div></div>
<div class="paragraph">
<p>You can check with the command below if the OSD can be safely removed.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>ceph osd safe-to-destroy osd.&lt;id&gt;</pre>
</div></div>
<div class="paragraph">
<p>Once the above check tells you that it is safe to remove the OSD, you can
continue with the following commands:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>systemctl stop ceph-osd@&lt;id&gt;.service
pveceph osd destroy &lt;id&gt;</pre>
</div></div>
<div class="paragraph">
<p>Replace the old disk with the new one and use the same procedure as described
in <a href="#pve_ceph_osd_create">Create OSDs</a>.</p></div>
</div>
<div class="sect3">
<h4 id="_trim_discard">8.12.2. Trim/Discard
 <a class="headerlink" href="#_trim_discard" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>It is good practice to run <em>fstrim</em> (discard) regularly on VMs and containers.
This releases data blocks that the filesystem isn’t using anymore. It reduces
data usage and resource load. Most modern operating systems issue such discard
commands to their disks regularly. You only need to ensure that the Virtual
Machines enable the <a href="#qm_hard_disk_discard">disk discard option</a>.</p></div>
</div>
<div class="sect3">
<h4 id="pveceph_scrub">8.12.3. Scrub &amp; Deep Scrub
 <a class="headerlink" href="#pveceph_scrub" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Ceph ensures data integrity by <em>scrubbing</em> placement groups. Ceph checks every
object in a PG for its health. There are two forms of Scrubbing, daily
cheap metadata checks and weekly deep data checks. The weekly deep scrub reads
the objects and uses checksums to ensure data integrity. If a running scrub
interferes with business (performance) needs, you can adjust the time when
scrubs <span class="footnote" data-note="Ceph scrubbing <a href=&quot;https://docs.ceph.com/en/quincy/rados/configuration/osd-config-ref/#scrubbing&quot;>https://docs.ceph.com/en/quincy/rados/configuration/osd-config-ref/#scrubbing</a>">[<a id="_footnoteref_30" href="#_footnote_30" title="View footnote" class="footnote">30</a>]</span>
are executed.</p></div>
</div>
<div class="sect3">
<h4 id="pveceph_shutdown">8.12.4. Shutdown Proxmox VE + Ceph HCI cluster
 <a class="headerlink" href="#pveceph_shutdown" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>To shut down the whole Proxmox VE + Ceph cluster, first stop all Ceph clients. These
will mainly be VMs and containers. If you have additional clients that might
access a Ceph FS or an installed RADOS GW, stop these as well.
Highly available guests will switch their state to <em>stopped</em> when powered down
via the Proxmox VE tooling.</p></div>
<div class="paragraph">
<p>Once all clients, VMs and containers are off or not accessing the Ceph cluster
anymore, verify that the Ceph cluster is in a healthy state. Either via the Web UI
or the CLI:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>ceph -s</pre>
</div></div>
<div class="paragraph">
<p>To disable all self-healing actions, and to pause any client IO in the Ceph
cluster, enable the following OSD flags in the <strong>Ceph → OSD</strong> panel or via the
CLI:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>ceph osd set noout
ceph osd set norecover
ceph osd set norebalance
ceph osd set nobackfill
ceph osd set nodown
ceph osd set pause</pre>
</div></div>
<div class="paragraph">
<p>Start powering down your nodes without a monitor (MON). After these nodes are
down, continue by shutting down nodes with monitors on them.</p></div>
<div class="paragraph">
<p>When powering on the cluster, start the nodes with monitors (MONs) first. Once
all nodes are up and running, confirm that all Ceph services are up and running
before you unset the OSD flags again:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>ceph osd unset pause
ceph osd unset nodown
ceph osd unset nobackfill
ceph osd unset norebalance
ceph osd unset norecover
ceph osd unset noout</pre>
</div></div>
<div class="paragraph">
<p>You can now start up the guests. Highly available guests will change their state
to <em>started</em> when they power on.</p></div>
</div>
</div>
<div class="sect2">
<h3 id="_ceph_monitoring_and_troubleshooting">
<span>8.13. Ceph Monitoring and Troubleshooting</span>
 <a class="headerlink" href="#_ceph_monitoring_and_troubleshooting" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>It is important to continuously monitor the health of a Ceph deployment from the
beginning, either by using the Ceph tools or by accessing
the status through the Proxmox VE <a href="api-viewer/index.html">API</a>.</p></div>
<div class="paragraph">
<p>The following Ceph commands can be used to see if the cluster is healthy
(<em>HEALTH_OK</em>), if there are warnings (<em>HEALTH_WARN</em>), or even errors
(<em>HEALTH_ERR</em>). If the cluster is in an unhealthy state, the status commands
below will also give you an overview of the current events and actions to take.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># single time output
pve# ceph -s
# continuously output status changes (press CTRL+C to stop)
pve# ceph -w</pre>
</div></div>
<div class="paragraph">
<p>To get a more detailed view, every Ceph service has a log file under
<span class="monospaced">/var/log/ceph/</span>. If more detail is required, the log level can be
adjusted <span class="footnote" data-note="Ceph log and debugging <a href=&quot;https://docs.ceph.com/en/quincy/rados/troubleshooting/log-and-debug/&quot;>https://docs.ceph.com/en/quincy/rados/troubleshooting/log-and-debug/</a>">[<a id="_footnoteref_31" href="#_footnote_31" title="View footnote" class="footnote">31</a>]</span>.</p></div>
<div class="paragraph">
<p>You can find more information about troubleshooting
<span class="footnote" data-note="Ceph troubleshooting <a href=&quot;https://docs.ceph.com/en/quincy/rados/troubleshooting/&quot;>https://docs.ceph.com/en/quincy/rados/troubleshooting/</a>">[<a id="_footnoteref_32" href="#_footnote_32" title="View footnote" class="footnote">32</a>]</span>
a Ceph cluster on the official website.</p></div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="chapter_pvesr">
9. Storage Replication
 <a class="headerlink" href="#chapter_pvesr" title="Permalink to this heading"></a>
</h2>
<div class="sectionbody">
<div class="paragraph">
<p>The <span class="monospaced">pvesr</span> command-line tool manages the Proxmox VE storage replication
framework. Storage replication brings redundancy for guests using
local storage and reduces migration time.</p></div>
<div class="paragraph">
<p>It replicates guest volumes to another node so that all data is available
without using shared storage. Replication uses snapshots to minimize traffic
sent over the network. Therefore, new data is sent only incrementally after
the initial full sync. In the case of a node failure, your guest data is
still available on the replicated node.</p></div>
<div class="paragraph">
<p>The replication is done automatically in configurable intervals.
The minimum replication interval is one minute, and the maximal interval
once a week. The format used to specify those intervals is a subset of
<span class="monospaced">systemd</span> calendar events, see
<a href="#pvesr_schedule_time_format">Schedule Format</a> section:</p></div>
<div class="paragraph">
<p>It is possible to replicate a guest to multiple target nodes,
but not twice to the same target node.</p></div>
<div class="paragraph">
<p>Each replications bandwidth can be limited, to avoid overloading a storage
or server.</p></div>
<div class="paragraph">
<p>Only changes since the last replication (so-called <span class="monospaced">deltas</span>) need to be
transferred if the guest is migrated to a node to which it already is
replicated. This reduces the time needed significantly. The replication
direction automatically switches if you migrate a guest to the replication
target node.</p></div>
<div class="paragraph">
<p>For example: VM100 is currently on <span class="monospaced">nodeA</span> and gets replicated to <span class="monospaced">nodeB</span>.
You migrate it to <span class="monospaced">nodeB</span>, so now it gets automatically replicated back from
<span class="monospaced">nodeB</span> to <span class="monospaced">nodeA</span>.</p></div>
<div class="paragraph">
<p>If you migrate to a node where the guest is not replicated, the whole disk
data must send over. After the migration, the replication job continues to
replicate this guest to the configured nodes.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Important" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAALa0lEQVRogdWZa2wc1RXHfzM7O/te
P9e1vXHSmEdjx3YeDkkaF6REKRRCEDSEFNkRjdSWSsgC2iqoRWqLQKiqIBg1NOQDiMeHtkQIQkRR
S9S4aWwgCQoUgl232KkT28J21l6vd3d2dx79sDuTXdtre03Uqlc62tl53Pv/n3PuOffcKxiGwf9z
E//XAL5sk65WR0a6AaDrOjMtKwgCoigiCAKCIAhXa9wvTcAEHhoc5KMjR7jw3nucf+MN67k/GCS4
YQOr77yTpl27kGUZm81mXC0ywlLnQDbwEwcO0P2b3yz4jV5RwZq2Nm5/5BH8fj+SJGGz2b4UiSUR
MAzD0DSNEwcO8MdHHln8d0AcsAUCfOe551h7yy04HA7sdvuSrVEwAV3XjdDgIC/v2cOl06dznqWA
JCAAMvn9MwlEgF1PP8037rsPt9uN3W5fkjUKIqDrujF+4QLP3XQTU0NDOc8UIAa4y8sJtrQQqK/H
7XQSfv99Pn/nHQRRxNB16/0EEAUatm9n76uv4vP5cDgcBZNYNIF84I0MEK20lIZvf5sNe/dSWlqK
1+vFxBEZHubT3/+evqNHmR4dJRWP55BevW0bra+8gt/vx+l0IknSokksikA+8ElgClixYwc3799P
ZWUlTqczbz+x0VFOPvEEQ6dPE/niCwxdt0jUbd1K68svU1xcXBCJBQnMBz5eWkrL/v2s++Y3KSsr
QxQXlxc7f/5zBjo7CQ8NoadSKKQn943t7dyyf79FYjHuNC8BwzAMVVV56Z57+OzNN3PAJ8vK+FZH
B2s2b55X6/nahRMnOP7TnxLOKEXJyO2/+hVf37uXoqIinE6nmfzyksirMjNUvvGjH+WAVwG1vJxb
Dhxg1bp1uFwuMwQWJCu3bWPN3r24SkoAcGbA/OXXv2bws8+IRqOkUqlZGX1RBAzDMHRd51+nTvHe
wYNX7gPTQMO+faxav56SkpIlgTdlU3s7TXv2IGUs6AGioRDH2tuZmJggFouhqirGPCzyEUBVVX7X
1pZzPwI0fP/7bNmzh7KysrzADh8+zP3338++ffs4ePBg3vdEUWTDD3/IsuZmRLsdW4bEpd5eOp99
lnA4jKIoaJqWl8QsAqb2Txw4kDNp40BxXR0bWlspLy/PC+qZZ55hdHQUSZIYHByko6ODxx57bBZw
cy3kq6xk6y9/ib+qCgQBmbQ7nT50iIs9PUSjUZLJJIZhzEliLgKMDQzwp5/9zLqnkQ51W3/xCwKB
AE6nMy+B3t5exsfH6ezspLe3l3g8zuHDh2cBz5bKpibWtrXhLi0FwA3owNsPP8zExATxeBxVVecy
QC4Bc+L+taNjluvU33031dddN6/2BUGgpKSE/v5+JicncweaA3i2tDz8MIHrr0eU0gsQBzDW10ff
yZNEIhESicScrpRDQNd1xgYG+OC3v7XuJQBXIMDWn/yEioqKeUGMjY0hCAKKoszS1MjIyLzfAmx/
4gmKli0DQcBJOmicfOopJicnicfjJoG5LWBq/+Szz+a8oABNe/bgdrvndR1BEKiqqiIejxOLxWYR
CAaDC0alqjVrqGxowFNWhkDalUKff07f3/5mWSFTLFksLAK6rjN+4UKO9lXAUV7O2tbWeaNOthZl
WZ4FHlh0aN3y4IPIHk+6L8AGvNfRQTgctuZCthVEU/u6rnPutddyBk0A9bt3W4us+QY2fbyurg6v
15vTzw033LBoAsHmZoLNzdgcDgBcQKi/n391dTE9PU0ymcyxgkmAVCrF+4cO5WjfFgiwfoGwOTOy
lJWVUVtba1mkUAKCILClvR1veTkA9owVPnzllZy8YFnA1H5/dzfTw8PWgziwcutWfD4fsiwvCNyU
xsZG3G43lZWVVl933XVXQQSq161j+ebNlhXcwBdnzxIOh60lhmkFEUDTNP7++uvWgBrp6mrNHBk3
H/BsAtFoFEdm8GAwyMaNGwsiIAgCjbt346uosKwgAp8cPWolNj1THInmsqHn6FGLQApY1thIxcqV
+Hy+WX6+0ETesWMHLpcLWZZ54IEHCgYvCAKrbrsNyem0JrQEXMzkhKzlBaKu61z+97+ZHhnJmbwr
tm2zJu5CWp8pra2tNDU1sXPnTtrb25dEQBAEvrplC57MXJCB8XPniEQixONxa6Uq6brOQHd3jvvo
QP2ttxZUpMxsL7zwwpK+AyxLXrt9Oxe6uxFEEXvGZc4fO0bVD35AMplM1wuapjGYtbugAp5AgKLq
aquu/W/JTEs37dqFqihWzSADk8PD1jzQNC3tQtmrThUINDbm+H4h0tfXx6OPPsodd9xBW1sbPT09
BQPPluu2b7fWRxIQ6usjFotZBCRN06wZbbZAfT1FRUWWKQtpzz//PMPDw/T09DA1NcXo6CjHjx+f
11Xma95AwHJjEdB0HUVRrhCYuRGrAiXBoFUqFtq6u7sZHx9nJBMUurq65uxnsX0Hm5v56A9/ANIJ
LXrpEolEwgqlUqZQyPlo6OOPlzx5ly1bxnBWQmxpackBW6hSBEC02axrpqasNZGmaek8kN0kYPDj
j1FVdUlz4Mknn6Surs4Cb5aUhYZiUy59+GGOgm3V1WiadiUPCILAV9autV5wAaM9PXQeOWJprBBp
aGjgxIkTpFIpOjs7aWxsXBJwUwZOnUKZmgLS2zl4vUjSlV1XURRFqtavv3IDKAVe/O53efPxx69a
SCxUBs+c4dV77+Vyfz+xy5eBdG3iX7ECm81m9S2JokiwuZmiujrCPT1AuqgOAqcef5zzL77Imt27
uemhhyipqVnYZ5cw8bPbR0eOcOall7g8MEB4aIhEJAJk9qOAwKZNyLJsHpQgRKNRIxQK0XPmDH/+
3vdQQ6G8na/YtIk199zDypYWVmzceFWAT1y8yMWzZzn/9tv0vfsudqeTlKIQHRvL2QSOAqU7d/K1
W2/l2muvpba2Nl3iJhIJIxKJMDw8TO/Zs3zw4x+jTUzkHVCw2fCUleHw+QiuXYu/upqqhgb8VVUY
hkFNczPFweCs7z556y3r+vyxYwiiyD+PHwdBwNB1krEYhqYRC4UwdB2D9LJmmnRtXHz77VTfeCPL
ly+ntraWmpqa9MaaqqqGoiiEQiGGhoYYHBzk00OHiHR2LkqDNlnG7nJhd7nQswqN+ZogiuipFMlY
DDWRgBl5KEF6woqAUFFByc03U756NZWVlSxfvpyamhoCgQAejwdB13VDVVVisRihUIiRkRGGh4cZ
+sc/GO3qIt7VhZGJAle7mVpWuVKDCBng8jXX4N68Gc8111BUVER5eTnV1dVUV1cTCASuFFqGYaDr
upFKpYjH40xMTDA+Ps7Y2Bjj4+OEw2EmenqYPneO1OAgZCWpxTY9S9QMcLhyHGUDRI8HqbISZ309
zoYGnE4nHo8Hv99PaWkpgUCAiooK6/DE4XCkI1HW2a6hqiqKojA9Pc3k5KQl4XCY6elpotFoetuk
v59UKIR66RLToRBJTcMYHUWIRvMSMPO6rbY2/SsIOFatQpYk5NWrkSQJWZZxOBy43W68Xi9+v5+i
oiJKSkooLi7G5/PhdruRZdnads85HzD3hpLJJIqiEIvFiEajRCIRIpEI0WiUWCxGPB631iOKopBK
pazUbi4Oc7Y+snKCzWbDZrMhSRKSJGG3262w6HK5cLlceDwevF4vPp8Pr9eL2+3G5XLhcDjMkxuE
TNibdcBhnv9qmkYqlSKZTJJIJFAUhXg8jqIoKIpCIpGwSCSTSVRVtWSuk3qTiAnebrfngHc4HDid
TpxOJy6Xy7qWZdk6wZzrxCbvCY1JRNd1NE3LAWgSM69ngp+PgCiK1gF3NhFTzHvmO+Y3wCzw8xLI
JpL5tYCZpEyw2f/nWt1ag2UtM0x3Mq1iAp1j+bH0M7KFCGWDnfmbd8AMnuxfUzL/C0rp/wFnFd4n
EQn3XQAAAABJRU5ErkJggg==">
</td>
<td class="content">
<div class="paragraph">
<p>High-Availability is allowed in combination with storage replication, but there
may be some data loss between the last synced time and the time a node failed.</p></div>
</td>
</tr></tbody></table>
</div>
<div class="sect2">
<h3 id="_supported_storage_types">
<span>9.1. Supported Storage Types</span>
 <a class="headerlink" href="#_supported_storage_types" title="Permalink to this heading"></a>
</h3>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<caption class="title">Table 17. Storage Types</caption>
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Description    </th>
<th class="tableblock halign-left valign-top">Plugin type </th>
<th class="tableblock halign-left valign-top">Snapshots</th>
<th class="tableblock halign-left valign-top">Stable</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">ZFS (local)</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">zfspool</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
</tr>
</tbody>
</table>
</div>
<div class="sect2">
<h3 id="pvesr_schedule_time_format">
<span>9.2. Schedule Format</span>
 <a class="headerlink" href="#pvesr_schedule_time_format" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Replication uses <a href="#chapter_calendar_events">calendar events</a> for
configuring the schedule.</p></div>
</div>
<div class="sect2">
<h3 id="_error_handling">
<span>9.3. Error Handling</span>
 <a class="headerlink" href="#_error_handling" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>If a replication job encounters problems, it is placed in an error state.
In this state, the configured replication intervals get suspended
temporarily. The failed replication is repeatedly tried again in a
30 minute interval.
Once this succeeds, the original schedule gets activated again.</p></div>
<div class="sect3">
<h4 id="_possible_issues">9.3.1. Possible issues
 <a class="headerlink" href="#_possible_issues" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Some of the most common issues are in the following list. Depending on your
setup there may be another cause.</p></div>
<div class="ulist"><ul>
<li>
<p>
Network is not working.
</p>
</li>
<li>
<p>
No free space left on the replication target storage.
</p>
</li>
<li>
<p>
Storage with same storage ID available on the target node
</p>
</li>
</ul></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">You can always use the replication log to find out what is causing the problem.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="_migrating_a_guest_in_case_of_error">9.3.2. Migrating a guest in case of Error
 <a class="headerlink" href="#_migrating_a_guest_in_case_of_error" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>In the case of a grave error, a virtual guest may get stuck on a failed
node. You then need to move it manually to a working node again.</p></div>
</div>
<div class="sect3">
<h4 id="_example_2">9.3.3. Example
 <a class="headerlink" href="#_example_2" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Let’s assume that you have two guests (VM 100 and CT 200) running on node A
and replicate to node B.
Node A failed and can not get back online. Now you have to migrate the guest
to Node B manually.</p></div>
<div class="ulist"><ul>
<li>
<p>
connect to node B over ssh or open its shell via the web UI
</p>
</li>
<li>
<p>
check if that the cluster is quorate
</p>
<div class="listingblock">
<div class="content monospaced">
<pre># pvecm status</pre>
</div></div>
</li>
<li>
<p>
If you have no quorum, we strongly advise to fix this first and make the
  node operable again. Only if this is not possible at the moment, you may
  use the following command to enforce quorum on the current node:
</p>
<div class="listingblock">
<div class="content monospaced">
<pre># pvecm expected 1</pre>
</div></div>
</li>
</ul></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">Avoid changes which affect the cluster if <span class="monospaced">expected votes</span> are set
(for example adding/removing nodes, storages, virtual guests) at all costs.
Only use it to get vital guests up and running again or to resolve the quorum
issue itself.</td>
</tr></tbody></table>
</div>
<div class="ulist"><ul>
<li>
<p>
move both guest configuration files form the origin node A to node B:
</p>
<div class="listingblock">
<div class="content monospaced">
<pre># mv /etc/pve/nodes/A/qemu-server/100.conf /etc/pve/nodes/B/qemu-server/100.conf
# mv /etc/pve/nodes/A/lxc/200.conf /etc/pve/nodes/B/lxc/200.conf</pre>
</div></div>
</li>
<li>
<p>
Now you can start the guests again:
</p>
<div class="listingblock">
<div class="content monospaced">
<pre># qm start 100
# pct start 200</pre>
</div></div>
</li>
</ul></div>
<div class="paragraph">
<p>Remember to replace the VMIDs and node names with your respective values.</p></div>
</div>
</div>
<div class="sect2">
<h3 id="_managing_jobs">
<span>9.4. Managing Jobs</span>
 <a class="headerlink" href="#_managing_jobs" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-qemu-add-replication-job.png">
<img src="images/screenshot/gui-qemu-add-replication-job.png" alt="screenshot/gui-qemu-add-replication-job.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>You can use the web GUI to create, modify, and remove replication jobs
easily. Additionally, the command-line interface (CLI) tool <span class="monospaced">pvesr</span> can be
used to do this.</p></div>
<div class="paragraph">
<p>You can find the replication panel on all levels (datacenter, node, virtual
guest) in the web GUI. They differ in which jobs get shown:
all, node- or guest-specific jobs.</p></div>
<div class="paragraph">
<p>When adding a new job, you need to specify the guest if not already selected
as well as the target node. The replication
<a href="#pvesr_schedule_time_format">schedule</a> can be set if the default of <span class="monospaced">all
15 minutes</span> is not desired. You may impose a rate-limit on a replication
job. The rate limit can help to keep the load on the storage acceptable.</p></div>
<div class="paragraph">
<p>A replication job is identified by a cluster-wide unique ID. This ID is
composed of the VMID in addition to a job number.
This ID must only be specified manually if the CLI tool is used.</p></div>
</div>
<div class="sect2">
<h3 id="_command_line_interface_examples">
<span>9.5. Command-line Interface Examples</span>
 <a class="headerlink" href="#_command_line_interface_examples" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Create a replication job which runs every 5 minutes with a limited bandwidth
of 10 Mbps (megabytes per second) for the guest with ID 100.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># pvesr create-local-job 100-0 pve1 --schedule "*/5" --rate 10</pre>
</div></div>
<div class="paragraph">
<p>Disable an active job with ID <span class="monospaced">100-0</span>.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># pvesr disable 100-0</pre>
</div></div>
<div class="paragraph">
<p>Enable a deactivated job with ID <span class="monospaced">100-0</span>.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># pvesr enable 100-0</pre>
</div></div>
<div class="paragraph">
<p>Change the schedule interval of the job with ID <span class="monospaced">100-0</span> to once per hour.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># pvesr update 100-0 --schedule '*/00'</pre>
</div></div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="chapter_virtual_machines">
10. QEMU/KVM Virtual Machines
 <a class="headerlink" href="#chapter_virtual_machines" title="Permalink to this heading"></a>
</h2>
<div class="sectionbody">
<div class="paragraph">
<p>QEMU (short form for Quick Emulator) is an open source hypervisor that emulates a
physical computer. From the perspective of the host system where QEMU is
running, QEMU is a user program which has access to a number of local resources
like partitions, files, network cards which are then passed to an
emulated computer which sees them as if they were real devices.</p></div>
<div class="paragraph">
<p>A guest operating system running in the emulated computer accesses these
devices, and runs as if it were running on real hardware. For instance, you can pass
an ISO image as a parameter to QEMU, and the OS running in the emulated computer
will see a real CD-ROM inserted into a CD drive.</p></div>
<div class="paragraph">
<p>QEMU can emulate a great variety of hardware from ARM to Sparc, but Proxmox VE is
only concerned with 32 and 64 bits PC clone emulation, since it represents the
overwhelming majority of server hardware. The emulation of PC clones is also one
of the fastest due to the availability of processor extensions which greatly
speed up QEMU when the emulated architecture is the same as the host
architecture.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">You may sometimes encounter the term <em>KVM</em> (Kernel-based Virtual Machine).
It means that QEMU is running with the support of the virtualization processor
extensions, via the Linux KVM module. In the context of Proxmox VE <em>QEMU</em> and
<em>KVM</em> can be used interchangeably, as QEMU in Proxmox VE will always try to load the KVM
module.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>QEMU inside Proxmox VE runs as a root process, since this is required to access block
and PCI devices.</p></div>
<div class="sect2">
<h3 id="_emulated_devices_and_paravirtualized_devices">
<span>10.1. Emulated devices and paravirtualized devices</span>
 <a class="headerlink" href="#_emulated_devices_and_paravirtualized_devices" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>The PC hardware emulated by QEMU includes a motherboard, network controllers,
SCSI, IDE and SATA controllers, serial ports (the complete list can be seen in
the <span class="monospaced">kvm(1)</span> man page) all of them emulated in software. All these devices
are the exact software equivalent of existing hardware devices, and if the OS
running in the guest has the proper drivers it will use the devices as if it
were running on real hardware. This allows QEMU to run <em>unmodified</em> operating
systems.</p></div>
<div class="paragraph">
<p>This however has a performance cost, as running in software what was meant to
run in hardware involves a lot of extra work for the host CPU. To mitigate this,
QEMU can present to the guest operating system <em>paravirtualized devices</em>, where
the guest OS recognizes it is running inside QEMU and cooperates with the
hypervisor.</p></div>
<div class="paragraph">
<p>QEMU relies on the virtio virtualization standard, and is thus able to present
paravirtualized virtio devices, which includes a paravirtualized generic disk
controller, a paravirtualized network card, a paravirtualized serial port,
a paravirtualized SCSI controller, etc …</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">It is <strong>highly recommended</strong> to use the virtio devices whenever you can, as
they provide a big performance improvement and are generally better maintained.
Using the virtio generic disk controller versus an emulated IDE controller will
double the sequential write throughput, as measured with <span class="monospaced">bonnie++(8)</span>. Using
the virtio network interface can deliver up to three times the throughput of an
emulated Intel E1000 network card, as measured with <span class="monospaced">iperf(1)</span>. <span class="footnote" data-note="See
this benchmark on the KVM wiki <a href=&quot;https://www.linux-kvm.org/page/Using_VirtIO_NIC&quot;>https://www.linux-kvm.org/page/Using_VirtIO_NIC</a>">[<a id="_footnoteref_33" href="#_footnote_33" title="View footnote" class="footnote">33</a>]</span></td>
</tr></tbody></table>
</div>
</div>
<div class="sect2">
<h3 id="qm_virtual_machines_settings">
<span>10.2. Virtual Machines Settings</span>
 <a class="headerlink" href="#qm_virtual_machines_settings" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Generally speaking Proxmox VE tries to choose sane defaults for virtual machines
(VM). Make sure you understand the meaning of the settings you change, as it
could incur a performance slowdown, or putting your data at risk.</p></div>
<div class="sect3">
<h4 id="qm_general_settings">10.2.1. General Settings
 <a class="headerlink" href="#qm_general_settings" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-create-vm-general.png">
<img src="images/screenshot/gui-create-vm-general.png" alt="screenshot/gui-create-vm-general.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>General settings of a VM include</p></div>
<div class="ulist"><ul>
<li>
<p>
the <strong>Node</strong> : the physical server on which the VM will run
</p>
</li>
<li>
<p>
the <strong>VM ID</strong>: a unique number in this Proxmox VE installation used to identify your VM
</p>
</li>
<li>
<p>
<strong>Name</strong>: a free form text string you can use to describe the VM
</p>
</li>
<li>
<p>
<strong>Resource Pool</strong>: a logical group of VMs
</p>
</li>
</ul></div>
</div>
<div class="sect3">
<h4 id="qm_os_settings">10.2.2. OS Settings
 <a class="headerlink" href="#qm_os_settings" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-create-vm-os.png">
<img src="images/screenshot/gui-create-vm-os.png" alt="screenshot/gui-create-vm-os.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>When creating a virtual machine (VM), setting the proper Operating System(OS)
allows Proxmox VE to optimize some low level parameters. For instance Windows OS
expect the BIOS clock to use the local time, while Unix based OS expect the
BIOS clock to have the UTC time.</p></div>
</div>
<div class="sect3">
<h4 id="qm_system_settings">10.2.3. System Settings
 <a class="headerlink" href="#qm_system_settings" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>On VM creation you can change some basic system components of the new VM. You
can specify which <a href="#qm_display">display type</a> you want to use.</p></div>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-create-vm-system.png">
<img src="images/screenshot/gui-create-vm-system.png" alt="screenshot/gui-create-vm-system.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>Additionally, the <a href="#qm_hard_disk">SCSI controller</a> can be changed.
If you plan to install the QEMU Guest Agent, or if your selected ISO image
already ships and installs it automatically, you may want to tick the <em>QEMU
Agent</em> box, which lets Proxmox VE know that it can use its features to show some
more information, and complete some actions (for example, shutdown or
snapshots) more intelligently.</p></div>
<div class="paragraph">
<p>Proxmox VE allows to boot VMs with different firmware and machine types, namely
<a href="#qm_bios_and_uefi">SeaBIOS and OVMF</a>. In most cases you want to switch from
the default SeaBIOS to OVMF only if you plan to use
<a href="#qm_pci_passthrough">PCIe passthrough</a>.</p></div>
<div class="sect4">
<h5 id="qm_machine_type">Machine Type
 <a class="headerlink" href="#qm_machine_type" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>A VM’s <em>Machine Type</em> defines the hardware layout of the VM’s virtual
motherboard. You can choose between the default
<a href="https://en.wikipedia.org/wiki/Intel_440FX">Intel 440FX</a> or the
<a href="https://ark.intel.com/content/www/us/en/ark/products/31918/intel-82q35-graphics-and-memory-controller.html">Q35</a>
chipset, which also provides a virtual PCIe bus, and thus may be
desired if you want to pass through PCIe hardware.
Additionally, you can select a <a href="#qm_pci_viommu">vIOMMU</a> implementation.</p></div>
<div class="sect5">
<h6 id="_machine_version">Machine Version
 <a class="headerlink" href="#_machine_version" title="Permalink to this heading"></a>
</h6>
<div class="paragraph">
<p>Each machine type is versioned in QEMU and a given QEMU binary supports many
machine versions. New versions might bring support for new features, fixes or
general improvements. However, they also change properties of the virtual
hardware. To avoid sudden changes from the guest’s perspective and ensure
compatibility of the VM state, live-migration and snapshots with RAM will keep
using the same machine version in the new QEMU instance.</p></div>
<div class="paragraph">
<p>For Windows guests, the machine version is pinned during creation, because
Windows is sensitive to changes in the virtual hardware - even between cold
boots. For example, the enumeration of network devices might be different with
different machine versions. Other OSes like Linux can usually deal with such
changes just fine. For those, the <em>Latest</em> machine version is used by default.
This means that after a fresh start, the newest machine version supported by the
QEMU binary is used (e.g. the newest machine version QEMU 8.1 supports is
version 8.1 for each machine type).</p></div>
</div>
<div class="sect5">
<h6 id="qm_machine_update">Update to a Newer Machine Version
 <a class="headerlink" href="#qm_machine_update" title="Permalink to this heading"></a>
</h6>
<div class="paragraph">
<p>Very old machine versions might become deprecated in QEMU. For example, this is
the case for versions 1.4 to 1.7 for the i440fx machine type. It is expected
that support for these machine versions will be dropped at some point. If you
see a deprecation warning, you should change the machine version to a newer one.
Be sure to have a working backup first and be prepared for changes to how the
guest sees hardware. In some scenarios, re-installing certain drivers might be
required. You should also check for snapshots with RAM that were taken with
these machine versions (i.e. the <span class="monospaced">runningmachine</span> configuration entry).
Unfortunately, there is no way to change the machine version of a snapshot, so
you’d need to load the snapshot to salvage any data from it.</p></div>
</div>
</div>
</div>
<div class="sect3">
<h4 id="qm_hard_disk">10.2.4. Hard Disk
 <a class="headerlink" href="#qm_hard_disk" title="Permalink to this heading"></a>
</h4>
<div class="sect4">
<h5 id="qm_hard_disk_bus">Bus/Controller
 <a class="headerlink" href="#qm_hard_disk_bus" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>QEMU can emulate a number of storage controllers:</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">It is highly recommended to use the <strong>VirtIO SCSI</strong> or <strong>VirtIO Block</strong>
controller for performance reasons and because they are better maintained.</td>
</tr></tbody></table>
</div>
<div class="ulist"><ul>
<li>
<p>
the <strong>IDE</strong> controller, has a design which goes back to the 1984 PC/AT disk
controller. Even if this controller has been superseded by recent designs,
each and every OS you can think of has support for it, making it a great choice
if you want to run an OS released before 2003. You can connect up to 4 devices
on this controller.
</p>
</li>
<li>
<p>
the <strong>SATA</strong> (Serial ATA) controller, dating from 2003, has a more modern
design, allowing higher throughput and a greater number of devices to be
connected. You can connect up to 6 devices on this controller.
</p>
</li>
<li>
<p>
the <strong>SCSI</strong> controller, designed in 1985, is commonly found on server grade
hardware, and can connect up to 14 storage devices. Proxmox VE emulates by default a
LSI 53C895A controller.
</p>
<div class="paragraph">
<p>A SCSI controller of type <em>VirtIO SCSI single</em> and enabling the
<a href="#qm_hard_disk_iothread">IO Thread</a> setting for the attached disks is
recommended if you aim for performance. This is the default for newly created
Linux VMs since Proxmox VE 7.3. Each disk will have its own <em>VirtIO SCSI</em> controller,
and QEMU will handle the disks IO in a dedicated thread. Linux distributions
have support for this controller since 2012, and FreeBSD since 2014. For Windows
OSes, you need to provide an extra ISO containing the drivers during the
installation.</p></div>
</li>
<li>
<p>
The <strong>VirtIO Block</strong> controller, often just called VirtIO or virtio-blk,
is an older type of paravirtualized controller. It has been superseded by the
VirtIO SCSI Controller, in terms of features.
</p>
</li>
</ul></div>
</div>
<div class="sect4">
<h5 id="qm_hard_disk_formats">Image Format
 <a class="headerlink" href="#qm_hard_disk_formats" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>On each controller you attach a number of emulated hard disks, which are backed
by a file or a block device residing in the configured storage. The choice of
a storage type will determine the format of the hard disk image. Storages which
present block devices (LVM, ZFS, Ceph) will require the <strong>raw disk image format</strong>,
whereas files based storages (Ext4, NFS, CIFS, GlusterFS) will let you to choose
either the <strong>raw disk image format</strong> or the <strong>QEMU image format</strong>.</p></div>
<div class="ulist"><ul>
<li>
<p>
the <strong>QEMU image format</strong> is a copy on write format which allows snapshots, and
  thin provisioning of the disk image.
</p>
</li>
<li>
<p>
the <strong>raw disk image</strong> is a bit-to-bit image of a hard disk, similar to what
 you would get when executing the <span class="monospaced">dd</span> command on a block device in Linux. This
 format does not support thin provisioning or snapshots by itself, requiring
 cooperation from the storage layer for these tasks. It may, however, be up to
 10% faster than the <strong>QEMU image format</strong>. <span class="footnote" data-note="See this benchmark for details
 <a href=&quot;https://events.static.linuxfound.org/sites/events/files/slides/CloudOpen2013_Khoa_Huynh_v3.pdf&quot;>https://events.static.linuxfound.org/sites/events/files/slides/CloudOpen2013_Khoa_Huynh_v3.pdf</a>">[<a id="_footnoteref_34" href="#_footnote_34" title="View footnote" class="footnote">34</a>]</span>
</p>
</li>
<li>
<p>
the <strong>VMware image format</strong> only makes sense if you intend to import/export the
 disk image to other hypervisors.
</p>
</li>
</ul></div>
</div>
<div class="sect4">
<h5 id="qm_hard_disk_cache">Cache Mode
 <a class="headerlink" href="#qm_hard_disk_cache" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Setting the <strong>Cache</strong> mode of the hard drive will impact how the host system will
notify the guest systems of block write completions. The <strong>No cache</strong> default
means that the guest system will be notified that a write is complete when each
block reaches the physical storage write queue, ignoring the host page cache.
This provides a good balance between safety and speed.</p></div>
<div class="paragraph">
<p>If you want the Proxmox VE backup manager to skip a disk when doing a backup of a VM,
you can set the <strong>No backup</strong> option on that disk.</p></div>
<div class="paragraph">
<p>If you want the Proxmox VE storage replication mechanism to skip a disk when starting
 a replication job, you can set the <strong>Skip replication</strong> option on that disk.
As of Proxmox VE 5.0, replication requires the disk images to be on a storage of type
<span class="monospaced">zfspool</span>, so adding a disk image to other storages when the VM has replication
configured requires to skip replication for this disk image.</p></div>
</div>
<div class="sect4">
<h5 id="qm_hard_disk_discard">Trim/Discard
 <a class="headerlink" href="#qm_hard_disk_discard" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>If your storage supports <em>thin provisioning</em> (see the storage chapter in the
Proxmox VE guide), you can activate the <strong>Discard</strong> option on a drive. With <strong>Discard</strong>
set and a <em>TRIM</em>-enabled guest OS <span class="footnote" data-note="TRIM, UNMAP, and discard
<a href=&quot;https://en.wikipedia.org/wiki/Trim_%28computing%29&quot;>https://en.wikipedia.org/wiki/Trim_%28computing%29</a>">[<a id="_footnoteref_35" href="#_footnote_35" title="View footnote" class="footnote">35</a>]</span>, when the VM’s filesystem
marks blocks as unused after deleting files, the controller will relay this
information to the storage, which will then shrink the disk image accordingly.
For the guest to be able to issue <em>TRIM</em> commands, you must enable the <strong>Discard</strong>
option on the drive. Some guest operating systems may also require the
<strong>SSD Emulation</strong> flag to be set. Note that <strong>Discard</strong> on <strong>VirtIO Block</strong> drives is
only supported on guests using Linux Kernel 5.0 or higher.</p></div>
<div class="paragraph">
<p>If you would like a drive to be presented to the guest as a solid-state drive
rather than a rotational hard disk, you can set the <strong>SSD emulation</strong> option on
that drive. There is no requirement that the underlying storage actually be
backed by SSDs; this feature can be used with physical media of any type.
Note that <strong>SSD emulation</strong> is not supported on <strong>VirtIO Block</strong> drives.</p></div>
</div>
<div class="sect4">
<h5 id="qm_hard_disk_iothread">IO Thread
 <a class="headerlink" href="#qm_hard_disk_iothread" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>The option <strong>IO Thread</strong> can only be used when using a disk with the <strong>VirtIO</strong>
controller, or with the <strong>SCSI</strong> controller, when the emulated controller type is
<strong>VirtIO SCSI single</strong>. With <strong>IO Thread</strong> enabled, QEMU creates one I/O thread per
storage controller rather than handling all I/O in the main event loop or vCPU
threads. One benefit is better work distribution and utilization of the
underlying storage. Another benefit is reduced latency (hangs) in the guest for
very I/O-intensive host workloads, since neither the main thread nor a vCPU
thread can be blocked by disk I/O.</p></div>
</div>
</div>
<div class="sect3">
<h4 id="qm_cpu">10.2.5. CPU
 <a class="headerlink" href="#qm_cpu" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-create-vm-cpu.png">
<img src="images/screenshot/gui-create-vm-cpu.png" alt="screenshot/gui-create-vm-cpu.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>A <strong>CPU socket</strong> is a physical slot on a PC motherboard where you can plug a CPU.
This CPU can then contain one or many <strong>cores</strong>, which are independent
processing units. Whether you have a single CPU socket with 4 cores, or two CPU
sockets with two cores is mostly irrelevant from a performance point of view.
However some software licenses depend on the number of sockets a machine has,
in that case it makes sense to set the number of sockets to what the license
allows you.</p></div>
<div class="paragraph">
<p>Increasing the number of virtual CPUs (cores and sockets) will usually provide a
performance improvement though that is heavily dependent on the use of the VM.
Multi-threaded applications will of course benefit from a large number of
virtual CPUs, as for each virtual cpu you add, QEMU will create a new thread of
execution on the host system. If you’re not sure about the workload of your VM,
it is usually a safe bet to set the number of <strong>Total cores</strong> to 2.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">It is perfectly safe if the <em>overall</em> number of cores of all your VMs
is greater than the number of cores on the server (for example, 4 VMs each with
4 cores (= total 16) on a machine with only 8 cores). In that case the host
system will balance the QEMU execution threads between your server cores, just
like if you were running a standard multi-threaded application. However, Proxmox VE
will prevent you from starting VMs with more virtual CPU cores than physically
available, as this will only bring the performance down due to the cost of
context switches.</td>
</tr></tbody></table>
</div>
<div class="sect4">
<h5 id="qm_cpu_resource_limits">Resource Limits
 <a class="headerlink" href="#qm_cpu_resource_limits" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p><strong>cpulimit</strong></p></div>
<div class="paragraph">
<p>In addition to the number of virtual cores, the total available “Host CPU
Time” for the VM can be set with the <strong>cpulimit</strong> option. It is a floating point
value representing CPU time in percent, so <span class="monospaced">1.0</span> is equal to <span class="monospaced">100%</span>, <span class="monospaced">2.5</span> to
<span class="monospaced">250%</span> and so on. If a single process would fully use one single core it would
have <span class="monospaced">100%</span> CPU Time usage. If a VM with four cores utilizes all its cores
fully it would theoretically use <span class="monospaced">400%</span>. In reality the usage may be even a bit
higher as QEMU can have additional threads for VM peripherals besides the vCPU
core ones.</p></div>
<div class="paragraph">
<p>This setting can be useful when a VM should have multiple vCPUs because it is
running some processes in parallel, but the VM as a whole should not be able to
run all vCPUs at 100% at the same time.</p></div>
<div class="paragraph">
<p>For example, suppose you have a virtual machine that would benefit from having 8
virtual CPUs, but you don’t want the VM to be able to max out all 8 cores
running at full load - because that would overload the server and leave other
virtual machines and containers with too little CPU time. To solve this, you
could set <strong>cpulimit</strong> to <span class="monospaced">4.0</span> (=400%). This means that if the VM fully utilizes
all 8 virtual CPUs by running 8 processes simultaneously, each vCPU will receive
a maximum of 50% CPU time from the physical cores. However, if the VM workload
only fully utilizes 4 virtual CPUs, it could still receive up to 100% CPU time
from a physical core, for a total of 400%.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">VMs can, depending on their configuration, use additional threads, such
as for networking or IO operations but also live migration. Thus a VM can show
up to use more CPU time than just its virtual CPUs could use. To ensure that a
VM never uses more CPU time than vCPUs assigned, set the <strong>cpulimit</strong> to
the same value as the total core count.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p><strong>cpuunits</strong></p></div>
<div class="paragraph">
<p>With the <strong>cpuunits</strong> option, nowadays often called CPU shares or CPU weight, you
can control how much CPU time a VM gets compared to other running VMs. It is a
relative weight which defaults to <span class="monospaced">100</span> (or <span class="monospaced">1024</span> if the host uses legacy
cgroup v1). If you increase this for a VM it will be prioritized by the
scheduler in comparison to other VMs with lower weight.</p></div>
<div class="paragraph">
<p>For example, if VM 100 has set the default <span class="monospaced">100</span> and VM 200 was changed to
<span class="monospaced">200</span>, the latter VM 200 would receive twice the CPU bandwidth than the first
VM 100.</p></div>
<div class="paragraph">
<p>For more information see <span class="monospaced">man systemd.resource-control</span>, here <span class="monospaced">CPUQuota</span>
corresponds to <span class="monospaced">cpulimit</span> and <span class="monospaced">CPUWeight</span> to our <span class="monospaced">cpuunits</span> setting. Visit its
Notes section for references and implementation details.</p></div>
<div class="paragraph">
<p><strong>affinity</strong></p></div>
<div class="paragraph">
<p>With the <strong>affinity</strong> option, you can specify the physical CPU cores that are used
to run the VM’s vCPUs. Peripheral VM processes, such as those for I/O, are not
affected by this setting. Note that the <strong>CPU affinity is not a security
feature</strong>.</p></div>
<div class="paragraph">
<p>Forcing a CPU <strong>affinity</strong> can make sense in certain cases but is accompanied by
an increase in complexity and maintenance effort. For example, if you want to
add more VMs later or migrate VMs to nodes with fewer CPU cores. It can also
easily lead to asynchronous and therefore limited system performance if some
CPUs are fully utilized while others are almost idle.</p></div>
<div class="paragraph">
<p>The <strong>affinity</strong> is set through the <span class="monospaced">taskset</span> CLI tool. It accepts the host CPU
numbers (see <span class="monospaced">lscpu</span>) in the <span class="monospaced">List Format</span> from <span class="monospaced">man cpuset</span>. This ASCII decimal
list can contain numbers but also number ranges. For example, the <strong>affinity</strong>
<span class="monospaced">0-1,8-11</span> (expanded <span class="monospaced">0, 1, 8, 9, 10, 11</span>) would allow the VM to run on only
these six specific host cores.</p></div>
</div>
<div class="sect4">
<h5 id="_cpu_type">CPU Type
 <a class="headerlink" href="#_cpu_type" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>QEMU can emulate a number different of <strong>CPU types</strong> from 486 to the latest Xeon
processors. Each new processor generation adds new features, like hardware
assisted 3d rendering, random number generation, memory protection, etc. Also,
a current generation can be upgraded through
<a href="#chapter_firmware_updates">microcode update</a> with bug or security fixes.</p></div>
<div class="paragraph">
<p>Usually you should select for your VM a processor type which closely matches the
CPU of the host system, as it means that the host CPU features (also called <em>CPU
flags</em> ) will be available in your VMs. If you want an exact match, you can set
the CPU type to <strong>host</strong> in which case the VM will have exactly the same CPU flags
as your host system.</p></div>
<div class="paragraph">
<p>This has a downside though. If you want to do a live migration of VMs between
different hosts, your VM might end up on a new system with a different CPU type
or a different microcode version.
If the CPU flags passed to the guest are missing, the QEMU process will stop. To
remedy this QEMU has also its own virtual CPU types, that Proxmox VE uses by default.</p></div>
<div class="paragraph">
<p>The backend default is <em>kvm64</em> which works on essentially all x86_64 host CPUs
and the UI default when creating a new VM is <em>x86-64-v2-AES</em>, which requires a
host CPU starting from Westmere for Intel or at least a fourth generation
Opteron for AMD.</p></div>
<div class="paragraph">
<p>In short:</p></div>
<div class="paragraph">
<p>If you don’t care about live migration or have a homogeneous cluster where all
nodes have the same CPU and same microcode version, set the CPU type to host, as
in theory this will give your guests maximum performance.</p></div>
<div class="paragraph">
<p>If you care about live migration and security, and you have only Intel CPUs or
only AMD CPUs, choose the lowest generation CPU model of your cluster.</p></div>
<div class="paragraph">
<p>If you care about live migration without security, or have mixed Intel/AMD
cluster, choose the lowest compatible virtual QEMU CPU type.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Live migrations between Intel and AMD host CPUs have no guarantee to work.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>See also
<a href="#chapter_qm_vcpu_list">List of AMD and Intel CPU Types as Defined in QEMU</a>.</p></div>
</div>
<div class="sect4">
<h5 id="_qemu_cpu_types">QEMU CPU Types
 <a class="headerlink" href="#_qemu_cpu_types" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>QEMU also provide virtual CPU types, compatible with both Intel and AMD host
CPUs.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">To mitigate the Spectre vulnerability for virtual CPU types, you need to
add the relevant CPU flags, see
<a href="#qm_meltdown_spectre">Meltdown / Spectre related CPU flags</a>.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>Historically, Proxmox VE had the <em>kvm64</em> CPU model, with CPU flags at the level of
Pentium 4 enabled, so performance was not great for certain workloads.</p></div>
<div class="paragraph">
<p>In the summer of 2020, AMD, Intel, Red Hat, and SUSE collaborated to define
three x86-64 microarchitecture levels on top of the x86-64 baseline, with modern
flags enabled. For details, see the
<a href="https://gitlab.com/x86-psABIs/x86-64-ABI">x86-64-ABI specification</a>.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Some newer distributions like CentOS 9 are now built with <em>x86-64-v2</em>
flags as a minimum requirement.</td>
</tr></tbody></table>
</div>
<div class="ulist"><ul>
<li>
<p>
<em>kvm64 (x86-64-v1)</em>: Compatible with Intel CPU &gt;= Pentium 4, AMD CPU &gt;=
Phenom.
</p>
</li>
<li>
<p>
<em>x86-64-v2</em>: Compatible with Intel CPU &gt;= Nehalem, AMD CPU &gt;= Opteron_G3.
Added CPU flags compared to <em>x86-64-v1</em>: <em>+cx16</em>, <em>+lahf-lm</em>, <em>+popcnt</em>, <em>+pni</em>,
<em>+sse4.1</em>, <em>+sse4.2</em>, <em>+ssse3</em>.
</p>
</li>
<li>
<p>
<em>x86-64-v2-AES</em>: Compatible with Intel CPU &gt;= Westmere, AMD CPU &gt;= Opteron_G4.
Added CPU flags compared to <em>x86-64-v2</em>: <em>+aes</em>.
</p>
</li>
<li>
<p>
<em>x86-64-v3</em>: Compatible with Intel CPU &gt;= Broadwell, AMD CPU &gt;= EPYC. Added
CPU flags compared to <em>x86-64-v2-AES</em>: <em>+avx</em>, <em>+avx2</em>, <em>+bmi1</em>, <em>+bmi2</em>,
<em>+f16c</em>, <em>+fma</em>, <em>+movbe</em>, <em>+xsave</em>.
</p>
</li>
<li>
<p>
<em>x86-64-v4</em>: Compatible with Intel CPU &gt;= Skylake, AMD CPU &gt;= EPYC v4 Genoa.
Added CPU flags compared to <em>x86-64-v3</em>: <em>+avx512f</em>, <em>+avx512bw</em>, <em>+avx512cd</em>,
<em>+avx512dq</em>, <em>+avx512vl</em>.
</p>
</li>
</ul></div>
</div>
<div class="sect4">
<h5 id="_custom_cpu_types">Custom CPU Types
 <a class="headerlink" href="#_custom_cpu_types" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>You can specify custom CPU types with a configurable set of features. These are
maintained in the configuration file <span class="monospaced">/etc/pve/virtual-guest/cpu-models.conf</span> by
an administrator. See <span class="monospaced">man cpu-models.conf</span> for format details.</p></div>
<div class="paragraph">
<p>Specified custom types can be selected by any user with the <span class="monospaced">Sys.Audit</span>
privilege on <span class="monospaced">/nodes</span>. When configuring a custom CPU type for a VM via the CLI
or API, the name needs to be prefixed with <em>custom-</em>.</p></div>
</div>
<div class="sect4">
<h5 id="qm_meltdown_spectre">Meltdown / Spectre related CPU flags
 <a class="headerlink" href="#qm_meltdown_spectre" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>There are several CPU flags related to the Meltdown and Spectre vulnerabilities
<span class="footnote" data-note="Meltdown Attack <a href=&quot;https://meltdownattack.com/&quot;>https://meltdownattack.com/</a>">[<a id="_footnoteref_36" href="#_footnote_36" title="View footnote" class="footnote">36</a>]</span> which need to be set
manually unless the selected CPU type of your VM already enables them by default.</p></div>
<div class="paragraph">
<p>There are two requirements that need to be fulfilled in order to use these
CPU flags:</p></div>
<div class="ulist"><ul>
<li>
<p>
The host CPU(s) must support the feature and propagate it to the guest’s virtual CPU(s)
</p>
</li>
<li>
<p>
The guest operating system must be updated to a version which mitigates the
  attacks and is able to utilize the CPU feature
</p>
</li>
</ul></div>
<div class="paragraph">
<p>Otherwise you need to set the desired CPU flag of the virtual CPU, either by
editing the CPU options in the web UI, or by setting the <em>flags</em> property of the
<em>cpu</em> option in the VM configuration file.</p></div>
<div class="paragraph">
<p>For Spectre v1,v2,v4 fixes, your CPU or system vendor also needs to provide a
so-called “microcode update” for your CPU, see
<a href="#chapter_firmware_updates">chapter Firmware Updates</a>. Note that not all
affected CPUs can be updated to support spec-ctrl.</p></div>
<div class="paragraph">
<p>To check if the Proxmox VE host is vulnerable, execute the following command as root:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>for f in /sys/devices/system/cpu/vulnerabilities/*; do echo "${f##*/} -" $(cat "$f"); done</pre>
</div></div>
<div class="paragraph">
<p>A community script is also available to detect if the host is still vulnerable.
<span class="footnote" data-note="spectre-meltdown-checker <a href=&quot;https://meltdown.ovh/&quot;>https://meltdown.ovh/</a>">[<a id="_footnoteref_37" href="#_footnote_37" title="View footnote" class="footnote">37</a>]</span></p></div>
</div>
<div class="sect4">
<h5 id="_intel_processors">Intel processors
 <a class="headerlink" href="#_intel_processors" title="Permalink to this heading"></a>
</h5>
<div class="ulist"><ul>
<li>
<p>
<em>pcid</em>
</p>
<div class="paragraph">
<p>This reduces the performance impact of the Meltdown (CVE-2017-5754) mitigation
called <em>Kernel Page-Table Isolation (KPTI)</em>, which effectively hides
the Kernel memory from the user space. Without PCID, KPTI is quite an expensive
mechanism <span class="footnote" data-note="PCID is now a critical performance/security feature on x86
<a href=&quot;https://groups.google.com/forum/m/#!topic/mechanical-sympathy/L9mHTbeQLNU&quot;>https://groups.google.com/forum/m/#!topic/mechanical-sympathy/L9mHTbeQLNU</a>">[<a id="_footnoteref_38" href="#_footnote_38" title="View footnote" class="footnote">38</a>]</span>.</p></div>
<div class="paragraph">
<p>To check if the Proxmox VE host supports PCID, execute the following command as root:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># grep ' pcid ' /proc/cpuinfo</pre>
</div></div>
<div class="paragraph">
<p>If this does not return empty your host’s CPU has support for <em>pcid</em>.</p></div>
</li>
<li>
<p>
<em>spec-ctrl</em>
</p>
<div class="paragraph">
<p>Required to enable the Spectre v1 (CVE-2017-5753) and Spectre v2 (CVE-2017-5715) fix,
in cases where retpolines are not sufficient.
Included by default in Intel CPU models with -IBRS suffix.
Must be explicitly turned on for Intel CPU models without -IBRS suffix.
Requires an updated host CPU microcode (intel-microcode &gt;= 20180425).</p></div>
</li>
<li>
<p>
<em>ssbd</em>
</p>
<div class="paragraph">
<p>Required to enable the Spectre V4 (CVE-2018-3639) fix. Not included by default in any Intel CPU model.
Must be explicitly turned on for all Intel CPU models.
Requires an updated host CPU microcode(intel-microcode &gt;= 20180703).</p></div>
</li>
</ul></div>
</div>
<div class="sect4">
<h5 id="_amd_processors">AMD processors
 <a class="headerlink" href="#_amd_processors" title="Permalink to this heading"></a>
</h5>
<div class="ulist"><ul>
<li>
<p>
<em>ibpb</em>
</p>
<div class="paragraph">
<p>Required to enable the Spectre v1 (CVE-2017-5753) and Spectre v2 (CVE-2017-5715) fix,
in cases where retpolines are not sufficient.
Included by default in AMD CPU models with -IBPB suffix.
Must be explicitly turned on for AMD CPU models without -IBPB suffix.
Requires the host CPU microcode to support this feature before it can be used for guest CPUs.</p></div>
</li>
<li>
<p>
<em>virt-ssbd</em>
</p>
<div class="paragraph">
<p>Required to enable the Spectre v4 (CVE-2018-3639) fix.
Not included by default in any AMD CPU model.
Must be explicitly turned on for all AMD CPU models.
This should be provided to guests, even if amd-ssbd is also provided, for maximum guest compatibility.
Note that this must be explicitly enabled when when using the "host" cpu model,
because this is a virtual feature which does not exist in the physical CPUs.</p></div>
</li>
<li>
<p>
<em>amd-ssbd</em>
</p>
<div class="paragraph">
<p>Required to enable the Spectre v4 (CVE-2018-3639) fix.
Not included by default in any AMD CPU model. Must be explicitly turned on for all AMD CPU models.
This provides higher performance than virt-ssbd, therefore a host supporting this should always expose this to guests if possible.
virt-ssbd should none the less also be exposed for maximum guest compatibility as some kernels only know about virt-ssbd.</p></div>
</li>
<li>
<p>
<em>amd-no-ssb</em>
</p>
<div class="paragraph">
<p>Recommended to indicate the host is not vulnerable to Spectre V4 (CVE-2018-3639).
Not included by default in any AMD CPU model.
Future hardware generations of CPU will not be vulnerable to CVE-2018-3639,
and thus the guest should be told not to enable its mitigations, by exposing amd-no-ssb.
This is mutually exclusive with virt-ssbd and amd-ssbd.</p></div>
</li>
</ul></div>
</div>
<div class="sect4">
<h5 id="_numa">NUMA
 <a class="headerlink" href="#_numa" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>You can also optionally emulate a <strong>NUMA</strong>
<span class="footnote" data-note="<a href=&quot;https://en.wikipedia.org/wiki/Non-uniform_memory_access&quot;>https://en.wikipedia.org/wiki/Non-uniform_memory_access</a>">[<a id="_footnoteref_39" href="#_footnote_39" title="View footnote" class="footnote">39</a>]</span> architecture
in your VMs. The basics of the NUMA architecture mean that instead of having a
global memory pool available to all your cores, the memory is spread into local
banks close to each socket.
This can bring speed improvements as the memory bus is not a bottleneck
anymore. If your system has a NUMA architecture <span class="footnote" data-note="if the command
<span class=&quot;monospaced&quot;>numactl --hardware | grep available</span> returns more than one node, then your host
system has a NUMA architecture">[<a id="_footnoteref_40" href="#_footnote_40" title="View footnote" class="footnote">40</a>]</span> we recommend to activate the option, as this
will allow proper distribution of the VM resources on the host system.
This option is also required to hot-plug cores or RAM in a VM.</p></div>
<div class="paragraph">
<p>If the NUMA option is used, it is recommended to set the number of sockets to
the number of nodes of the host system.</p></div>
</div>
<div class="sect4">
<h5 id="_vcpu_hot_plug">vCPU hot-plug
 <a class="headerlink" href="#_vcpu_hot_plug" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Modern operating systems introduced the capability to hot-plug and, to a
certain extent, hot-unplug CPUs in a running system. Virtualization allows us
to avoid a lot of the (physical) problems real hardware can cause in such
scenarios.
Still, this is a rather new and complicated feature, so its use should be
restricted to cases where its absolutely needed. Most of the functionality can
be replicated with other, well tested and less complicated, features, see
<a href="#qm_cpu_resource_limits">Resource Limits</a>.</p></div>
<div class="paragraph">
<p>In Proxmox VE the maximal number of plugged CPUs is always <span class="monospaced">cores * sockets</span>.
To start a VM with less than this total core count of CPUs you may use the
<strong>vcpus</strong> setting, it denotes how many vCPUs should be plugged in at VM start.</p></div>
<div class="paragraph">
<p>Currently only this feature is only supported on Linux, a kernel newer than 3.10
is needed, a kernel newer than 4.7 is recommended.</p></div>
<div class="paragraph">
<p>You can use a udev rule as follow to automatically set new CPUs as online in
the guest:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>SUBSYSTEM=="cpu", ACTION=="add", TEST=="online", ATTR{online}=="0", ATTR{online}="1"</pre>
</div></div>
<div class="paragraph">
<p>Save this under /etc/udev/rules.d/ as a file ending in <span class="monospaced">.rules</span>.</p></div>
<div class="paragraph">
<p>Note: CPU hot-remove is machine dependent and requires guest cooperation.  The
deletion command does not guarantee CPU removal to actually happen, typically
it’s a request forwarded to guest OS using target dependent mechanism, such as
ACPI on x86/amd64.</p></div>
</div>
</div>
<div class="sect3">
<h4 id="qm_memory">10.2.6. Memory
 <a class="headerlink" href="#qm_memory" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>For each VM you have the option to set a fixed size memory or asking
Proxmox VE to dynamically allocate memory based on the current RAM usage of the
host.</p></div>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-create-vm-memory.png">
<img src="images/screenshot/gui-create-vm-memory.png" alt="screenshot/gui-create-vm-memory.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<div class="title">Fixed Memory Allocation</div><p>When setting memory and minimum memory to the same amount
Proxmox VE will simply allocate what you specify to your VM.</p></div>
<div class="paragraph">
<p>Even when using a fixed memory size, the ballooning device gets added to the
VM, because it delivers useful information such as how much memory the guest
really uses.
In general, you should leave <strong>ballooning</strong> enabled, but if you want to disable
it (like for debugging purposes), simply uncheck <strong>Ballooning Device</strong> or set</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>balloon: 0</pre>
</div></div>
<div class="paragraph">
<p>in the configuration.</p></div>
<div class="paragraph">
<div class="title">Automatic Memory Allocation</div><p>When setting the minimum memory lower than memory, Proxmox VE will make sure that the
minimum amount you specified is always available to the VM, and if RAM usage on
the host is below 80%, will dynamically add memory to the guest up to the
maximum memory specified.</p></div>
<div class="paragraph">
<p>When the host is running low on RAM, the VM will then release some memory
back to the host, swapping running processes if needed and starting the oom
killer in last resort. The passing around of memory between host and guest is
done via a special <span class="monospaced">balloon</span> kernel driver running inside the guest, which will
grab or release memory pages from the host.
<span class="footnote" data-note="A good explanation of the inner workings of the balloon driver can be found here <a href=&quot;https://rwmj.wordpress.com/2010/07/17/virtio-balloon/&quot;>https://rwmj.wordpress.com/2010/07/17/virtio-balloon/</a>">[<a id="_footnoteref_41" href="#_footnote_41" title="View footnote" class="footnote">41</a>]</span></p></div>
<div class="paragraph">
<p>When multiple VMs use the autoallocate facility, it is possible to set a
<strong>Shares</strong> coefficient which indicates the relative amount of the free host memory
that each VM should take. Suppose for instance you have four VMs, three of them
running an HTTP server and the last one is a database server. To cache more
database blocks in the database server RAM, you would like to prioritize the
database VM when spare RAM is available. For this you assign a Shares property
of 3000 to the database VM, leaving the other VMs to the Shares default setting
of 1000. The host server has 32GB of RAM, and is currently using 16GB, leaving 32
* 80/100 - 16 = 9GB RAM to be allocated to the VMs on top of their configured
minimum memory amount. The database VM will benefit from 9 * 3000 / (3000
+ 1000 + 1000 + 1000) = 4.5 GB extra RAM and each HTTP server from 1.5 GB.</p></div>
<div class="paragraph">
<p>All Linux distributions released after 2010 have the balloon kernel driver
included. For Windows OSes, the balloon driver needs to be added manually and can
incur a slowdown of the guest, so we don’t recommend using it on critical
systems.</p></div>
<div class="paragraph">
<p>When allocating RAM to your VMs, a good rule of thumb is always to leave 1GB
of RAM available to the host.</p></div>
</div>
<div class="sect3">
<h4 id="qm_network_device">10.2.7. Network Device
 <a class="headerlink" href="#qm_network_device" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-create-vm-network.png">
<img src="images/screenshot/gui-create-vm-network.png" alt="screenshot/gui-create-vm-network.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>Each VM can have many <em>Network interface controllers</em> (NIC), of four different
types:</p></div>
<div class="ulist"><ul>
<li>
<p>
<strong>Intel E1000</strong> is the default, and emulates an Intel Gigabit network card.
</p>
</li>
<li>
<p>
the <strong>VirtIO</strong> paravirtualized NIC should be used if you aim for maximum
performance. Like all VirtIO devices, the guest OS should have the proper driver
installed.
</p>
</li>
<li>
<p>
the <strong>Realtek 8139</strong> emulates an older 100 MB/s network card, and should
only be used when emulating older operating systems ( released before 2002 )
</p>
</li>
<li>
<p>
the <strong>vmxnet3</strong> is another paravirtualized device, which should only be used
when importing a VM from another hypervisor.
</p>
</li>
</ul></div>
<div class="paragraph">
<p>Proxmox VE will generate for each NIC a random <strong>MAC address</strong>, so that your VM is
addressable on Ethernet networks.</p></div>
<div class="paragraph">
<p>The NIC you added to the VM can follow one of two different models:</p></div>
<div class="ulist"><ul>
<li>
<p>
in the default <strong>Bridged mode</strong> each virtual NIC is backed on the host by a
<em>tap device</em>, ( a software loopback device simulating an Ethernet NIC ). This
tap device is added to a bridge, by default vmbr0 in Proxmox VE. In this mode, VMs
have direct access to the Ethernet LAN on which the host is located.
</p>
</li>
<li>
<p>
in the alternative <strong>NAT mode</strong>, each virtual NIC will only communicate with
the QEMU user networking stack, where a built-in router and DHCP server can
provide network access. This built-in DHCP will serve addresses in the private
10.0.2.0/24 range. The NAT mode is much slower than the bridged mode, and
should only be used for testing. This mode is only available via CLI or the API,
but not via the web UI.
</p>
</li>
</ul></div>
<div class="paragraph">
<p>You can also skip adding a network device when creating a VM by selecting <strong>No
network device</strong>.</p></div>
<div class="paragraph">
<p>You can overwrite the <strong>MTU</strong> setting for each VM network device. The option
<span class="monospaced">mtu=1</span> represents a special case, in which the MTU value will be inherited
from the underlying bridge.
This option is only available for <strong>VirtIO</strong> network devices.</p></div>
<div class="paragraph">
<div class="title">Multiqueue</div><p>If you are using the VirtIO driver, you can optionally activate the
<strong>Multiqueue</strong> option. This option allows the guest OS to process networking
packets using multiple virtual CPUs, providing an increase in the total number
of packets transferred.</p></div>
<div class="paragraph">
<p>When using the VirtIO driver with Proxmox VE, each NIC network queue is passed to the
host kernel, where the queue will be processed by a kernel thread spawned by the
vhost driver. With this option activated, it is possible to pass <em>multiple</em>
network queues to the host kernel for each NIC.</p></div>
<div class="paragraph">
<p>When using Multiqueue, it is recommended to set it to a value equal to the
number of vCPUs of your guest. Remember that the number of vCPUs is the number
of sockets times the number of cores configured for the VM. You also need to set
the number of multi-purpose channels on each VirtIO NIC in the VM with this
ethtool command:</p></div>
<div class="paragraph">
<p><span class="monospaced">ethtool -L ens1 combined X</span></p></div>
<div class="paragraph">
<p>where X is the number of the number of vCPUs of the VM.</p></div>
<div class="paragraph">
<p>To configure a Windows guest for Multiqueue install the
<a href="https://pve.proxmox.com/wiki/Windows_VirtIO_Drivers">Redhat VirtIO Ethernet
Adapter drivers</a>, then adapt the NIC’s configuration as follows. Open the
device manager, right click the NIC under "Network adapters", and select
"Properties". Then open the "Advanced" tab and select "Receive Side Scaling"
from the list on the left. Make sure it is set to "Enabled". Next, navigate to
"Maximum number of RSS Queues" in the list and set it to the number of vCPUs of
your VM. Once you verified that the settings are correct, click "OK" to confirm
them.</p></div>
<div class="paragraph">
<p>You should note that setting the Multiqueue parameter to a value greater
than one will increase the CPU load on the host and guest systems as the
traffic increases. We recommend to set this option only when the VM has to
process a great number of incoming connections, such as when the VM is running
as a router, reverse proxy or a busy HTTP server doing long polling.</p></div>
</div>
<div class="sect3">
<h4 id="qm_display">10.2.8. Display
 <a class="headerlink" href="#qm_display" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>QEMU can virtualize a few types of VGA hardware. Some examples are:</p></div>
<div class="ulist"><ul>
<li>
<p>
<strong>std</strong>, the default, emulates a card with Bochs VBE extensions.
</p>
</li>
<li>
<p>
<strong>cirrus</strong>, this was once the default, it emulates a very old hardware module
with all its problems. This display type should only be used if really
necessary <span class="footnote" data-note="<a href=&quot;https://www.kraxel.org/blog/2014/10/qemu-using-cirrus-considered-harmful/&quot;>https://www.kraxel.org/blog/2014/10/qemu-using-cirrus-considered-harmful/</a>
qemu: using cirrus considered harmful">[<a id="_footnoteref_42" href="#_footnote_42" title="View footnote" class="footnote">42</a>]</span>, for example, if using Windows XP or
earlier
</p>
</li>
<li>
<p>
<strong>vmware</strong>, is a VMWare SVGA-II compatible adapter.
</p>
</li>
<li>
<p>
<strong>qxl</strong>, is the QXL paravirtualized graphics card. Selecting this also
enables <a href="https://www.spice-space.org/">SPICE</a> (a remote viewer protocol) for the
VM.
</p>
</li>
<li>
<p>
<strong>virtio-gl</strong>, often named VirGL is a virtual 3D GPU for use inside VMs that
  can offload workloads to the host GPU without requiring special (expensive)
  models and drivers and neither binding the host GPU completely, allowing
  reuse between multiple guests and or the host.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">VirGL support needs some extra libraries that aren’t installed by
default due to being relatively big and also not available as open source for
all GPU models/vendors. For most setups you’ll just need to do:
<span class="monospaced">apt install libgl1 libegl1</span></td>
</tr></tbody></table>
</div>
</li>
</ul></div>
<div class="paragraph">
<p>You can edit the amount of memory given to the virtual GPU, by setting
the <em>memory</em> option. This can enable higher resolutions inside the VM,
especially with SPICE/QXL.</p></div>
<div class="paragraph">
<p>As the memory is reserved by display device, selecting Multi-Monitor mode
for SPICE (such as <span class="monospaced">qxl2</span> for dual monitors) has some implications:</p></div>
<div class="ulist"><ul>
<li>
<p>
Windows needs a device for each monitor, so if your <em>ostype</em> is some
version of Windows, Proxmox VE gives the VM an extra device per monitor.
Each device gets the specified amount of memory.
</p>
</li>
<li>
<p>
Linux VMs, can always enable more virtual monitors, but selecting
a Multi-Monitor mode multiplies the memory given to the device with
the number of monitors.
</p>
</li>
</ul></div>
<div class="paragraph">
<p>Selecting <span class="monospaced">serialX</span> as display <em>type</em> disables the VGA output, and redirects
the Web Console to the selected serial port. A configured display <em>memory</em>
setting will be ignored in that case.</p></div>
<div class="paragraph">
<div class="title">VNC clipboard</div><p>You can enable the VNC clipboard by setting <span class="monospaced">clipboard</span> to <span class="monospaced">vnc</span>.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># qm set &lt;vmid&gt; -vga &lt;displaytype&gt;,clipboard=vnc</pre>
</div></div>
<div class="paragraph">
<p>In order to use the clipboard feature, you must first install the
SPICE guest tools. On Debian-based distributions, this can be achieved
by installing <span class="monospaced">spice-vdagent</span>. For other Operating Systems search for it
in the offical repositories or see: <a href="https://www.spice-space.org/download.html">https://www.spice-space.org/download.html</a></p></div>
<div class="paragraph">
<p>Once you have installed the spice guest tools, you can use the VNC clipboard
function (e.g. in the noVNC console panel). However, if you’re using
SPICE, virtio or virgl, you’ll need to choose which clipboard to use.
This is because the default <strong>SPICE</strong> clipboard will be replaced by the
<strong>VNC</strong> clipboard, if <span class="monospaced">clipboard</span> is set to <span class="monospaced">vnc</span>.</p></div>
</div>
<div class="sect3">
<h4 id="qm_usb_passthrough">10.2.9. USB Passthrough
 <a class="headerlink" href="#qm_usb_passthrough" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>There are two different types of USB passthrough devices:</p></div>
<div class="ulist"><ul>
<li>
<p>
Host USB passthrough
</p>
</li>
<li>
<p>
SPICE USB passthrough
</p>
</li>
</ul></div>
<div class="paragraph">
<p>Host USB passthrough works by giving a VM a USB device of the host.
This can either be done via the vendor- and product-id, or
via the host bus and port.</p></div>
<div class="paragraph">
<p>The vendor/product-id looks like this: <strong>0123:abcd</strong>,
where <strong>0123</strong> is the id of the vendor, and <strong>abcd</strong> is the id
of the product, meaning two pieces of the same usb device
have the same id.</p></div>
<div class="paragraph">
<p>The bus/port looks like this: <strong>1-2.3.4</strong>, where <strong>1</strong> is the bus
and <strong>2.3.4</strong> is the port path. This represents the physical
ports of your host (depending of the internal order of the
usb controllers).</p></div>
<div class="paragraph">
<p>If a device is present in a VM configuration when the VM starts up,
but the device is not present in the host, the VM can boot without problems.
As soon as the device/port is available in the host, it gets passed through.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">Using this kind of USB passthrough means that you cannot move
a VM online to another host, since the hardware is only available
on the host the VM is currently residing.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>The second type of passthrough is SPICE USB passthrough. If you add one or more
SPICE USB ports to your VM, you can dynamically pass a local USB device from
your SPICE client through to the VM. This can be useful to redirect an input
device or hardware dongle temporarily.</p></div>
<div class="paragraph">
<p>It is also possible to map devices on a cluster level, so that they can be
properly used with HA and hardware changes are detected and non root users
can configure them. See <a href="#resource_mapping">Resource Mapping</a>
for details on that.</p></div>
</div>
<div class="sect3">
<h4 id="qm_bios_and_uefi">10.2.10. BIOS and UEFI
 <a class="headerlink" href="#qm_bios_and_uefi" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>In order to properly emulate a computer, QEMU needs to use a firmware.
Which, on common PCs often known as BIOS or (U)EFI, is executed as one of the
first steps when booting a VM. It is responsible for doing basic hardware
initialization and for providing an interface to the firmware and hardware for
the operating system. By default QEMU uses <strong>SeaBIOS</strong> for this, which is an
open-source, x86 BIOS implementation. SeaBIOS is a good choice for most
standard setups.</p></div>
<div class="paragraph">
<p>Some operating systems (such as Windows 11) may require use of an UEFI
compatible implementation. In such cases, you must use <strong>OVMF</strong> instead,
which is an open-source UEFI implementation. <span class="footnote" data-note="See the OVMF Project <a href=&quot;https://github.com/tianocore/tianocore.github.io/wiki/OVMF&quot;>https://github.com/tianocore/tianocore.github.io/wiki/OVMF</a>">[<a id="_footnoteref_43" href="#_footnote_43" title="View footnote" class="footnote">43</a>]</span></p></div>
<div class="paragraph">
<p>There are other scenarios in which the SeaBIOS may not be the ideal firmware to
boot from, for example if you want to do VGA passthrough. <span class="footnote" data-note="Alex
Williamson has a good blog entry about this
<a href=&quot;https://vfio.blogspot.co.at/2014/08/primary-graphics-assignment-without-vga.html&quot;>https://vfio.blogspot.co.at/2014/08/primary-graphics-assignment-without-vga.html</a>">[<a id="_footnoteref_44" href="#_footnote_44" title="View footnote" class="footnote">44</a>]</span></p></div>
<div class="paragraph">
<p>If you want to use OVMF, there are several things to consider:</p></div>
<div class="paragraph">
<p>In order to save things like the <strong>boot order</strong>, there needs to be an EFI Disk.
This disk will be included in backups and snapshots, and there can only be one.</p></div>
<div class="paragraph">
<p>You can create such a disk with the following command:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># qm set &lt;vmid&gt; -efidisk0 &lt;storage&gt;:1,format=&lt;format&gt;,efitype=4m,pre-enrolled-keys=1</pre>
</div></div>
<div class="paragraph">
<p>Where <strong>&lt;storage&gt;</strong> is the storage where you want to have the disk, and
<strong>&lt;format&gt;</strong> is a format which the storage supports. Alternatively, you can
create such a disk through the web interface with <em>Add</em> → <em>EFI Disk</em> in the
hardware section of a VM.</p></div>
<div class="paragraph">
<p>The <strong>efitype</strong> option specifies which version of the OVMF firmware should be
used. For new VMs, this should always be <em>4m</em>, as it supports Secure Boot and
has more space allocated to support future development (this is the default in
the GUI).</p></div>
<div class="paragraph">
<p><strong>pre-enroll-keys</strong> specifies if the efidisk should come pre-loaded with
distribution-specific and Microsoft Standard Secure Boot keys. It also enables
Secure Boot by default (though it can still be disabled in the OVMF menu within
the VM).</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">If you want to start using Secure Boot in an existing VM (that still uses
a <em>2m</em> efidisk), you need to recreate the efidisk. To do so, delete the old one
(<span class="monospaced">qm set &lt;vmid&gt; -delete efidisk0</span>) and add a new one as described above. This
will reset any custom configurations you have made in the OVMF menu!</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>When using OVMF with a virtual display (without VGA passthrough),
you need to set the client resolution in the OVMF menu (which you can reach
with a press of the ESC button during boot), or you have to choose
SPICE as the display type.</p></div>
</div>
<div class="sect3">
<h4 id="qm_tpm">10.2.11. Trusted Platform Module (TPM)
 <a class="headerlink" href="#qm_tpm" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>A <strong>Trusted Platform Module</strong> is a device which stores secret data - such as
encryption keys - securely and provides tamper-resistance functions for
validating system boot.</p></div>
<div class="paragraph">
<p>Certain operating systems (such as Windows 11) require such a device to be
attached to a machine (be it physical or virtual).</p></div>
<div class="paragraph">
<p>A TPM is added by specifying a <strong>tpmstate</strong> volume. This works similar to an
efidisk, in that it cannot be changed (only removed) once created. You can add
one via the following command:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># qm set &lt;vmid&gt; -tpmstate0 &lt;storage&gt;:1,version=&lt;version&gt;</pre>
</div></div>
<div class="paragraph">
<p>Where <strong>&lt;storage&gt;</strong> is the storage you want to put the state on, and <strong>&lt;version&gt;</strong>
is either <em>v1.2</em> or <em>v2.0</em>. You can also add one via the web interface, by
choosing <em>Add</em> → <em>TPM State</em> in the hardware section of a VM.</p></div>
<div class="paragraph">
<p>The <em>v2.0</em> TPM spec is newer and better supported, so unless you have a specific
implementation that requires a <em>v1.2</em> TPM, it should be preferred.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Compared to a physical TPM, an emulated one does <strong>not</strong> provide any real
security benefits. The point of a TPM is that the data on it cannot be modified
easily, except via commands specified as part of the TPM spec. Since with an
emulated device the data storage happens on a regular volume, it can potentially
be edited by anyone with access to it.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="qm_ivshmem">10.2.12. Inter-VM shared memory
 <a class="headerlink" href="#qm_ivshmem" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>You can add an Inter-VM shared memory device (<span class="monospaced">ivshmem</span>), which allows one to
share memory between the host and a guest, or also between multiple guests.</p></div>
<div class="paragraph">
<p>To add such a device, you can use <span class="monospaced">qm</span>:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># qm set &lt;vmid&gt; -ivshmem size=32,name=foo</pre>
</div></div>
<div class="paragraph">
<p>Where the size is in MiB. The file will be located under
<span class="monospaced">/dev/shm/pve-shm-$name</span> (the default name is the vmid).</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Currently the device will get deleted as soon as any VM using it got
shutdown or stopped. Open connections will still persist, but new connections
to the exact same device cannot be made anymore.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>A use case for such a device is the Looking Glass
<span class="footnote" data-note="Looking Glass: <a href=&quot;https://looking-glass.io/&quot;>https://looking-glass.io/</a>">[<a id="_footnoteref_45" href="#_footnote_45" title="View footnote" class="footnote">45</a>]</span> project, which enables high
performance, low-latency display mirroring between host and guest.</p></div>
</div>
<div class="sect3">
<h4 id="qm_audio_device">10.2.13. Audio Device
 <a class="headerlink" href="#qm_audio_device" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>To add an audio device run the following command:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>qm set &lt;vmid&gt; -audio0 device=&lt;device&gt;</pre>
</div></div>
<div class="paragraph">
<p>Supported audio devices are:</p></div>
<div class="ulist"><ul>
<li>
<p>
<span class="monospaced">ich9-intel-hda</span>: Intel HD Audio Controller, emulates ICH9
</p>
</li>
<li>
<p>
<span class="monospaced">intel-hda</span>: Intel HD Audio Controller, emulates ICH6
</p>
</li>
<li>
<p>
<span class="monospaced">AC97</span>: Audio Codec '97, useful for older operating systems like Windows XP
</p>
</li>
</ul></div>
<div class="paragraph">
<p>There are two backends available:</p></div>
<div class="ulist"><ul>
<li>
<p>
<em>spice</em>
</p>
</li>
<li>
<p>
<em>none</em>
</p>
</li>
</ul></div>
<div class="paragraph">
<p>The <em>spice</em> backend can be used in combination with <a href="#qm_display">SPICE</a> while
the <em>none</em> backend can be useful if an audio device is needed in the VM for some
software to work. To use the physical audio device of the host use device
passthrough (see <a href="#qm_pci_passthrough">PCI Passthrough</a> and
<a href="#qm_usb_passthrough">USB Passthrough</a>). Remote protocols like Microsoft’s RDP
have options to play sound.</p></div>
</div>
<div class="sect3">
<h4 id="qm_virtio_rng">10.2.14. VirtIO RNG
 <a class="headerlink" href="#qm_virtio_rng" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>A RNG (Random Number Generator) is a device providing entropy (<em>randomness</em>) to
a system. A virtual hardware-RNG can be used to provide such entropy from the
host system to a guest VM. This helps to avoid entropy starvation problems in
the guest (a situation where not enough entropy is available and the system may
slow down or run into problems), especially during the guests boot process.</p></div>
<div class="paragraph">
<p>To add a VirtIO-based emulated RNG, run the following command:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>qm set &lt;vmid&gt; -rng0 source=&lt;source&gt;[,max_bytes=X,period=Y]</pre>
</div></div>
<div class="paragraph">
<p><span class="monospaced">source</span> specifies where entropy is read from on the host and has to be one of
the following:</p></div>
<div class="ulist"><ul>
<li>
<p>
<span class="monospaced">/dev/urandom</span>: Non-blocking kernel entropy pool (preferred)
</p>
</li>
<li>
<p>
<span class="monospaced">/dev/random</span>: Blocking kernel pool (not recommended, can lead to entropy
  starvation on the host system)
</p>
</li>
<li>
<p>
<span class="monospaced">/dev/hwrng</span>: To pass through a hardware RNG attached to the host (if multiple
  are available, the one selected in
  <span class="monospaced">/sys/devices/virtual/misc/hw_random/rng_current</span> will be used)
</p>
</li>
</ul></div>
<div class="paragraph">
<p>A limit can be specified via the <span class="monospaced">max_bytes</span> and <span class="monospaced">period</span> parameters, they are
read as <span class="monospaced">max_bytes</span> per <span class="monospaced">period</span> in milliseconds. However, it does not represent
a linear relationship: 1024B/1000ms would mean that up to 1 KiB of data becomes
available on a 1 second timer, not that 1 KiB is streamed to the guest over the
course of one second. Reducing the <span class="monospaced">period</span> can thus be used to inject entropy
into the guest at a faster rate.</p></div>
<div class="paragraph">
<p>By default, the limit is set to 1024 bytes per 1000 ms (1 KiB/s). It is
recommended to always use a limiter to avoid guests using too many host
resources. If desired, a value of <em>0</em> for <span class="monospaced">max_bytes</span> can be used to disable
all limits.</p></div>
</div>
<div class="sect3">
<h4 id="qm_bootorder">10.2.15. Device Boot Order
 <a class="headerlink" href="#qm_bootorder" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>QEMU can tell the guest which devices it should boot from, and in which order.
This can be specified in the config via the <span class="monospaced">boot</span> property, for example:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>boot: order=scsi0;net0;hostpci0</pre>
</div></div>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-qemu-edit-bootorder.png">
<img src="images/screenshot/gui-qemu-edit-bootorder.png" alt="screenshot/gui-qemu-edit-bootorder.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>This way, the guest would first attempt to boot from the disk <span class="monospaced">scsi0</span>, if that
fails, it would go on to attempt network boot from <span class="monospaced">net0</span>, and in case that
fails too, finally attempt to boot from a passed through PCIe device (seen as
disk in case of NVMe, otherwise tries to launch into an option ROM).</p></div>
<div class="paragraph">
<p>On the GUI you can use a drag-and-drop editor to specify the boot order, and use
the checkbox to enable or disable certain devices for booting altogether.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">If your guest uses multiple disks to boot the OS or load the bootloader,
all of them must be marked as <em>bootable</em> (that is, they must have the checkbox
enabled or appear in the list in the config) for the guest to be able to boot.
This is because recent SeaBIOS and OVMF versions only initialize disks if they
are marked <em>bootable</em>.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>In any case, even devices not appearing in the list or having the checkmark
disabled will still be available to the guest, once it’s operating system has
booted and initialized them. The <em>bootable</em> flag only affects the guest BIOS and
bootloader.</p></div>
</div>
<div class="sect3">
<h4 id="qm_startup_and_shutdown">10.2.16. Automatic Start and Shutdown of Virtual Machines
 <a class="headerlink" href="#qm_startup_and_shutdown" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>After creating your VMs, you probably want them to start automatically
when the host system boots. For this you need to select the option <em>Start at
boot</em> from the <em>Options</em> Tab of your VM in the web interface, or set it with
the following command:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># qm set &lt;vmid&gt; -onboot 1</pre>
</div></div>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-qemu-edit-start-order.png">
<img src="images/screenshot/gui-qemu-edit-start-order.png" alt="screenshot/gui-qemu-edit-start-order.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<div class="title">Start and Shutdown Order</div><p>In some case you want to be able to fine tune the boot order of your
VMs, for instance if one of your VM is providing firewalling or DHCP
to other guest systems.  For this you can use the following
parameters:</p></div>
<div class="ulist"><ul>
<li>
<p>
<strong>Start/Shutdown order</strong>: Defines the start order priority. For example, set it
to 1 if you want the VM to be the first to be started. (We use the reverse
startup order for shutdown, so a machine with a start order of 1 would be the
last to be shut down). If multiple VMs have the same order defined on a host,
they will additionally be ordered by <em>VMID</em> in ascending order.
</p>
</li>
<li>
<p>
<strong>Startup delay</strong>: Defines the interval between this VM start and subsequent
VMs starts. For example, set it to 240 if you want to wait 240 seconds before
starting other VMs.
</p>
</li>
<li>
<p>
<strong>Shutdown timeout</strong>: Defines the duration in seconds Proxmox VE should wait
for the VM to be offline after issuing a shutdown command. By default this
value is set to 180, which means that Proxmox VE will issue a shutdown request and
wait 180 seconds for the machine to be offline. If the machine is still online
after the timeout it will be stopped forcefully.
</p>
</li>
</ul></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">VMs managed by the HA stack do not follow the <em>start on boot</em> and
<em>boot order</em> options currently. Those VMs will be skipped by the startup and
shutdown algorithm as the HA manager itself ensures that VMs get started and
stopped.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>Please note that machines without a Start/Shutdown order parameter will always
start after those where the parameter is set. Further, this parameter can only
be enforced between virtual machines running on the same host, not
cluster-wide.</p></div>
<div class="paragraph">
<p>If you require a delay between the host boot and the booting of the first VM,
see the section on <a href="#first_guest_boot_delay">Proxmox VE Node Management</a>.</p></div>
</div>
<div class="sect3">
<h4 id="qm_qemu_agent">10.2.17. QEMU Guest Agent
 <a class="headerlink" href="#qm_qemu_agent" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The QEMU Guest Agent is a service which runs inside the VM, providing a
communication channel between the host and the guest. It is used to exchange
information and allows the host to issue commands to the guest.</p></div>
<div class="paragraph">
<p>For example, the IP addresses in the VM summary panel are fetched via the guest
agent.</p></div>
<div class="paragraph">
<p>Or when starting a backup, the guest is told via the guest agent to sync
outstanding writes via the <em>fs-freeze</em> and <em>fs-thaw</em> commands.</p></div>
<div class="paragraph">
<p>For the guest agent to work properly the following steps must be taken:</p></div>
<div class="ulist"><ul>
<li>
<p>
install the agent in the guest and make sure it is running
</p>
</li>
<li>
<p>
enable the communication via the agent in Proxmox VE
</p>
</li>
</ul></div>
<div class="sect4">
<h5 id="_install_guest_agent">Install Guest Agent
 <a class="headerlink" href="#_install_guest_agent" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>For most Linux distributions, the guest agent is available. The package is
usually named <span class="monospaced">qemu-guest-agent</span>.</p></div>
<div class="paragraph">
<p>For Windows, it can be installed from the
<a href="https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virtio/virtio-win.iso">Fedora
VirtIO driver ISO</a>.</p></div>
</div>
<div class="sect4">
<h5 id="qm_qga_enable">Enable Guest Agent Communication
 <a class="headerlink" href="#qm_qga_enable" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Communication from Proxmox VE with the guest agent can be enabled in the VM’s
<strong>Options</strong> panel. A fresh start of the VM is necessary for the changes to take
effect.</p></div>
</div>
<div class="sect4">
<h5 id="qm_qga_auto_trim">Automatic TRIM Using QGA
 <a class="headerlink" href="#qm_qga_auto_trim" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>It is possible to enable the <em>Run guest-trim</em> option. With this enabled,
Proxmox VE will issue a trim command to the guest after the following
operations that have the potential to write out zeros to the storage:</p></div>
<div class="ulist"><ul>
<li>
<p>
moving a disk to another storage
</p>
</li>
<li>
<p>
live migrating a VM to another node with local storage
</p>
</li>
</ul></div>
<div class="paragraph">
<p>On a thin provisioned storage, this can help to free up unused space.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">There is a caveat with ext4 on Linux, because it uses an in-memory
optimization to avoid issuing duplicate TRIM requests. Since the guest doesn’t
know about the change in the underlying storage, only the first guest-trim will
run as expected. Subsequent ones, until the next reboot, will only consider
parts of the filesystem that changed since then.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect4">
<h5 id="qm_qga_fsfreeze">Filesystem Freeze &amp; Thaw on Backup
 <a class="headerlink" href="#qm_qga_fsfreeze" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>By default, guest filesystems are synced via the <em>fs-freeze</em> QEMU Guest Agent
Command when a backup is performed, to provide consistency.</p></div>
<div class="paragraph">
<p>On Windows guests, some applications might handle consistent backups themselves
by hooking into the Windows VSS (Volume Shadow Copy Service) layer, a
<em>fs-freeze</em> then might interfere with that. For example, it has been observed
that calling <em>fs-freeze</em> with some SQL Servers triggers VSS to call the SQL
Writer VSS module in a mode that breaks the SQL Server backup chain for
differential backups.</p></div>
<div class="paragraph">
<p>For such setups you can configure Proxmox VE to not issue a freeze-and-thaw cycle on
backup by setting the <span class="monospaced">freeze-fs-on-backup</span> QGA option to <span class="monospaced">0</span>. This can also be
done via the GUI with the <em>Freeze/thaw guest filesystems on backup for
consistency</em> option.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Important" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAALa0lEQVRogdWZa2wc1RXHfzM7O/te
P9e1vXHSmEdjx3YeDkkaF6REKRRCEDSEFNkRjdSWSsgC2iqoRWqLQKiqIBg1NOQDiMeHtkQIQkRR
S9S4aWwgCQoUgl232KkT28J21l6vd3d2dx79sDuTXdtre03Uqlc62tl53Pv/n3PuOffcKxiGwf9z
E//XAL5sk65WR0a6AaDrOjMtKwgCoigiCAKCIAhXa9wvTcAEHhoc5KMjR7jw3nucf+MN67k/GCS4
YQOr77yTpl27kGUZm81mXC0ywlLnQDbwEwcO0P2b3yz4jV5RwZq2Nm5/5BH8fj+SJGGz2b4UiSUR
MAzD0DSNEwcO8MdHHln8d0AcsAUCfOe551h7yy04HA7sdvuSrVEwAV3XjdDgIC/v2cOl06dznqWA
JCAAMvn9MwlEgF1PP8037rsPt9uN3W5fkjUKIqDrujF+4QLP3XQTU0NDOc8UIAa4y8sJtrQQqK/H
7XQSfv99Pn/nHQRRxNB16/0EEAUatm9n76uv4vP5cDgcBZNYNIF84I0MEK20lIZvf5sNe/dSWlqK
1+vFxBEZHubT3/+evqNHmR4dJRWP55BevW0bra+8gt/vx+l0IknSokksikA+8ElgClixYwc3799P
ZWUlTqczbz+x0VFOPvEEQ6dPE/niCwxdt0jUbd1K68svU1xcXBCJBQnMBz5eWkrL/v2s++Y3KSsr
QxQXlxc7f/5zBjo7CQ8NoadSKKQn943t7dyyf79FYjHuNC8BwzAMVVV56Z57+OzNN3PAJ8vK+FZH
B2s2b55X6/nahRMnOP7TnxLOKEXJyO2/+hVf37uXoqIinE6nmfzyksirMjNUvvGjH+WAVwG1vJxb
Dhxg1bp1uFwuMwQWJCu3bWPN3r24SkoAcGbA/OXXv2bws8+IRqOkUqlZGX1RBAzDMHRd51+nTvHe
wYNX7gPTQMO+faxav56SkpIlgTdlU3s7TXv2IGUs6AGioRDH2tuZmJggFouhqirGPCzyEUBVVX7X
1pZzPwI0fP/7bNmzh7KysrzADh8+zP3338++ffs4ePBg3vdEUWTDD3/IsuZmRLsdW4bEpd5eOp99
lnA4jKIoaJqWl8QsAqb2Txw4kDNp40BxXR0bWlspLy/PC+qZZ55hdHQUSZIYHByko6ODxx57bBZw
cy3kq6xk6y9/ib+qCgQBmbQ7nT50iIs9PUSjUZLJJIZhzEliLgKMDQzwp5/9zLqnkQ51W3/xCwKB
AE6nMy+B3t5exsfH6ezspLe3l3g8zuHDh2cBz5bKpibWtrXhLi0FwA3owNsPP8zExATxeBxVVecy
QC4Bc+L+taNjluvU33031dddN6/2BUGgpKSE/v5+JicncweaA3i2tDz8MIHrr0eU0gsQBzDW10ff
yZNEIhESicScrpRDQNd1xgYG+OC3v7XuJQBXIMDWn/yEioqKeUGMjY0hCAKKoszS1MjIyLzfAmx/
4gmKli0DQcBJOmicfOopJicnicfjJoG5LWBq/+Szz+a8oABNe/bgdrvndR1BEKiqqiIejxOLxWYR
CAaDC0alqjVrqGxowFNWhkDalUKff07f3/5mWSFTLFksLAK6rjN+4UKO9lXAUV7O2tbWeaNOthZl
WZ4FHlh0aN3y4IPIHk+6L8AGvNfRQTgctuZCthVEU/u6rnPutddyBk0A9bt3W4us+QY2fbyurg6v
15vTzw033LBoAsHmZoLNzdgcDgBcQKi/n391dTE9PU0ymcyxgkmAVCrF+4cO5WjfFgiwfoGwOTOy
lJWVUVtba1mkUAKCILClvR1veTkA9owVPnzllZy8YFnA1H5/dzfTw8PWgziwcutWfD4fsiwvCNyU
xsZG3G43lZWVVl933XVXQQSq161j+ebNlhXcwBdnzxIOh60lhmkFEUDTNP7++uvWgBrp6mrNHBk3
H/BsAtFoFEdm8GAwyMaNGwsiIAgCjbt346uosKwgAp8cPWolNj1THInmsqHn6FGLQApY1thIxcqV
+Hy+WX6+0ETesWMHLpcLWZZ54IEHCgYvCAKrbrsNyem0JrQEXMzkhKzlBaKu61z+97+ZHhnJmbwr
tm2zJu5CWp8pra2tNDU1sXPnTtrb25dEQBAEvrplC57MXJCB8XPniEQixONxa6Uq6brOQHd3jvvo
QP2ttxZUpMxsL7zwwpK+AyxLXrt9Oxe6uxFEEXvGZc4fO0bVD35AMplM1wuapjGYtbugAp5AgKLq
aquu/W/JTEs37dqFqihWzSADk8PD1jzQNC3tQtmrThUINDbm+H4h0tfXx6OPPsodd9xBW1sbPT09
BQPPluu2b7fWRxIQ6usjFotZBCRN06wZbbZAfT1FRUWWKQtpzz//PMPDw/T09DA1NcXo6CjHjx+f
11Xma95AwHJjEdB0HUVRrhCYuRGrAiXBoFUqFtq6u7sZHx9nJBMUurq65uxnsX0Hm5v56A9/ANIJ
LXrpEolEwgqlUqZQyPlo6OOPlzx5ly1bxnBWQmxpackBW6hSBEC02axrpqasNZGmaek8kN0kYPDj
j1FVdUlz4Mknn6Surs4Cb5aUhYZiUy59+GGOgm3V1WiadiUPCILAV9autV5wAaM9PXQeOWJprBBp
aGjgxIkTpFIpOjs7aWxsXBJwUwZOnUKZmgLS2zl4vUjSlV1XURRFqtavv3IDKAVe/O53efPxx69a
SCxUBs+c4dV77+Vyfz+xy5eBdG3iX7ECm81m9S2JokiwuZmiujrCPT1AuqgOAqcef5zzL77Imt27
uemhhyipqVnYZ5cw8bPbR0eOcOall7g8MEB4aIhEJAJk9qOAwKZNyLJsHpQgRKNRIxQK0XPmDH/+
3vdQQ6G8na/YtIk199zDypYWVmzceFWAT1y8yMWzZzn/9tv0vfsudqeTlKIQHRvL2QSOAqU7d/K1
W2/l2muvpba2Nl3iJhIJIxKJMDw8TO/Zs3zw4x+jTUzkHVCw2fCUleHw+QiuXYu/upqqhgb8VVUY
hkFNczPFweCs7z556y3r+vyxYwiiyD+PHwdBwNB1krEYhqYRC4UwdB2D9LJmmnRtXHz77VTfeCPL
ly+ntraWmpqa9MaaqqqGoiiEQiGGhoYYHBzk00OHiHR2LkqDNlnG7nJhd7nQswqN+ZogiuipFMlY
DDWRgBl5KEF6woqAUFFByc03U756NZWVlSxfvpyamhoCgQAejwdB13VDVVVisRihUIiRkRGGh4cZ
+sc/GO3qIt7VhZGJAle7mVpWuVKDCBng8jXX4N68Gc8111BUVER5eTnV1dVUV1cTCASuFFqGYaDr
upFKpYjH40xMTDA+Ps7Y2Bjj4+OEw2EmenqYPneO1OAgZCWpxTY9S9QMcLhyHGUDRI8HqbISZ309
zoYGnE4nHo8Hv99PaWkpgUCAiooK6/DE4XCkI1HW2a6hqiqKojA9Pc3k5KQl4XCY6elpotFoetuk
v59UKIR66RLToRBJTcMYHUWIRvMSMPO6rbY2/SsIOFatQpYk5NWrkSQJWZZxOBy43W68Xi9+v5+i
oiJKSkooLi7G5/PhdruRZdnads85HzD3hpLJJIqiEIvFiEajRCIRIpEI0WiUWCxGPB631iOKopBK
pazUbi4Oc7Y+snKCzWbDZrMhSRKSJGG3262w6HK5cLlceDwevF4vPp8Pr9eL2+3G5XLhcDjMkxuE
TNibdcBhnv9qmkYqlSKZTJJIJFAUhXg8jqIoKIpCIpGwSCSTSVRVtWSuk3qTiAnebrfngHc4HDid
TpxOJy6Xy7qWZdk6wZzrxCbvCY1JRNd1NE3LAWgSM69ngp+PgCiK1gF3NhFTzHvmO+Y3wCzw8xLI
JpL5tYCZpEyw2f/nWt1ag2UtM0x3Mq1iAp1j+bH0M7KFCGWDnfmbd8AMnuxfUzL/C0rp/wFnFd4n
EQn3XQAAAABJRU5ErkJggg==">
</td>
<td class="content">Disabling this option can potentially lead to backups with inconsistent
filesystems and should therefore only be disabled if you know what you are
doing.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect4">
<h5 id="_troubleshooting_2">Troubleshooting
 <a class="headerlink" href="#_troubleshooting_2" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<div class="title">VM does not shut down</div><p>Make sure the guest agent is installed and running.</p></div>
<div class="paragraph">
<p>Once the guest agent is enabled, Proxmox VE will send power commands like
<em>shutdown</em> via the guest agent. If the guest agent is not running, commands
cannot get executed properly and the shutdown command will run into a timeout.</p></div>
</div>
</div>
<div class="sect3">
<h4 id="qm_spice_enhancements">10.2.18. SPICE Enhancements
 <a class="headerlink" href="#qm_spice_enhancements" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>SPICE Enhancements are optional features that can improve the remote viewer
experience.</p></div>
<div class="paragraph">
<p>To enable them via the GUI go to the <strong>Options</strong> panel of the virtual machine. Run
the following command to enable them via the CLI:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>qm set &lt;vmid&gt; -spice_enhancements foldersharing=1,videostreaming=all</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">To use these features the <a href="#qm_display"><strong>Display</strong></a> of the virtual machine
must be set to SPICE (qxl).</td>
</tr></tbody></table>
</div>
<div class="sect4">
<h5 id="_folder_sharing">Folder Sharing
 <a class="headerlink" href="#_folder_sharing" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Share a local folder with the guest. The <span class="monospaced">spice-webdavd</span> daemon needs to be
installed in the guest. It makes the shared folder available through a local
WebDAV server located at <a href="http://localhost:9843">http://localhost:9843</a>.</p></div>
<div class="paragraph">
<p>For Windows guests the installer for the <em>Spice WebDAV daemon</em> can be downloaded
from the
<a href="https://www.spice-space.org/download.html#windows-binaries">official SPICE website</a>.</p></div>
<div class="paragraph">
<p>Most Linux distributions have a package called <span class="monospaced">spice-webdavd</span> that can be
installed.</p></div>
<div class="paragraph">
<p>To share a folder in Virt-Viewer (Remote Viewer) go to <em>File → Preferences</em>.
Select the folder to share and then enable the checkbox.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Folder sharing currently only works in the Linux version of Virt-Viewer.</td>
</tr></tbody></table>
</div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Caution" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKdUlEQVRoge1Ze1AV1x3+zt279wEi
DWCYGzRVktqa1MEmmtbWR22ncXxUrTDWV/5IqG2wUAUfmUwSkk6V+EAJCEQC6figwZBqZtRxqukf
tebRZBpFG1S0hiRErwiaKK977+6eX//YPXt37+UiIJlMZnpmdnb33PP4vt97z2VEhG9yc3zdAO60
/Z/A192+8QScX8Wifr+fWltbzffU1FT4fD72Vew15ASampqovr4eBw8eNPvGjRuHzMxMmj9//tCT
IKIhu958801yuVwEoNdr48aNNJT7EdHQEdi/fz/JshwTvLiKioqGlMSQLFJfX0+MMRvQsWNSqXLz
H2jez78fRWLTpk1DRuKOF6irqyOn02kDOP8XGdTz+VFSP91Hatu79NRvZ0SR2LJly5CQuKPJtbW1
JEmSDdivHp1AwctHSblYSsqFElLOF5PavIvWZU+LIrF169Y7JjHoibt3744CnzXrIQpeOUbKhVJS
mkpIPV9MyrnNpJwtIuVSNa15/MdRJLZt23ZHJBjRwIu5N954gxYvXgzOudm3aO5E1L5SBNZ1Hoxr
INLAyH6HJxVP/WkXtu9+37ZeSUkJVq9ePagQO+BMXFNTEwV+ybxHsPeVIjg6zwOkgSWNhuOuUUBC
KhxJY0AggDSgpwWbn1mO1csftq1ZUFCA0tLSQZXFA0pkVVVVlJOTA6vWsmY9hF07N0AS4OU48LgR
WLO+FPFxboxMS8G5hlPYmPMI4jwM1P0Jtj67DJyrKHvtNADdjPPz8+FwOCgvL29gmuivrVVUVESF
yqxZD1PoyjFSL+7QHbapmJTml+n9EzVRtl676ZekNKwl5VQ+KSfzKHR2E+UutodYxhiVl5cPyCf6
ZUJlZWWUm5trk/wTi6bgtVdf1M2Gq2DEwUgDNBWhUDBqjfSRwwGoIOEP3c3Y/uxirFw0zibMvLw8
VFZW9tucbkugoqKCVq1aZQO/YslUVJU+D9bVBIDroLgK4hygEDweybYGY8AD6d8yftfASL+j+2O8
9NwiPJk51kYiNzcXO3fu7BeJPgkUFxdTbm6ure/JZdNRub0QrOMcGFcBrgLgAHEAGqD2ID7OZZsz
Ji0RcR6HAV4DkaqPJxXouoSywkz8buF9NhIrV65EVVXVbUnEJHDixAlat26drS9n+U+xo/hZMMNh
yZA+uAbAIKOGMCJlmG3eA/clAYaJEamGBlRdC6QBXZewo3AhVswfbSORk5ODDz/8sE8SvRLw+/1U
XV0dBb5MgOe6LUNIEqpOwgCUlOCESw4HuAfG3KX7B2kAGaTJ0Jro77yI8ucX4DfzRtlIHD58GH6/
PyaJXgk0NjZi79695vuUiffrkr91Vt8cHAwc4FZJGiZEKqDcwsi0ZHP+d749POy8MS7iGtBxARWF
8/CDsQnm3BdeeAHWj6N+EYhspSVbgI5zYYkZGmAwzIEbwA0tUM9NfO+7aeb89HuGRQBWLcSNZxj3
zvOYMzk5NpjBEHi99mXD5lXdjqGDJh42h7CEOXhnK3448X5z/shUb1jSpIEM8yFuzOUcjAsNcnR0
KXdG4MEHH8TEiRPN9z0H3sfRE41m+GNcs9i0LjkGTTcrUoGem5g7/V5z/ohElw4O+jxGYQ3qJqmB
jN/+cbINtcfCJvPYY4+hr3qtVwI+n49lZmaa71fbbuH3hX/FycbPQKSCoIIMQEQcehjVo5Gw8/F3
3cSvH03H2HuHY1icQzcvYWKGIMiITMwQwjtnruPpqk9w/ZZq7p2RkYEJEyaAYrCIWY36/X7KyMhA
W1ub2XfP3Qmo3ZKFaQ+P0gmYIdSoOkU4lWSwxGRAdqHL/xm86k27w0blAw3/PncD6yo/xrsfdZj7
paSkoLGxESNGjAgDZsxWK8X0AZ/Px44fP460tLAzXrnWgUX5r+PYuxf1mG/Gf80sofWy2YuWbg/+
uONtPLHhPRw+0WIHL8hbwG95rcUGPiEhAYcOHUJSUhI456YZRWqiz+8BIqLm5mZMmjQJN27cMPsT
E9zYtWE25k4bYzh3mABIBRwOHPgPx+LH/6yDiXPi6pHZcEpk+I5qmtGpCzdQecCPPUevmevLsowj
R45gypQpcDgc5mVowKaJmBoQTEePHo2GhgaMGxcuum52BLHsqUOoP3oWgLB9Sz7QgpgxJoThw/SS
IutnaXBKZEpcgD998Uvs+3ubDbzT6URNTQ0mTZoEVVWhaRo459A0TeCy4etVA+JH8RvnHK2trZgz
Zw5Onz5tjnPJDrxSOAPLZt1nSVIqwDmIVJw6dx1nLt3A3J/cjZThkhE29cvf3o2KA59ja91lcz3G
GDZv3oylS5fC7XbD6XTC6XRCkiRIktS7Jm5HQLDXNA3Xr19HVlYWPvjgA4vEGLav+RHmTxsJX4rb
8AsO4kbOIG6GzTD4HvzlmB/PVH9q2zc/Px/Z2dnwer1wu92QZRkulwuSJEGWZZMEY8wkEGVCVvCC
AOcciqIgPj4e+/btw9SpU83xqkpYteU9HHm7Bf62rnC2RjikisTFSMPV9m4ceucannvVDn7p0qVY
uHAhgsEggsEgQqFQlAlZnVm0PjOxIKFpGogIqqrC5XKhuroa06dPt4wDcl78F+qPNeNKe6cRoTjs
ZYOKK+3dON7wBQrKm2H5pMbMmTOxZMkSBAIB9PT0oKenB4FAAIqiQFEUU4hRRyq3I2AlIZ4553A6
nSgvL8fs2bNtY9eVNaDub58YmhDZWpe+v70bJ5tu4clt/4WihqU4efJkrFixAkRkAg4Gg1AUBaqq
muDF/pGtX7WQNXeIZ0mSUFRUhAULFtjGPl3ZiJdev4QrbZ16zUQqrrZ342JLN5ZvaEJ3ICz68ePH
IycnB5qmmZIW4EOhUJTZWEKo+Rx1KsEYY9Zk4XA4IEmSKXnOuRkVJElCYWEhvF4v6urqzDVK9n2M
zu4Qnl5+L9q/7MHltgBWbr9kA5+eno7c3FzIsoyI5GrTeCzgosU8VmGMmSGLiEwSsixH2WNBQQEA
2EhUH/wc7330BaaOH479/2zHtS/CFabP58OaNWvg9Xp1EE4nZFmGLMtwu93wer1wuVy2ECpJko2I
iTNWJrZGI+HEwi6FisUVCAQQCASwZ88eVFZW9lk9JicnY/369fD5fPB4PCbo+Ph4k4TH44HL5YLX
64XX6zXH9EKE9aUBRkTEGLNJP1Kd4nI4HMjOzkZ8fDyKi4ttJ3eiJSYmYu3atRg1apQJzuv1wuPx
wOPxwOFwmEBFHnA6nXC5XGYSiywlbns2SnoDANN0hMNZnS0UCpkaOnPmDN566y20tLSY2ktLS8OM
GTOQkpICt9uNuLg4M2EJ8CJZSZJkZmFBwkrAWpH2+3BXEBEkRIhTVdVMOCL5qKqKy5cvQ1EUaJoG
VVWRnKx/Jlpt3eVy2QCKu9C68AFr9o0spwd0Oh1ZYggCIlNaiQktiXdhZgKQABcp3ci7LMtM7B0J
fsAErJoAYItG1mdr+hfvVlACtHDIXkploYXbHvQOikDEe6/ZmjFmAhf9ZvJxOk2Qol88W4kYz32S
GNQfHLG+TyMTkAVEr88xQfVnkBg7GAKixSIykDYQsL21/wFkW/B5QqT9lwAAAABJRU5ErkJggg==">
</td>
<td class="content">Experimental! Currently this feature does not work reliably.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect4">
<h5 id="_video_streaming">Video Streaming
 <a class="headerlink" href="#_video_streaming" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Fast refreshing areas are encoded into a video stream. Two options exist:</p></div>
<div class="ulist"><ul>
<li>
<p>
<strong>all</strong>: Any fast refreshing area will be encoded into a video stream.
</p>
</li>
<li>
<p>
<strong>filter</strong>: Additional filters are used to decide if video streaming should be
  used (currently only small window surfaces are skipped).
</p>
</li>
</ul></div>
<div class="paragraph">
<p>A general recommendation if video streaming should be enabled and which option
to choose from cannot be given. Your mileage may vary depending on the specific
circumstances.</p></div>
</div>
<div class="sect4">
<h5 id="_troubleshooting_3">Troubleshooting
 <a class="headerlink" href="#_troubleshooting_3" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<div class="title">Shared folder does not show up</div><p>Make sure the WebDAV service is enabled and running in the guest. On Windows it
is called <em>Spice webdav proxy</em>. In Linux the name is <em>spice-webdavd</em> but can be
different depending on the distribution.</p></div>
<div class="paragraph">
<p>If the service is running, check the WebDAV server by opening
<a href="http://localhost:9843">http://localhost:9843</a> in a browser in the guest.</p></div>
<div class="paragraph">
<p>It can help to restart the SPICE session.</p></div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="qm_migration">
<span>10.3. Migration</span>
 <a class="headerlink" href="#qm_migration" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-qemu-migrate.png">
<img src="images/screenshot/gui-qemu-migrate.png" alt="screenshot/gui-qemu-migrate.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>If you have a cluster, you can migrate your VM to another host with</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># qm migrate &lt;vmid&gt; &lt;target&gt;</pre>
</div></div>
<div class="paragraph">
<p>There are generally two mechanisms for this</p></div>
<div class="ulist"><ul>
<li>
<p>
Online Migration (aka Live Migration)
</p>
</li>
<li>
<p>
Offline Migration
</p>
</li>
</ul></div>
<div class="sect3">
<h4 id="_online_migration">10.3.1. Online Migration
 <a class="headerlink" href="#_online_migration" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>If your VM is running and no locally bound resources are configured (such as
devices that are passed through), you can initiate a live migration with the <span class="monospaced">--online</span>
flag in the <span class="monospaced">qm migration</span> command evocation. The web interface defaults to
live migration when the VM is running.</p></div>
<div class="sect4">
<h5 id="_how_it_works">How it works
 <a class="headerlink" href="#_how_it_works" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Online migration first starts a new QEMU process on the target host with the
<em>incoming</em> flag, which performs only basic initialization with the guest vCPUs
still paused and then waits for the guest memory and device state data streams
of the source Virtual Machine.
All other resources, such as disks, are either shared or got already sent
before runtime state migration of the VMs begins; so only the memory content
and device state remain to be transferred.</p></div>
<div class="paragraph">
<p>Once this connection is established, the source begins asynchronously sending
the memory content to the target. If the guest memory on the source changes,
those sections are marked dirty and another pass is made to send the guest
memory data.
This loop is repeated until the data difference between running source VM
and incoming target VM is small enough to be sent in a few milliseconds,
because then the source VM can be paused completely, without a user or program
noticing the pause, so that the remaining data can be sent to the target, and
then unpause the targets VM’s CPU to make it the new running VM in well under a
second.</p></div>
</div>
<div class="sect4">
<h5 id="_requirements_2">Requirements
 <a class="headerlink" href="#_requirements_2" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>For Live Migration to work, there are some things required:</p></div>
<div class="ulist"><ul>
<li>
<p>
The VM has no local resources that cannot be migrated. For example,
  PCI or USB devices that are passed through currently block live-migration.
  Local Disks, on the other hand, can be migrated by sending them to the target
  just fine.
</p>
</li>
<li>
<p>
The hosts are located in the same Proxmox VE cluster.
</p>
</li>
<li>
<p>
The hosts have a working (and reliable) network connection between them.
</p>
</li>
<li>
<p>
The target host must have the same, or higher versions of the
  Proxmox VE packages. Although it can sometimes work the other way around, this
  cannot be guaranteed.
</p>
</li>
<li>
<p>
The hosts have CPUs from the same vendor with similar capabilities. Different
  vendor  <strong>might</strong> work depending on the actual models and VMs CPU type
  configured, but it cannot be guaranteed - so please test before deploying
  such a setup in production.
</p>
</li>
</ul></div>
</div>
</div>
<div class="sect3">
<h4 id="_offline_migration">10.3.2. Offline Migration
 <a class="headerlink" href="#_offline_migration" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>If you have local resources, you can still migrate your VMs offline as long as
all disk are on storage defined on both hosts.
Migration then copies the disks to the target host over the network, as with
online migration. Note that any hardware passthrough configuration may need to
be adapted to the device location on the target host.</p></div>
</div>
</div>
<div class="sect2">
<h3 id="qm_copy_and_clone">
<span>10.4. Copies and Clones</span>
 <a class="headerlink" href="#qm_copy_and_clone" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-qemu-full-clone.png">
<img src="images/screenshot/gui-qemu-full-clone.png" alt="screenshot/gui-qemu-full-clone.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>VM installation is usually done using an installation media (CD-ROM)
from the operating system vendor. Depending on the OS, this can be a
time consuming task one might want to avoid.</p></div>
<div class="paragraph">
<p>An easy way to deploy many VMs of the same type is to copy an existing
VM. We use the term <em>clone</em> for such copies, and distinguish between
<em>linked</em> and <em>full</em> clones.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
Full Clone
</dt>
<dd>
<p>
The result of such copy is an independent VM. The
new VM does not share any storage resources with the original.
</p>
<div class="paragraph">
<p>It is possible to select a <strong>Target Storage</strong>, so one can use this to
migrate a VM to a totally different storage. You can also change the
disk image <strong>Format</strong> if the storage driver supports several formats.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">A full clone needs to read and copy all VM image data. This is
usually much slower than creating a linked clone.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>Some storage types allows to copy a specific <strong>Snapshot</strong>, which
defaults to the <em>current</em> VM data. This also means that the final copy
never includes any additional snapshots from the original VM.</p></div>
</dd>
<dt class="hdlist1">
Linked Clone
</dt>
<dd>
<p>
Modern storage drivers support a way to generate fast linked
clones. Such a clone is a writable copy whose initial contents are the
same as the original data. Creating a linked clone is nearly
instantaneous, and initially consumes no additional space.
</p>
<div class="paragraph">
<p>They are called <em>linked</em> because the new image still refers to the
original. Unmodified data blocks are read from the original image, but
modification are written (and afterwards read) from a new
location. This technique is called <em>Copy-on-write</em>.</p></div>
<div class="paragraph">
<p>This requires that the original volume is read-only. With Proxmox VE one
can convert any VM into a read-only <a href="#qm_templates">Template</a>). Such
templates can later be used to create linked clones efficiently.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">You cannot delete an original template while linked clones
exist.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>It is not possible to change the <strong>Target storage</strong> for linked clones,
because this is a storage internal feature.</p></div>
</dd>
</dl></div>
<div class="paragraph">
<p>The <strong>Target node</strong> option allows you to create the new VM on a
different node. The only restriction is that the VM is on shared
storage, and that storage is also available on the target node.</p></div>
<div class="paragraph">
<p>To avoid resource conflicts, all network interface MAC addresses get
randomized, and we generate a new <em>UUID</em> for the VM BIOS (smbios1)
setting.</p></div>
</div>
<div class="sect2">
<h3 id="qm_templates">
<span>10.5. Virtual Machine Templates</span>
 <a class="headerlink" href="#qm_templates" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>One can convert a VM into a Template. Such templates are read-only,
and you can use them to create linked clones.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">It is not possible to start templates, because this would modify
the disk images. If you want to change the template, create a linked
clone and modify that.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect2">
<h3 id="_vm_generation_id">
<span>10.6. VM Generation ID</span>
 <a class="headerlink" href="#_vm_generation_id" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Proxmox VE supports Virtual Machine Generation ID (<em>vmgenid</em>) <span class="footnote" data-note="Official
<em>vmgenid</em> Specification
<a href=&quot;https://docs.microsoft.com/en-us/windows/desktop/hyperv_v2/virtual-machine-generation-identifier&quot;>https://docs.microsoft.com/en-us/windows/desktop/hyperv_v2/virtual-machine-generation-identifier</a>">[<a id="_footnoteref_46" href="#_footnote_46" title="View footnote" class="footnote">46</a>]</span>
for virtual machines.
This can be used by the guest operating system to detect any event resulting
in a time shift event, for example, restoring a backup or a snapshot rollback.</p></div>
<div class="paragraph">
<p>When creating new VMs, a <em>vmgenid</em> will be automatically generated and saved
in its configuration file.</p></div>
<div class="paragraph">
<p>To create and add a <em>vmgenid</em> to an already existing VM one can pass the
special value ‘1’ to let Proxmox VE autogenerate one or manually set the <em>UUID</em>
<span class="footnote" data-note="Online GUID generator <a href=&quot;http://guid.one/&quot;>http://guid.one/</a>">[<a id="_footnoteref_47" href="#_footnote_47" title="View footnote" class="footnote">47</a>]</span> by using it as value, for
example:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># qm set VMID -vmgenid 1
# qm set VMID -vmgenid 00000000-0000-0000-0000-000000000000</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">The initial addition of a <em>vmgenid</em> device to an existing VM, may result
in the same effects as a change on snapshot rollback, backup restore, etc., has
as the VM can interpret this as generation change.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>In the rare case the <em>vmgenid</em> mechanism is not wanted one can pass ‘0’ for
its value on VM creation, or retroactively delete the property in the
configuration with:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># qm set VMID -delete vmgenid</pre>
</div></div>
<div class="paragraph">
<p>The most prominent use case for <em>vmgenid</em> are newer Microsoft Windows
operating systems, which use it to avoid problems in time sensitive or
replicate services (such as databases or domain controller
<span class="footnote" data-note="<a href=&quot;https://docs.microsoft.com/en-us/windows-server/identity/ad-ds/get-started/virtual-dc/virtualized-domain-controller-architecture&quot;>https://docs.microsoft.com/en-us/windows-server/identity/ad-ds/get-started/virtual-dc/virtualized-domain-controller-architecture</a>">[<a id="_footnoteref_48" href="#_footnote_48" title="View footnote" class="footnote">48</a>]</span>)
on snapshot rollback, backup restore or a whole VM clone operation.</p></div>
</div>
<div class="sect2">
<h3 id="qm_import_virtual_machines">
<span>10.7. Importing Virtual Machines</span>
 <a class="headerlink" href="#qm_import_virtual_machines" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Importing existing virtual machines from foreign hypervisors or other Proxmox VE
clusters can be achieved through various methods, the most common ones are:</p></div>
<div class="ulist"><ul>
<li>
<p>
Using the native import wizard, which utilizes the <em>import</em> content type, such
  as provided by the ESXi special storage.
</p>
</li>
<li>
<p>
Performing a backup on the source and then restoring on the target. This
  method works best when migrating from another Proxmox VE instance.
</p>
</li>
<li>
<p>
using the OVF-specific import command of the <span class="monospaced">qm</span> command-line tool.
</p>
</li>
</ul></div>
<div class="paragraph">
<p>If you import VMs to Proxmox VE from other hypervisors, it’s recommended to
familiarize yourself with the
<a href="https://pve.proxmox.com/wiki/Migrate_to_Proxmox_VE#Concepts">concepts of Proxmox VE</a>.</p></div>
<div class="sect3">
<h4 id="_import_wizard">10.7.1. Import Wizard
 <a class="headerlink" href="#_import_wizard" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-import-wizard-general.png">
<img src="images/screenshot/gui-import-wizard-general.png" alt="screenshot/gui-import-wizard-general.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>Proxmox VE provides an integrated VM importer using the storage plugin system for
native integration into the API and web-based user interface. You can use this
to import the VM as a whole, with most of its config mapped to Proxmox VE’s config
model and reduced downtime.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">The import wizard was added during the Proxmox VE 8.2 development cycle and is
in tech preview state. While it’s already promising and working stable, it’s
still under active development, focusing on adding other import-sources, like
for example OVF/OVA files, in the future.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>To use the import wizard you have to first set up a new storage for an import
source, you can do so on the web-interface under <em>Datacenter → Storage → Add</em>.</p></div>
<div class="paragraph">
<p>Then you can select the new storage in the resource tree and use the <em>Virtual
Guests</em> content tab to see all available guests that can be imported.</p></div>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-import-wizard-advanced.png">
<img src="images/screenshot/gui-import-wizard-advanced.png" alt="screenshot/gui-import-wizard-advanced.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>Select one and use the <em>Import</em> button (or double-click) to open the import
wizard. You can modify a subset of the available options here and then start the
import. Please note that you can do more advanced modifications after the import
finished.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">The import wizard is currently (2024-03) available for ESXi and has been
tested with ESXi versions 6.5 through 8.0. Note that guests using vSAN storage
cannot be directly imported directly; their disks must first be moved to another
storage. While it is possible to use a vCenter as the import source, performance
is dramatically degraded (5 to 10 times slower).</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>For a step-by-step guide and tips for how to adapt the virtual guest to the new
hyper-visor see our
<a href="https://pve.proxmox.com/wiki/Migrate_to_Proxmox_VE#Migration">migrate to Proxmox VE
wiki article</a>.</p></div>
</div>
<div class="sect3">
<h4 id="_import_ovf_ova_through_cli">10.7.2. Import OVF/OVA Through CLI
 <a class="headerlink" href="#_import_ovf_ova_through_cli" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>A VM export from a foreign hypervisor takes usually the form of one or more disk
 images, with a configuration file describing the settings of the VM (RAM,
 number of cores).<br>
The disk images can be in the vmdk format, if the disks come from
VMware or VirtualBox, or qcow2 if the disks come from a KVM hypervisor.
The most popular configuration format for VM exports is the OVF standard, but in
practice interoperation is limited because many settings are not implemented in
the standard itself, and hypervisors export the supplementary information
in non-standard extensions.</p></div>
<div class="paragraph">
<p>Besides the problem of format, importing disk images from other hypervisors
may fail if the emulated hardware changes too much from one hypervisor to
another. Windows VMs are particularly concerned by this, as the OS is very
picky about any changes of hardware. This problem may be solved by
installing the MergeIDE.zip utility available from the Internet before exporting
and choosing a hard disk type of <strong>IDE</strong> before booting the imported Windows VM.</p></div>
<div class="paragraph">
<p>Finally there is the question of paravirtualized drivers, which improve the
speed of the emulated system and are specific to the hypervisor.
GNU/Linux and other free Unix OSes have all the necessary drivers installed by
default and you can switch to the paravirtualized drivers right after importing
the VM. For Windows VMs, you need to install the Windows paravirtualized
drivers by yourself.</p></div>
<div class="paragraph">
<p>GNU/Linux and other free Unix can usually be imported without hassle. Note
that we cannot guarantee a successful import/export of Windows VMs in all
cases due to the problems above.</p></div>
<div class="sect4">
<h5 id="_step_by_step_example_of_a_windows_ovf_import">Step-by-step example of a Windows OVF import
 <a class="headerlink" href="#_step_by_step_example_of_a_windows_ovf_import" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Microsoft provides
<a href="https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/">Virtual Machines downloads</a>
 to get started with Windows development.We are going to use one of these
to demonstrate the OVF import feature.</p></div>
<div class="sect5">
<h6 id="_download_the_virtual_machine_zip">Download the Virtual Machine zip
 <a class="headerlink" href="#_download_the_virtual_machine_zip" title="Permalink to this heading"></a>
</h6>
<div class="paragraph">
<p>After getting informed about the user agreement, choose the <em>Windows 10
Enterprise (Evaluation - Build)</em> for the VMware platform, and download the zip.</p></div>
</div>
<div class="sect5">
<h6 id="_extract_the_disk_image_from_the_zip">Extract the disk image from the zip
 <a class="headerlink" href="#_extract_the_disk_image_from_the_zip" title="Permalink to this heading"></a>
</h6>
<div class="paragraph">
<p>Using the <span class="monospaced">unzip</span> utility or any archiver of your choice, unpack the zip,
and copy via ssh/scp the ovf and vmdk files to your Proxmox VE host.</p></div>
</div>
<div class="sect5">
<h6 id="_import_the_virtual_machine">Import the Virtual Machine
 <a class="headerlink" href="#_import_the_virtual_machine" title="Permalink to this heading"></a>
</h6>
<div class="paragraph">
<p>This will create a new virtual machine, using cores, memory and
VM name as read from the OVF manifest, and import the disks to the <span class="monospaced">local-lvm</span>
 storage. You have to configure the network manually.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># qm importovf 999 WinDev1709Eval.ovf local-lvm</pre>
</div></div>
<div class="paragraph">
<p>The VM is ready to be started.</p></div>
</div>
<div class="sect5">
<h6 id="_adding_an_external_disk_image_to_a_virtual_machine">Adding an external disk image to a Virtual Machine
 <a class="headerlink" href="#_adding_an_external_disk_image_to_a_virtual_machine" title="Permalink to this heading"></a>
</h6>
<div class="paragraph">
<p>You can also add an existing disk image to a VM, either coming from a
foreign hypervisor, or one that you created yourself.</p></div>
<div class="paragraph">
<p>Suppose you created a Debian/Ubuntu disk image with the <em>vmdebootstrap</em> tool:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>vmdebootstrap --verbose \
 --size 10GiB --serial-console \
 --grub --no-extlinux \
 --package openssh-server \
 --package avahi-daemon \
 --package qemu-guest-agent \
 --hostname vm600 --enable-dhcp \
 --customize=./copy_pub_ssh.sh \
 --sparse --image vm600.raw</pre>
</div></div>
<div class="paragraph">
<p>You can now create a new target VM, importing the image to the storage <span class="monospaced">pvedir</span>
and attaching it to the VM’s SCSI controller:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># qm create 600 --net0 virtio,bridge=vmbr0 --name vm600 --serial0 socket \
   --boot order=scsi0 --scsihw virtio-scsi-pci --ostype l26 \
   --scsi0 pvedir:0,import-from=/path/to/dir/vm600.raw</pre>
</div></div>
<div class="paragraph">
<p>The VM is ready to be started.</p></div>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="qm_cloud_init">
<span>10.8. Cloud-Init Support</span>
 <a class="headerlink" href="#qm_cloud_init" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p><a href="https://cloudinit.readthedocs.io">Cloud-Init</a> is the de facto
multi-distribution package that handles early initialization of a
virtual machine instance. Using Cloud-Init, configuration of network
devices and ssh keys on the hypervisor side is possible. When the VM
starts for the first time, the Cloud-Init software inside the VM will
apply those settings.</p></div>
<div class="paragraph">
<p>Many Linux distributions provide ready-to-use Cloud-Init images, mostly
designed for <em>OpenStack</em>. These images will also work with Proxmox VE. While
it may seem convenient to get such ready-to-use images, we usually
recommended to prepare the images by yourself. The advantage is that you
will know exactly what you have installed, and this helps you later to
easily customize the image for your needs.</p></div>
<div class="paragraph">
<p>Once you have created such a Cloud-Init image we recommend to convert it
into a VM template. From a VM template you can quickly create linked
clones, so this is a fast method to roll out new VM instances. You just
need to configure the network (and maybe the ssh keys) before you start
the new VM.</p></div>
<div class="paragraph">
<p>We recommend using SSH key-based authentication to login to the VMs
provisioned by Cloud-Init. It is also possible to set a password, but
this is not as safe as using SSH key-based authentication because Proxmox VE
needs to store an encrypted version of that password inside the
Cloud-Init data.</p></div>
<div class="paragraph">
<p>Proxmox VE generates an ISO image to pass the Cloud-Init data to the VM. For
that purpose, all Cloud-Init VMs need to have an assigned CD-ROM drive.
Usually, a serial console should be added and used as a display. Many Cloud-Init
images rely on this, it is a requirement for OpenStack. However, other images
might have problems with this configuration. Switch back to the default display
configuration if using a serial console doesn’t work.</p></div>
<div class="sect3">
<h4 id="_preparing_cloud_init_templates">10.8.1. Preparing Cloud-Init Templates
 <a class="headerlink" href="#_preparing_cloud_init_templates" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The first step is to prepare your VM. Basically you can use any VM.
Simply install the Cloud-Init packages <strong>inside the VM</strong> that you want to
prepare. On Debian/Ubuntu based systems this is as simple as:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>apt-get install cloud-init</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">This command is <strong>not</strong> intended to be executed on the Proxmox VE host, but
only inside the VM.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>Already many distributions provide ready-to-use Cloud-Init images (provided
as <span class="monospaced">.qcow2</span> files), so alternatively you can simply download and
import such images. For the following example, we will use the cloud
image provided by Ubuntu at <a href="https://cloud-images.ubuntu.com">https://cloud-images.ubuntu.com</a>.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># download the image
wget https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.img

# create a new VM with VirtIO SCSI controller
qm create 9000 --memory 2048 --net0 virtio,bridge=vmbr0 --scsihw virtio-scsi-pci

# import the downloaded disk to the local-lvm storage, attaching it as a SCSI drive
qm set 9000 --scsi0 local-lvm:0,import-from=/path/to/bionic-server-cloudimg-amd64.img</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Ubuntu Cloud-Init images require the <span class="monospaced">virtio-scsi-pci</span>
controller type for SCSI drives.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-cloudinit-hardware.png">
<img src="images/screenshot/gui-cloudinit-hardware.png" alt="screenshot/gui-cloudinit-hardware.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<div class="title">Add Cloud-Init CD-ROM drive</div><p>The next step is to configure a CD-ROM drive, which will be used to pass
the Cloud-Init data to the VM.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>qm set 9000 --ide2 local-lvm:cloudinit</pre>
</div></div>
<div class="paragraph">
<p>To be able to boot directly from the Cloud-Init image, set the <span class="monospaced">boot</span> parameter
to <span class="monospaced">order=scsi0</span> to restrict BIOS to boot from this disk only. This will speed
up booting, because VM BIOS skips the testing for a bootable CD-ROM.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>qm set 9000 --boot order=scsi0</pre>
</div></div>
<div class="paragraph">
<p>For many Cloud-Init images, it is required to configure a serial console and use
it as a display. If the configuration doesn’t work for a given image however,
switch back to the default display instead.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>qm set 9000 --serial0 socket --vga serial0</pre>
</div></div>
<div class="paragraph">
<p>In a last step, it is helpful to convert the VM into a template. From
this template you can then quickly create linked clones.
The deployment from VM templates is much faster than creating a full
clone (copy).</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>qm template 9000</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_deploying_cloud_init_templates">10.8.2. Deploying Cloud-Init Templates
 <a class="headerlink" href="#_deploying_cloud_init_templates" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-cloudinit-config.png">
<img src="images/screenshot/gui-cloudinit-config.png" alt="screenshot/gui-cloudinit-config.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>You can easily deploy such a template by cloning:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>qm clone 9000 123 --name ubuntu2</pre>
</div></div>
<div class="paragraph">
<p>Then configure the SSH public key used for authentication, and configure
the IP setup:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>qm set 123 --sshkey ~/.ssh/id_rsa.pub
qm set 123 --ipconfig0 ip=10.0.10.123/24,gw=10.0.10.1</pre>
</div></div>
<div class="paragraph">
<p>You can also configure all the Cloud-Init options using a single command
only. We have simply split the above example to separate the
commands for reducing the line length. Also make sure to adopt the IP
setup for your specific environment.</p></div>
</div>
<div class="sect3">
<h4 id="_custom_cloud_init_configuration">10.8.3. Custom Cloud-Init Configuration
 <a class="headerlink" href="#_custom_cloud_init_configuration" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The Cloud-Init integration also allows custom config files to be used instead
of the automatically generated configs. This is done via the <span class="monospaced">cicustom</span>
option on the command line:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>qm set 9000 --cicustom "user=&lt;volume&gt;,network=&lt;volume&gt;,meta=&lt;volume&gt;"</pre>
</div></div>
<div class="paragraph">
<p>The custom config files have to be on a storage that supports snippets and have
to be available on all nodes the VM is going to be migrated to. Otherwise the
VM won’t be able to start.
For example:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>qm set 9000 --cicustom "user=local:snippets/userconfig.yaml"</pre>
</div></div>
<div class="paragraph">
<p>There are three kinds of configs for Cloud-Init. The first one is the <span class="monospaced">user</span>
config as seen in the example above. The second is the <span class="monospaced">network</span> config and
the third the <span class="monospaced">meta</span> config. They can all be specified together or mixed
and matched however needed.
The automatically generated config will be used for any that don’t have a
custom config file specified.</p></div>
<div class="paragraph">
<p>The generated config can be dumped to serve as a base for custom configs:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>qm cloudinit dump 9000 user</pre>
</div></div>
<div class="paragraph">
<p>The same command exists for <span class="monospaced">network</span> and <span class="monospaced">meta</span>.</p></div>
</div>
<div class="sect3">
<h4 id="_cloud_init_on_windows">10.8.4. Cloud-Init on Windows
 <a class="headerlink" href="#_cloud_init_on_windows" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>There is a reimplementation of Cloud-Init available for Windows called
<a href="https://cloudbase.it/">cloudbase-init</a>. Not every feature of Cloud-Init is
available with Cloudbase-Init, and some features differ compared to Cloud-Init.</p></div>
<div class="paragraph">
<p>Cloudbase-Init requires both <span class="monospaced">ostype</span> set to any Windows version and the
<span class="monospaced">citype</span> set to <span class="monospaced">configdrive2</span>, which is the default with any Windows
<span class="monospaced">ostype</span>.</p></div>
<div class="paragraph">
<p>There are no ready-made cloud images for Windows available for free. Using
Cloudbase-Init requires manually installing and configuring a Windows guest.</p></div>
</div>
<div class="sect3">
<h4 id="_preparing_cloudbase_init_templates">10.8.5. Preparing Cloudbase-Init Templates
 <a class="headerlink" href="#_preparing_cloudbase_init_templates" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The first step is to install Windows in a VM. Download and install
Cloudbase-Init in the guest. It may be necessary to install the Beta version.
Don’t run Sysprep at the end of the installation. Instead configure
Cloudbase-Init first.</p></div>
<div class="paragraph">
<p>A few common options to set would be:</p></div>
<div class="ulist"><ul>
<li>
<p>
<em>username</em>: This sets the username of the administrator
</p>
</li>
<li>
<p>
<em>groups</em>: This allows one to add the user to the <span class="monospaced">Administrators</span> group
</p>
</li>
<li>
<p>
<em>inject_user_password</em>: Set this to <span class="monospaced">true</span> to allow setting the password
in the VM config
</p>
</li>
<li>
<p>
<em>first_logon_behaviour</em>: Set this to <span class="monospaced">no</span> to not require a new password on
login
</p>
</li>
<li>
<p>
<em>rename_admin_user</em>: Set this to <span class="monospaced">true</span> to allow renaming the default
<span class="monospaced">Administrator</span> user to the username specified with <span class="monospaced">username</span>
</p>
</li>
<li>
<p>
<em>metadata_services</em>: Set this to
<span class="monospaced">cloudbaseinit.metadata.services.configdrive.ConfigDriveService</span> for
Cloudbase-Init to first check this serivce. Otherwise it may take a few minutes
for Cloudbase-Init to configure the system after boot.
</p>
</li>
</ul></div>
<div class="paragraph">
<p>Some plugins, for example the SetHostnamePlugin, require reboots and will do
so automatically. To disable automatic reboots by Cloudbase-Init, you can set
<span class="monospaced">allow_reboot</span> to <span class="monospaced">false</span>.</p></div>
<div class="paragraph">
<p>A full set of configuration options can be found in the
<a href="https://cloudbase-init.readthedocs.io/en/latest/config.html">official
cloudbase-init documentation</a>.</p></div>
<div class="paragraph">
<p>It can make sense to make a snapshot after configuring in case some parts of
the config still need adjustments.
After configuring Cloudbase-Init you can start creating the template. Shutdown
the Windows guest, add a Cloud-Init disk and make it into a template.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>qm set 9000 --ide2 local-lvm:cloudinit
qm template 9000</pre>
</div></div>
<div class="paragraph">
<p>Clone the template into a new VM:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>qm clone 9000 123 --name windows123</pre>
</div></div>
<div class="paragraph">
<p>Then set the password, network config and SSH key:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>qm set 123 --cipassword &lt;password&gt;
qm set 123 --ipconfig0 ip=10.0.10.123/24,gw=10.0.10.1
qm set 123 --sshkey ~/.ssh/id_rsa.pub</pre>
</div></div>
<div class="paragraph">
<p>Make sure that the <span class="monospaced">ostype</span> is set to any Windows version before setting the
password. Otherwise the password will be encrypted and Cloudbase-Init will use
the encrypted password as plaintext password.</p></div>
<div class="paragraph">
<p>When everything is set, start the cloned guest. On the first boot the login
won’t work and it will reboot automatically for the changed hostname.
After the reboot the new password should be set and login should work.</p></div>
</div>
<div class="sect3">
<h4 id="_cloudbase_init_and_sysprep">10.8.6. Cloudbase-Init and Sysprep
 <a class="headerlink" href="#_cloudbase_init_and_sysprep" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Sysprep is a feature to reset the configuration of Windows and provide a <span class="monospaced">new</span>
system. This can be used in conjunction with Cloudbase-Init to create a clean
template.</p></div>
<div class="paragraph">
<p>When using Sysprep there are 2 configuration files that need to be adapted.
The first one is the normal configuration file, the second one is the one
ending in <span class="monospaced">-unattend.conf</span>.</p></div>
<div class="paragraph">
<p>Cloudbase-Init runs in 2 steps, first the Sysprep step using the
<span class="monospaced">-unattend.conf</span> and then the regular step using the primary config file.</p></div>
<div class="paragraph">
<p>For <span class="monospaced">Windows Server</span> running Sysprep with the provided <span class="monospaced">Unattend.xml</span> file
should work out of the box. Normal Windows versions however require additional
steps:</p></div>
<div class="olist arabic"><ol class="arabic">
<li>
<p>
Open a PowerShell instance
</p>
</li>
<li>
<p>
Enable the Administrator user:
</p>
<div class="listingblock">
<div class="content monospaced">
<pre>net user Administrator /active:yes`</pre>
</div></div>
</li>
<li>
<p>
Install Cloudbase-Init using the Administrator user
</p>
</li>
<li>
<p>
Modify <span class="monospaced">Unattend.xml</span> to include the command to enable the Administrator user
on the first boot after sysprepping:
</p>
<div class="listingblock">
<div class="content monospaced">
<pre>&lt;RunSynchronousCommand wcm:action="add"&gt;
  &lt;Path&gt;net user administrator /active:yes&lt;/Path&gt;
  &lt;Order&gt;1&lt;/Order&gt;
  &lt;Description&gt;Enable Administrator User&lt;/Description&gt;
&lt;/RunSynchronousCommand&gt;</pre>
</div></div>
<div class="paragraph">
<p>Make sure the <span class="monospaced">&lt;Order&gt;</span> does not conflict with other synchronous commands.
Modify <span class="monospaced">&lt;Order&gt;</span> of the Cloudbase-Init command to run after this one by
increasing the number to a higher value: <span class="monospaced">&lt;Order&gt;2&lt;/Order&gt;</span></p></div>
</li>
<li>
<p>
(Windows 11 only) Remove the conflicting Microsoft.OneDriveSync package:
</p>
<div class="listingblock">
<div class="content monospaced">
<pre>Get-AppxPackage -AllUsers Microsoft.OneDriveSync | Remove-AppxPackage -AllUsers</pre>
</div></div>
</li>
<li>
<p>
<span class="monospaced">cd</span> into the Cloudbase-Init config directory:
</p>
<div class="listingblock">
<div class="content monospaced">
<pre>cd 'C:\Program Files\Cloudbase Solutions\Cloudbase-Init\conf'</pre>
</div></div>
</li>
<li>
<p>
(optional) Create a snapshot of the VM before Sysprep in case of a
misconfiguration
</p>
</li>
<li>
<p>
Run Sysprep:
</p>
<div class="listingblock">
<div class="content monospaced">
<pre>C:\Windows\System32\Sysprep\sysprep.exe /generalize /oobe /unattend:Unattend.xml</pre>
</div></div>
</li>
</ol></div>
<div class="paragraph">
<p>After following the above steps the VM should be in shut down state due to
the Sysprep. Now you can make it into a template, clone it and configure
it as needed.</p></div>
</div>
<div class="sect3">
<h4 id="_cloud_init_specific_options">10.8.7. Cloud-Init specific Options
 <a class="headerlink" href="#_cloud_init_specific_options" title="Permalink to this heading"></a>
</h4>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">cicustom</span>: <span class="monospaced">[meta=&lt;volume&gt;] [,network=&lt;volume&gt;] [,user=&lt;volume&gt;] [,vendor=&lt;volume&gt;]</span> 
</dt>
<dd>
<p>
Specify custom files to replace the automatically generated ones at start.
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">meta</span>=<span class="monospaced">&lt;volume&gt;</span> 
</dt>
<dd>
<p>
Specify a custom file containing all meta data passed to the VM via"
            ." cloud-init. This is provider specific meaning configdrive2 and nocloud differ.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">network</span>=<span class="monospaced">&lt;volume&gt;</span> 
</dt>
<dd>
<p>
To pass a custom file containing all network data to the VM via cloud-init.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">user</span>=<span class="monospaced">&lt;volume&gt;</span> 
</dt>
<dd>
<p>
To pass a custom file containing all user data to the VM via cloud-init.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">vendor</span>=<span class="monospaced">&lt;volume&gt;</span> 
</dt>
<dd>
<p>
To pass a custom file containing all vendor data to the VM via cloud-init.
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">cipassword</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Password to assign the user. Using this is generally not recommended. Use ssh keys instead. Also note that older cloud-init versions do not support hashed passwords.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">citype</span>: <span class="monospaced">&lt;configdrive2 | nocloud | opennebula&gt;</span> 
</dt>
<dd>
<p>
Specifies the cloud-init configuration format. The default depends on the configured operating system type (<span class="monospaced">ostype</span>. We use the <span class="monospaced">nocloud</span> format for Linux, and <span class="monospaced">configdrive2</span> for windows.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">ciupgrade</span>: <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
do an automatic package upgrade after the first boot.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">ciuser</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
User name to change ssh keys and password for instead of the image’s configured default user.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">ipconfig[n]</span>: <span class="monospaced">[gw=&lt;GatewayIPv4&gt;] [,gw6=&lt;GatewayIPv6&gt;] [,ip=&lt;IPv4Format/CIDR&gt;] [,ip6=&lt;IPv6Format/CIDR&gt;]</span> 
</dt>
<dd>
<p>
Specify IP addresses and gateways for the corresponding interface.
</p>
<div class="paragraph">
<p>IP addresses use CIDR notation, gateways are optional but need an IP of the same type specified.</p></div>
<div class="paragraph">
<p>The special string <em>dhcp</em> can be used for IP addresses to use DHCP, in which case no explicit
gateway should be provided.
For IPv6 the special string <em>auto</em> can be used to use stateless autoconfiguration. This requires
cloud-init 19.4 or newer.</p></div>
<div class="paragraph">
<p>If cloud-init is enabled and neither an IPv4 nor an IPv6 address is specified, it defaults to using
dhcp on IPv4.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">gw</span>=<span class="monospaced">&lt;GatewayIPv4&gt;</span> 
</dt>
<dd>
<p>
Default gateway for IPv4 traffic.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Requires option(s): <span class="monospaced">ip</span></td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">gw6</span>=<span class="monospaced">&lt;GatewayIPv6&gt;</span> 
</dt>
<dd>
<p>
Default gateway for IPv6 traffic.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Requires option(s): <span class="monospaced">ip6</span></td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">ip</span>=<span class="monospaced">&lt;IPv4Format/CIDR&gt;</span> (<em>default =</em> <span class="monospaced">dhcp</span>)
</dt>
<dd>
<p>
IPv4 address in CIDR format.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">ip6</span>=<span class="monospaced">&lt;IPv6Format/CIDR&gt;</span> (<em>default =</em> <span class="monospaced">dhcp</span>)
</dt>
<dd>
<p>
IPv6 address in CIDR format.
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">nameserver</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Sets DNS server IP address for a container. Create will automatically use the setting from the host if neither searchdomain nor nameserver are set.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">searchdomain</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Sets DNS search domains for a container. Create will automatically use the setting from the host if neither searchdomain nor nameserver are set.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">sshkeys</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Setup public SSH keys (one key per line, OpenSSH format).
</p>
</dd>
</dl></div>
</div>
</div>
<div class="sect2">
<h3 id="qm_pci_passthrough">
<span>10.9. PCI(e) Passthrough</span>
 <a class="headerlink" href="#qm_pci_passthrough" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>PCI(e) passthrough is a mechanism to give a virtual machine control over
a PCI device from the host. This can have some advantages over using
virtualized hardware, for example lower latency, higher performance, or more
features (e.g., offloading).</p></div>
<div class="paragraph">
<p>But, if you pass through a device to a virtual machine, you cannot use that
device anymore on the host or in any other VM.</p></div>
<div class="paragraph">
<p>Note that, while PCI passthrough is available for i440fx and q35 machines, PCIe
passthrough is only available on q35 machines. This does not mean that
PCIe capable devices that are passed through as PCI devices will only run at
PCI speeds. Passing through devices as PCIe just sets a flag for the guest to
tell it that the device is a  PCIe device instead of a "really fast legacy PCI
device". Some guest applications benefit from this.</p></div>
<div class="sect3">
<h4 id="_general_requirements">10.9.1. General Requirements
 <a class="headerlink" href="#_general_requirements" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Since passthrough is performed on real hardware, it needs to fulfill some
requirements. A brief overview of these requirements is given below, for more
information on specific devices, see
<a href="https://pve.proxmox.com/wiki/PCI_Passthrough">PCI Passthrough Examples</a>.</p></div>
<div class="sect4">
<h5 id="_hardware_3">Hardware
 <a class="headerlink" href="#_hardware_3" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Your hardware needs to support <span class="monospaced">IOMMU</span> (<strong>I</strong>/<strong>O</strong> <strong>M</strong>emory <strong>M</strong>anagement
<strong>U</strong>nit) interrupt remapping, this includes the CPU and the motherboard.</p></div>
<div class="paragraph">
<p>Generally, Intel systems with VT-d and AMD systems with AMD-Vi support this.
But it is not guaranteed that everything will work out of the box, due
to bad hardware implementation and missing or low quality drivers.</p></div>
<div class="paragraph">
<p>Further, server grade hardware has often better support than consumer grade
hardware, but even then, many modern system can support this.</p></div>
<div class="paragraph">
<p>Please refer to your hardware vendor to check if they support this feature
under Linux for your specific setup.</p></div>
</div>
<div class="sect4">
<h5 id="_determining_pci_card_address">Determining PCI Card Address
 <a class="headerlink" href="#_determining_pci_card_address" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>The easiest way is to use the GUI to add a device of type "Host PCI" in the VM’s
hardware tab. Alternatively, you can use the command line.</p></div>
<div class="paragraph">
<p>You can locate your card using</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre> lspci</pre>
</div></div>
</div>
<div class="sect4">
<h5 id="_configuration_14">Configuration
 <a class="headerlink" href="#_configuration_14" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Once you ensured that your hardware supports passthrough, you will need to do
some configuration to enable PCI(e) passthrough.</p></div>
<div class="paragraph">
<div class="title">IOMMU</div><p>First, you will have to enable IOMMU support in your BIOS/UEFI. Usually the
corresponding setting is called <span class="monospaced">IOMMU</span> or <span class="monospaced">VT-d</span>, but you should find the exact
option name in the manual of your motherboard.</p></div>
<div class="paragraph">
<p>For Intel CPUs, you also need to enable the IOMMU on the
<a href="#sysboot_edit_kernel_cmdline">kernel command line</a> kernels by adding:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre> intel_iommu=on</pre>
</div></div>
<div class="paragraph">
<p>For AMD CPUs it should be enabled automatically.</p></div>
<div class="paragraph">
<div class="title">IOMMU Passthrough Mode</div><p>If your hardware supports IOMMU passthrough mode, enabling this mode might
increase performance.
This is because VMs then bypass the (default) DMA translation normally
performed by the hyper-visor and instead pass DMA requests directly to the
hardware IOMMU. To enable these options, add:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre> iommu=pt</pre>
</div></div>
<div class="paragraph">
<p>to the <a href="#sysboot_edit_kernel_cmdline">kernel commandline</a>.</p></div>
<div class="paragraph">
<div class="title">Kernel Modules</div><p>You have to make sure the following modules are loaded. This can be achieved by
adding them to ‘<em>/etc/modules</em>’. In kernels newer than 6.2 (Proxmox VE 8 and onward)
the <em>vfio_virqfd</em> module is part of the <em>vfio</em> module, therefore loading
<em>vfio_virqfd</em> in Proxmox VE 8 and newer is not necessary.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre> vfio
 vfio_iommu_type1
 vfio_pci
 vfio_virqfd #not needed if on kernel 6.2 or newer</pre>
</div></div>
<div class="paragraph" id="qm_pci_passthrough_update_initramfs">
<p>After changing anything modules related, you need to refresh your
<span class="monospaced">initramfs</span>. On Proxmox VE this can be done by executing:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># update-initramfs -u -k all</pre>
</div></div>
<div class="paragraph">
<p>To check if the modules are being loaded, the output of</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># lsmod | grep vfio</pre>
</div></div>
<div class="paragraph">
<p>should include the four modules from above.</p></div>
<div class="paragraph">
<div class="title">Finish Configuration</div><p>Finally reboot to bring the changes into effect and check that it is indeed
enabled.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># dmesg | grep -e DMAR -e IOMMU -e AMD-Vi</pre>
</div></div>
<div class="paragraph">
<p>should display that <span class="monospaced">IOMMU</span>, <span class="monospaced">Directed I/O</span> or <span class="monospaced">Interrupt Remapping</span> is
enabled, depending on hardware and kernel the exact message can vary.</p></div>
<div class="paragraph">
<p>For notes on how to troubleshoot or verify if IOMMU is working as intended, please
see the <a href="https://pve.proxmox.com/wiki/PCI_Passthrough#Verifying_IOMMU_parameters">Verifying IOMMU Parameters</a>
section in our wiki.</p></div>
<div class="paragraph">
<p>It is also important that the device(s) you want to pass through
are in a <strong>separate</strong> <span class="monospaced">IOMMU</span> group. This can be checked with a call to the Proxmox VE
API:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># pvesh get /nodes/{nodename}/hardware/pci --pci-class-blacklist ""</pre>
</div></div>
<div class="paragraph">
<p>It is okay if the device is in an <span class="monospaced">IOMMU</span> group together with its functions
(e.g. a GPU with the HDMI Audio device) or with its root port or PCI(e) bridge.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">
<div class="title">PCI(e) slots</div>
<div class="paragraph">
<p>Some platforms handle their physical PCI(e) slots differently. So, sometimes
it can help to put the card in a another PCI(e) slot, if you do not get the
desired <span class="monospaced">IOMMU</span> group separation.</p></div>
</td>
</tr></tbody></table>
</div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">
<div class="title">Unsafe interrupts</div>
<div class="paragraph">
<p>For some platforms, it may be necessary to allow unsafe interrupts.
For this add  the following line in a file ending with ‘.conf’ file in
<strong>/etc/modprobe.d/</strong>:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre> options vfio_iommu_type1 allow_unsafe_interrupts=1</pre>
</div></div>
<div class="paragraph">
<p>Please be aware that this option can make your system unstable.</p></div>
</td>
</tr></tbody></table>
</div>
</div>
<div class="sect4">
<h5 id="_gpu_passthrough_notes">GPU Passthrough Notes
 <a class="headerlink" href="#_gpu_passthrough_notes" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>It is not possible to display the frame buffer of the GPU via NoVNC or SPICE on
the Proxmox VE web interface.</p></div>
<div class="paragraph">
<p>When passing through a whole GPU or a vGPU and graphic output is wanted, one
has to either physically connect a monitor to the card, or configure a remote
desktop software (for example, VNC or RDP) inside the guest.</p></div>
<div class="paragraph">
<p>If you want to use the GPU as a hardware accelerator, for example, for
programs using OpenCL or CUDA, this is not required.</p></div>
</div>
</div>
<div class="sect3">
<h4 id="_host_device_passthrough">10.9.2. Host Device Passthrough
 <a class="headerlink" href="#_host_device_passthrough" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The most used variant of PCI(e) passthrough is to pass through a whole
PCI(e) card, for example a GPU or a network card.</p></div>
<div class="sect4">
<h5 id="_host_configuration">Host Configuration
 <a class="headerlink" href="#_host_configuration" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Proxmox VE tries to automatically make the PCI(e) device unavailable for the host.
However, if this doesn’t work, there are two things that can be done:</p></div>
<div class="ulist"><ul>
<li>
<p>
pass the device IDs to the options of the <em>vfio-pci</em> modules by adding
</p>
<div class="listingblock">
<div class="content monospaced">
<pre> options vfio-pci ids=1234:5678,4321:8765</pre>
</div></div>
<div class="paragraph">
<p>to a .conf file in <strong>/etc/modprobe.d/</strong> where <span class="monospaced">1234:5678</span> and <span class="monospaced">4321:8765</span> are
the vendor and device IDs obtained by:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># lspci -nn</pre>
</div></div>
</li>
<li>
<p>
blacklist the driver on the host completely, ensuring that it is free to bind
for passthrough, with
</p>
<div class="listingblock">
<div class="content monospaced">
<pre> blacklist DRIVERNAME</pre>
</div></div>
<div class="paragraph">
<p>in a .conf file in <strong>/etc/modprobe.d/</strong>.</p></div>
<div class="paragraph">
<p>To find the drivername, execute</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># lspci -k</pre>
</div></div>
<div class="paragraph">
<p>for example:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># lspci -k | grep -A 3 "VGA"</pre>
</div></div>
<div class="paragraph">
<p>will output something similar to</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>01:00.0 VGA compatible controller: NVIDIA Corporation GP108 [GeForce GT 1030] (rev a1)
        Subsystem: Micro-Star International Co., Ltd. [MSI] GP108 [GeForce GT 1030]
        Kernel driver in use: &lt;some-module&gt;
        Kernel modules: &lt;some-module&gt;</pre>
</div></div>
<div class="paragraph">
<p>Now we can blacklist the drivers by writing them into a .conf file:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>echo "blacklist &lt;some-module&gt;" &gt;&gt; /etc/modprobe.d/blacklist.conf</pre>
</div></div>
</li>
</ul></div>
<div class="paragraph">
<p>For both methods you need to
<a href="#qm_pci_passthrough_update_initramfs">update the <span class="monospaced">initramfs</span></a> again and
reboot after that.</p></div>
<div class="paragraph">
<p>Should this not work, you might need to set a soft dependency to load the gpu
modules before loading <em>vfio-pci</em>. This can be done with the <em>softdep</em> flag, see
also the manpages on <em>modprobe.d</em> for more information.</p></div>
<div class="paragraph">
<p>For example, if you are using drivers named &lt;some-module&gt;:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># echo "softdep &lt;some-module&gt; pre: vfio-pci" &gt;&gt; /etc/modprobe.d/&lt;some-module&gt;.conf</pre>
</div></div>
<div class="paragraph">
<div class="title">Verify Configuration</div><p>To check if your changes were successful, you can use</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># lspci -nnk</pre>
</div></div>
<div class="paragraph">
<p>and check your device entry. If it says</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>Kernel driver in use: vfio-pci</pre>
</div></div>
<div class="paragraph">
<p>or the <em>in use</em> line is missing entirely, the device is ready to be used for
passthrough.</p></div>
</div>
<div class="sect4">
<h5 id="qm_pci_passthrough_vm_config">VM Configuration
 <a class="headerlink" href="#qm_pci_passthrough_vm_config" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>When passing through a GPU, the best compatibility is reached when using
<em>q35</em> as machine type, <em>OVMF</em> (<em>UEFI</em> for VMs) instead of SeaBIOS and PCIe
instead of PCI. Note that if you want to use <em>OVMF</em> for GPU passthrough, the
GPU needs to have an UEFI capable ROM, otherwise use SeaBIOS instead. To check if
the ROM is UEFI capable, see the
<a href="https://pve.proxmox.com/wiki/PCI_Passthrough#How_to_know_if_a_graphics_card_is_UEFI_.28OVMF.29_compatible">PCI Passthrough Examples</a>
wiki.</p></div>
<div class="paragraph">
<p>Furthermore, using OVMF, disabling vga arbitration may be possible, reducing the
amount of legacy code needed to be run during boot. To disable vga arbitration:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre> echo "options vfio-pci ids=&lt;vendor-id&gt;,&lt;device-id&gt; disable_vga=1" &gt; /etc/modprobe.d/vfio.conf</pre>
</div></div>
<div class="paragraph">
<p>replacing the &lt;vendor-id&gt; and &lt;device-id&gt; with the ones obtained from:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># lspci -nn</pre>
</div></div>
<div class="paragraph">
<p>PCI devices can be added in the web interface in the hardware section of the VM.
Alternatively, you can use the command line; set the <strong>hostpciX</strong> option in the VM
configuration, for example by executing:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># qm set VMID -hostpci0 00:02.0</pre>
</div></div>
<div class="paragraph">
<p>or by adding a line to the VM configuration file:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre> hostpci0: 00:02.0</pre>
</div></div>
<div class="paragraph">
<p>If your device has multiple functions (e.g., ‘<span class="monospaced">00:02.0</span>’ and ‘<span class="monospaced">00:02.1</span>’ ),
you can pass them through all together with the shortened syntax ``00:02`<em>.
This is equivalent with checking the ``All Functions`</em> checkbox in the
web interface.</p></div>
<div class="paragraph">
<p>There are some options to which may be necessary, depending on the device
and guest OS:</p></div>
<div class="ulist"><ul>
<li>
<p>
<strong>x-vga=on|off</strong> marks the PCI(e) device as the primary GPU of the VM.
With this enabled the <strong>vga</strong> configuration option will be ignored.
</p>
</li>
<li>
<p>
<strong>pcie=on|off</strong> tells Proxmox VE to use a PCIe or PCI port. Some guests/device
combination require PCIe rather than PCI. PCIe is only available for <em>q35</em>
machine types.
</p>
</li>
<li>
<p>
<strong>rombar=on|off</strong> makes the firmware ROM visible for the guest. Default is on.
Some PCI(e) devices need this disabled.
</p>
</li>
<li>
<p>
<strong>romfile=&lt;path&gt;</strong>, is an optional path to a ROM file for the device to use.
This is a relative path under <strong>/usr/share/kvm/</strong>.
</p>
</li>
</ul></div>
<div class="paragraph">
<div class="title">Example</div><p>An example of PCIe passthrough with a GPU set to primary:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># qm set VMID -hostpci0 02:00,pcie=on,x-vga=on</pre>
</div></div>
<div class="paragraph">
<div class="title">PCI ID overrides</div><p>You can override the PCI vendor ID, device ID, and subsystem IDs that will be
seen by the guest. This is useful if your device is a variant with an ID that
your guest’s drivers don’t recognize, but you want to force those drivers to be
loaded anyway (e.g. if you know your device shares the same chipset as a
supported variant).</p></div>
<div class="paragraph">
<p>The available options are <span class="monospaced">vendor-id</span>, <span class="monospaced">device-id</span>, <span class="monospaced">sub-vendor-id</span>, and
<span class="monospaced">sub-device-id</span>. You can set any or all of these to override your device’s
default IDs.</p></div>
<div class="paragraph">
<p>For example:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># qm set VMID -hostpci0 02:00,device-id=0x10f6,sub-vendor-id=0x0000</pre>
</div></div>
</div>
</div>
<div class="sect3">
<h4 id="_sr_iov">10.9.3. SR-IOV
 <a class="headerlink" href="#_sr_iov" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Another variant for passing through PCI(e) devices is to use the hardware
virtualization features of your devices, if available.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">
<div class="title">Enabling SR-IOV</div>
<div class="paragraph">
<p>To use SR-IOV, platform support is especially important. It may be necessary
to enable this feature in the BIOS/UEFI first, or to use a specific PCI(e) port
for it to work. In doubt, consult the manual of the platform or contact its
vendor.</p></div>
</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p><em>SR-IOV</em> (<strong>S</strong>ingle-<strong>R</strong>oot <strong>I</strong>nput/<strong>O</strong>utput <strong>V</strong>irtualization) enables
a single device to provide multiple <em>VF</em> (<strong>V</strong>irtual <strong>F</strong>unctions) to the
system. Each of those <em>VF</em> can be used in a different VM, with full hardware
features and also better performance and lower latency than software
virtualized devices.</p></div>
<div class="paragraph">
<p>Currently, the most common use case for this are NICs (<strong>N</strong>etwork
<strong>I</strong>nterface <strong>C</strong>ard) with SR-IOV support, which can provide multiple VFs per
physical port. This allows using features such as checksum offloading, etc. to
be used inside a VM, reducing the (host) CPU overhead.</p></div>
<div class="sect4">
<h5 id="_host_configuration_2">Host Configuration
 <a class="headerlink" href="#_host_configuration_2" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Generally, there are two methods for enabling virtual functions on a device.</p></div>
<div class="ulist"><ul>
<li>
<p>
sometimes there is an option for the driver module e.g. for some
Intel drivers
</p>
<div class="listingblock">
<div class="content monospaced">
<pre> max_vfs=4</pre>
</div></div>
<div class="paragraph">
<p>which could be put file with <em>.conf</em> ending under <strong>/etc/modprobe.d/</strong>.
(Do not forget to update your initramfs after that)</p></div>
<div class="paragraph">
<p>Please refer to your driver module documentation for the exact
parameters and options.</p></div>
</li>
<li>
<p>
The second, more generic, approach is using the <span class="monospaced">sysfs</span>.
If a device and driver supports this you can change the number of VFs on
the fly. For example, to setup 4 VFs on device 0000:01:00.0 execute:
</p>
<div class="listingblock">
<div class="content monospaced">
<pre># echo 4 &gt; /sys/bus/pci/devices/0000:01:00.0/sriov_numvfs</pre>
</div></div>
<div class="paragraph">
<p>To make this change persistent you can use the ‘sysfsutils` Debian package.
After installation configure it via <strong>/etc/sysfs.conf</strong> or a `FILE.conf’ in
<strong>/etc/sysfs.d/</strong>.</p></div>
</li>
</ul></div>
</div>
<div class="sect4">
<h5 id="_vm_configuration">VM Configuration
 <a class="headerlink" href="#_vm_configuration" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>After creating VFs, you should see them as separate PCI(e) devices when
outputting them with <span class="monospaced">lspci</span>. Get their ID and pass them through like a
<a href="#qm_pci_passthrough_vm_config">normal PCI(e) device</a>.</p></div>
</div>
</div>
<div class="sect3">
<h4 id="_mediated_devices_vgpu_gvt_g">10.9.4. Mediated Devices (vGPU, GVT-g)
 <a class="headerlink" href="#_mediated_devices_vgpu_gvt_g" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Mediated devices are another method to reuse features and performance from
physical hardware for virtualized hardware. These are found most common in
virtualized GPU setups such as Intel’s GVT-g and NVIDIA’s vGPUs used in their
GRID technology.</p></div>
<div class="paragraph">
<p>With this, a physical Card is able to create virtual cards, similar to SR-IOV.
The difference is that mediated devices do not appear as PCI(e) devices in the
host, and are such only suited for using in virtual machines.</p></div>
<div class="sect4">
<h5 id="_host_configuration_3">Host Configuration
 <a class="headerlink" href="#_host_configuration_3" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>In general your card’s driver must support that feature, otherwise it will
not work. So please refer to your vendor for compatible drivers and how to
configure them.</p></div>
<div class="paragraph">
<p>Intel’s drivers for GVT-g are integrated in the Kernel and should work
with 5th, 6th and 7th generation Intel Core Processors, as well as E3 v4, E3
v5 and E3 v6 Xeon Processors.</p></div>
<div class="paragraph">
<p>To enable it for Intel Graphics, you have to make sure to load the module
<em>kvmgt</em> (for example via <span class="monospaced">/etc/modules</span>) and to enable it on the
<a href="#sysboot_edit_kernel_cmdline">Kernel commandline</a> and add the following parameter:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre> i915.enable_gvt=1</pre>
</div></div>
<div class="paragraph">
<p>After that remember to
<a href="#qm_pci_passthrough_update_initramfs">update the <span class="monospaced">initramfs</span></a>,
and reboot your host.</p></div>
</div>
<div class="sect4">
<h5 id="_vm_configuration_2">VM Configuration
 <a class="headerlink" href="#_vm_configuration_2" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>To use a mediated device, simply specify the <span class="monospaced">mdev</span> property on a <span class="monospaced">hostpciX</span>
VM configuration option.</p></div>
<div class="paragraph">
<p>You can get the supported devices via the <em>sysfs</em>. For example, to list the
supported types for the device <em>0000:00:02.0</em> you would simply execute:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># ls /sys/bus/pci/devices/0000:00:02.0/mdev_supported_types</pre>
</div></div>
<div class="paragraph">
<p>Each entry is a directory which contains the following important files:</p></div>
<div class="ulist"><ul>
<li>
<p>
<em>available_instances</em> contains the amount of still available instances of
this type, each <em>mdev</em> use in a VM reduces this.
</p>
</li>
<li>
<p>
<em>description</em> contains a short description about the capabilities of the type
</p>
</li>
<li>
<p>
<em>create</em> is the endpoint to create such a device, Proxmox VE does this
automatically for you, if a <em>hostpciX</em> option with <span class="monospaced">mdev</span> is configured.
</p>
</li>
</ul></div>
<div class="paragraph">
<p>Example configuration with an <span class="monospaced">Intel GVT-g vGPU</span> (<span class="monospaced">Intel Skylake 6700k</span>):</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># qm set VMID -hostpci0 00:02.0,mdev=i915-GVTg_V5_4</pre>
</div></div>
<div class="paragraph">
<p>With this set, Proxmox VE automatically creates such a device on VM start, and
cleans it up again when the VM stops.</p></div>
</div>
</div>
<div class="sect3">
<h4 id="_use_in_clusters">10.9.5. Use in Clusters
 <a class="headerlink" href="#_use_in_clusters" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>It is also possible to map devices on a cluster level, so that they can be
properly used with HA and hardware changes are detected and non root users
can configure them. See <a href="#resource_mapping">Resource Mapping</a>
for details on that.</p></div>
</div>
<div class="sect3">
<h4 id="qm_pci_viommu">10.9.6. vIOMMU (emulated IOMMU)
 <a class="headerlink" href="#qm_pci_viommu" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>vIOMMU is the emulation of a hardware IOMMU within a virtual machine, providing
improved memory access control and security for virtualized I/O devices. Using
the vIOMMU option also allows you to pass through PCI(e) devices to level-2 VMs
in level-1 VMs via
<a href="https://pve.proxmox.com/wiki/Nested_Virtualization">Nested Virtualization</a>.
To pass through physical PCI(e) devices from the host to nested VMs, follow the
PCI(e) passthrough instructions.</p></div>
<div class="paragraph">
<p>There are currently two vIOMMU implementations available: Intel and VirtIO.</p></div>
<div class="sect4">
<h5 id="_intel_viommu">Intel vIOMMU
 <a class="headerlink" href="#_intel_viommu" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Intel vIOMMU specific VM requirements:</p></div>
<div class="ulist"><ul>
<li>
<p>
Whether you are using an Intel or AMD CPU on your host, it is important to set
<span class="monospaced">intel_iommu=on</span> in the VMs kernel parameters.
</p>
</li>
<li>
<p>
To use Intel vIOMMU you need to set <strong>q35</strong> as the machine type.
</p>
</li>
</ul></div>
<div class="paragraph">
<p>If all requirements are met, you can add <span class="monospaced">viommu=intel</span> to the machine parameter
in the configuration of the VM that should be able to pass through PCI devices.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># qm set VMID -machine q35,viommu=intel</pre>
</div></div>
<div class="paragraph">
<p><a href="https://wiki.qemu.org/Features/VT-d">QEMU documentation for VT-d</a></p></div>
</div>
<div class="sect4">
<h5 id="_virtio_viommu">VirtIO vIOMMU
 <a class="headerlink" href="#_virtio_viommu" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>This vIOMMU implementation is more recent and does not have as many limitations
as Intel vIOMMU but is currently less used in production and less documentated.</p></div>
<div class="paragraph">
<p>With VirtIO vIOMMU there is <strong>no</strong> need to set any kernel parameters. It is also
<strong>not</strong> necessary to use q35 as the machine type, but it is advisable if you want
to use PCIe.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># qm set VMID -machine q35,viommu=virtio</pre>
</div></div>
<div class="paragraph">
<p><a href="https://web.archive.org/web/20230804075844/https://michael2012z.medium.com/virtio-iommu-789369049443">Blog-Post by Michael Zhao explaining virtio-iommu</a></p></div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_hookscripts">
<span>10.10. Hookscripts</span>
 <a class="headerlink" href="#_hookscripts" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>You can add a hook script to VMs with the config property <span class="monospaced">hookscript</span>.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># qm set 100 --hookscript local:snippets/hookscript.pl</pre>
</div></div>
<div class="paragraph">
<p>It will be called during various phases of the guests lifetime.
For an example and documentation see the example script under
<span class="monospaced">/usr/share/pve-docs/examples/guest-example-hookscript.pl</span>.</p></div>
</div>
<div class="sect2">
<h3 id="qm_hibernate">
<span>10.11. Hibernation</span>
 <a class="headerlink" href="#qm_hibernate" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>You can suspend a VM to disk with the GUI option <span class="monospaced">Hibernate</span> or with</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># qm suspend ID --todisk</pre>
</div></div>
<div class="paragraph">
<p>That means that the current content of the memory will be saved onto disk
and the VM gets stopped. On the next start, the memory content will be
loaded and the VM can continue where it was left off.</p></div>
<div class="paragraph" id="qm_vmstatestorage">
<div class="title">State storage selection</div><p>If no target storage for the memory is given, it will be automatically
chosen, the first of:</p></div>
<div class="olist arabic"><ol class="arabic">
<li>
<p>
The storage <span class="monospaced">vmstatestorage</span> from the VM config.
</p>
</li>
<li>
<p>
The first shared storage from any VM disk.
</p>
</li>
<li>
<p>
The first non-shared storage from any VM disk.
</p>
</li>
<li>
<p>
The storage <span class="monospaced">local</span> as a fallback.
</p>
</li>
</ol></div>
</div>
<div class="sect2">
<h3 id="resource_mapping">
<span>10.12. Resource Mapping</span>
 <a class="headerlink" href="#resource_mapping" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-datacenter-resource-mappings.png">
<img src="images/screenshot/gui-datacenter-resource-mappings.png" alt="screenshot/gui-datacenter-resource-mappings.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>When using or referencing local resources (e.g. address of a pci device), using
the raw address or id is sometimes problematic, for example:</p></div>
<div class="ulist"><ul>
<li>
<p>
when using HA, a different device with the same id or path may exist on the
  target node, and if one is not careful when assigning such guests to HA
  groups, the wrong device could be used, breaking configurations.
</p>
</li>
<li>
<p>
changing hardware can change ids and paths, so one would have to check all
  assigned devices and see if the path or id is still correct.
</p>
</li>
</ul></div>
<div class="paragraph">
<p>To handle this better, one can define cluster wide resource mappings, such that
a resource has a cluster unique, user selected identifier which can correspond
to different devices on different hosts. With this, HA won’t start a guest with
a wrong device, and hardware changes can be detected.</p></div>
<div class="paragraph">
<p>Creating such a mapping can be done with the Proxmox VE web GUI under <span class="monospaced">Datacenter</span>
in the relevant tab in the <span class="monospaced">Resource Mappings</span> category, or on the cli with</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># pvesh create /cluster/mapping/&lt;type&gt; &lt;options&gt;</pre>
</div></div>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-datacenter-mapping-pci-edit.png">
<img src="images/screenshot/gui-datacenter-mapping-pci-edit.png" alt="screenshot/gui-datacenter-mapping-pci-edit.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>Where <span class="monospaced">&lt;type&gt;</span> is the hardware type (currently either <span class="monospaced">pci</span> or <span class="monospaced">usb</span>) and
<span class="monospaced">&lt;options&gt;</span> are the device mappings and other configuration parameters.</p></div>
<div class="paragraph">
<p>Note that the options must include a map property with all identifying
properties of that hardware, so that it’s possible to verify the hardware did
not change and the correct device is passed through.</p></div>
<div class="paragraph">
<p>For example to add a PCI device as <span class="monospaced">device1</span> with the path <span class="monospaced">0000:01:00.0</span> that
has the device id <span class="monospaced">0001</span> and the vendor id <span class="monospaced">0002</span> on the node <span class="monospaced">node1</span>, and
<span class="monospaced">0000:02:00.0</span> on <span class="monospaced">node2</span> you can add it with:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># pvesh create /cluster/mapping/pci --id device1 \
 --map node=node1,path=0000:01:00.0,id=0002:0001 \
 --map node=node2,path=0000:02:00.0,id=0002:0001</pre>
</div></div>
<div class="paragraph">
<p>You must repeat the <span class="monospaced">map</span> parameter for each node where that device should have
a mapping (note that you can currently only map one USB device per node per
mapping).</p></div>
<div class="paragraph">
<p>Using the GUI makes this much easier, as the correct properties are
automatically picked up and sent to the API.</p></div>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-datacenter-mapping-usb-edit.png">
<img src="images/screenshot/gui-datacenter-mapping-usb-edit.png" alt="screenshot/gui-datacenter-mapping-usb-edit.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>It’s also possible for PCI devices to provide multiple devices per node with
multiple map properties for the nodes. If such a device is assigned to a guest,
the first free one will be used when the guest is started. The order of the
paths given is also the order in which they are tried, so arbitrary allocation
policies can be implemented.</p></div>
<div class="paragraph">
<p>This is useful for devices with SR-IOV, since some times it is not important
which exact virtual function is passed through.</p></div>
<div class="paragraph">
<p>You can assign such a device to a guest either with the GUI or with</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># qm set ID -hostpci0 &lt;name&gt;</pre>
</div></div>
<div class="paragraph">
<p>for PCI devices, or</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># qm set &lt;vmid&gt; -usb0 &lt;name&gt;</pre>
</div></div>
<div class="paragraph">
<p>for USB devices.</p></div>
<div class="paragraph">
<p>Where <span class="monospaced">&lt;vmid&gt;</span> is the guests id and <span class="monospaced">&lt;name&gt;</span> is the chosen name for the created
mapping. All usual options for passing through the devices are allowed, such as
<span class="monospaced">mdev</span>.</p></div>
<div class="paragraph">
<p>To create mappings <span class="monospaced">Mapping.Modify</span> on <span class="monospaced">/mapping/&lt;type&gt;/&lt;name&gt;</span> is necessary
(where <span class="monospaced">&lt;type&gt;</span> is the device type and <span class="monospaced">&lt;name&gt;</span> is the name of the mapping).</p></div>
<div class="paragraph">
<p>To use these mappings, <span class="monospaced">Mapping.Use</span> on <span class="monospaced">/mapping/&lt;type&gt;/&lt;name&gt;</span> is necessary
(in addition to the normal guest privileges to edit the configuration).</p></div>
</div>
<div class="sect2">
<h3 id="_managing_virtual_machines_with_span_class_monospaced_qm_span">
<span>10.13. Managing Virtual Machines with <span class="monospaced">qm</span></span>
 <a class="headerlink" href="#_managing_virtual_machines_with_span_class_monospaced_qm_span" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>qm is the tool to manage QEMU/KVM virtual machines on Proxmox VE. You can
create and destroy virtual machines, and control execution
(start/stop/suspend/resume). Besides that, you can use qm to set
parameters in the associated config file. It is also possible to
create and delete virtual disks.</p></div>
<div class="sect3">
<h4 id="_cli_usage_examples">10.13.1. CLI Usage Examples
 <a class="headerlink" href="#_cli_usage_examples" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Using an iso file uploaded on the <em>local</em> storage, create a VM
with a 4 GB IDE disk on the <em>local-lvm</em> storage</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># qm create 300 -ide0 local-lvm:4 -net0 e1000 -cdrom local:iso/proxmox-mailgateway_2.1.iso</pre>
</div></div>
<div class="paragraph">
<p>Start the new VM</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># qm start 300</pre>
</div></div>
<div class="paragraph">
<p>Send a shutdown request, then wait until the VM is stopped.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># qm shutdown 300 &amp;&amp; qm wait 300</pre>
</div></div>
<div class="paragraph">
<p>Same as above, but only wait for 40 seconds.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># qm shutdown 300 &amp;&amp; qm wait 300 -timeout 40</pre>
</div></div>
<div class="paragraph">
<p>If the VM does not shut down, force-stop it and overrule any running shutdown
tasks. As stopping VMs may incur data loss, use it with caution.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># qm stop 300 -overrule-shutdown 1</pre>
</div></div>
<div class="paragraph">
<p>Destroying a VM always removes it from Access Control Lists and it always
removes the firewall configuration of the VM. You have to activate
<em>--purge</em>, if you want to additionally remove the VM from replication jobs,
backup jobs and HA resource configurations.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># qm destroy 300 --purge</pre>
</div></div>
<div class="paragraph">
<p>Move a disk image to a different storage.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># qm move-disk 300 scsi0 other-storage</pre>
</div></div>
<div class="paragraph">
<p>Reassign a disk image to a different VM. This will remove the disk <span class="monospaced">scsi1</span> from
the source VM and attaches it as <span class="monospaced">scsi3</span> to the target VM. In the background
the disk image is being renamed so that the name matches the new owner.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># qm move-disk 300 scsi1 --target-vmid 400 --target-disk scsi3</pre>
</div></div>
</div>
</div>
<div class="sect2">
<h3 id="qm_configuration">
<span>10.14. Configuration</span>
 <a class="headerlink" href="#qm_configuration" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>VM configuration files are stored inside the Proxmox cluster file
system, and can be accessed at <span class="monospaced">/etc/pve/qemu-server/&lt;VMID&gt;.conf</span>.
Like other files stored inside <span class="monospaced">/etc/pve/</span>, they get automatically
replicated to all other cluster nodes.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">VMIDs &lt; 100 are reserved for internal purposes, and VMIDs need to be
unique cluster wide.</td>
</tr></tbody></table>
</div>
<div class="listingblock">
<div class="title">Example VM Configuration</div>
<div class="content monospaced">
<pre>boot: order=virtio0;net0
cores: 1
sockets: 1
memory: 512
name: webmail
ostype: l26
net0: e1000=EE:D2:28:5F:B6:3E,bridge=vmbr0
virtio0: local:vm-100-disk-1,size=32G</pre>
</div></div>
<div class="paragraph">
<p>Those configuration files are simple text files, and you can edit them
using a normal text editor (<span class="monospaced">vi</span>, <span class="monospaced">nano</span>, …). This is sometimes
useful to do small corrections, but keep in mind that you need to
restart the VM to apply such changes.</p></div>
<div class="paragraph">
<p>For that reason, it is usually better to use the <span class="monospaced">qm</span> command to
generate and modify those files, or do the whole thing using the GUI.
Our toolkit is smart enough to instantaneously apply most changes to
running VM. This feature is called "hot plug", and there is no
need to restart the VM in that case.</p></div>
<div class="sect3">
<h4 id="_file_format">10.14.1. File Format
 <a class="headerlink" href="#_file_format" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>VM configuration files use a simple colon separated key/value
format. Each line has the following format:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># this is a comment
OPTION: value</pre>
</div></div>
<div class="paragraph">
<p>Blank lines in those files are ignored, and lines starting with a <span class="monospaced">#</span>
character are treated as comments and are also ignored.</p></div>
</div>
<div class="sect3">
<h4 id="qm_snapshots">10.14.2. Snapshots
 <a class="headerlink" href="#qm_snapshots" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>When you create a snapshot, <span class="monospaced">qm</span> stores the configuration at snapshot
time into a separate snapshot section within the same configuration
file. For example, after creating a snapshot called “testsnapshot”,
your configuration file will look like this:</p></div>
<div class="listingblock">
<div class="title">VM configuration with snapshot</div>
<div class="content monospaced">
<pre>memory: 512
swap: 512
parent: testsnaphot
...

[testsnaphot]
memory: 512
swap: 512
snaptime: 1457170803
...</pre>
</div></div>
<div class="paragraph">
<p>There are a few snapshot related properties like <span class="monospaced">parent</span> and
<span class="monospaced">snaptime</span>. The <span class="monospaced">parent</span> property is used to store the parent/child
relationship between snapshots. <span class="monospaced">snaptime</span> is the snapshot creation
time stamp (Unix epoch).</p></div>
<div class="paragraph">
<p>You can optionally save the memory of a running VM with the option <span class="monospaced">vmstate</span>.
For details about how the target storage gets chosen for the VM state, see
<a href="#qm_vmstatestorage">State storage selection</a> in the chapter
<a href="#qm_hibernate">Hibernation</a>.</p></div>
</div>
<div class="sect3">
<h4 id="qm_options">10.14.3. Options
 <a class="headerlink" href="#qm_options" title="Permalink to this heading"></a>
</h4>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">acpi</span>: <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Enable/disable ACPI.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">affinity</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
List of host cores used to execute guest processes, for example: 0,5,8-11
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">agent</span>: <span class="monospaced">[enabled=]&lt;1|0&gt; [,freeze-fs-on-backup=&lt;1|0&gt;] [,fstrim_cloned_disks=&lt;1|0&gt;] [,type=&lt;virtio|isa&gt;]</span> 
</dt>
<dd>
<p>
Enable/disable communication with the QEMU Guest Agent and its properties.
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">enabled</span>=<span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Enable/disable communication with a QEMU Guest Agent (QGA) running in the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">freeze-fs-on-backup</span>=<span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Freeze/thaw guest filesystems on backup for consistency.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">fstrim_cloned_disks</span>=<span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Run fstrim after moving a disk or migrating the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">type</span>=<span class="monospaced">&lt;isa | virtio&gt;</span> (<em>default =</em> <span class="monospaced">virtio</span>)
</dt>
<dd>
<p>
Select the agent type
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">arch</span>: <span class="monospaced">&lt;aarch64 | x86_64&gt;</span> 
</dt>
<dd>
<p>
Virtual processor architecture. Defaults to the host.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">args</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Arbitrary arguments passed to kvm, for example:
</p>
<div class="paragraph">
<p>args: -no-reboot -smbios <em>type=0,vendor=FOO</em></p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">this option is for experts only.</td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">audio0</span>: <span class="monospaced">device=&lt;ich9-intel-hda|intel-hda|AC97&gt; [,driver=&lt;spice|none&gt;]</span> 
</dt>
<dd>
<p>
Configure a audio device, useful in combination with QXL/Spice.
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">device</span>=<span class="monospaced">&lt;AC97 | ich9-intel-hda | intel-hda&gt;</span> 
</dt>
<dd>
<p>
Configure an audio device.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">driver</span>=<span class="monospaced">&lt;none | spice&gt;</span> (<em>default =</em> <span class="monospaced">spice</span>)
</dt>
<dd>
<p>
Driver backend for the audio device.
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">autostart</span>: <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Automatic restart after crash (currently ignored).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">balloon</span>: <span class="monospaced">&lt;integer&gt; (0 - N)</span> 
</dt>
<dd>
<p>
Amount of target RAM for the VM in MiB. Using zero disables the ballon driver.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">bios</span>: <span class="monospaced">&lt;ovmf | seabios&gt;</span> (<em>default =</em> <span class="monospaced">seabios</span>)
</dt>
<dd>
<p>
Select BIOS implementation.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">boot</span>: <span class="monospaced">[[legacy=]&lt;[acdn]{1,4}&gt;] [,order=&lt;device[;device...]&gt;]</span> 
</dt>
<dd>
<p>
Specify guest boot order. Use the <em>order=</em> sub-property as usage with no key or <em>legacy=</em> is deprecated.
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">legacy</span>=<span class="monospaced">&lt;[acdn]{1,4}&gt;</span> (<em>default =</em> <span class="monospaced">cdn</span>)
</dt>
<dd>
<p>
Boot on floppy (a), hard disk (c), CD-ROM (d), or network (n). Deprecated, use <em>order=</em> instead.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">order</span>=<span class="monospaced">&lt;device[;device...]&gt;</span> 
</dt>
<dd>
<p>
The guest will attempt to boot from devices in the order they appear here.
</p>
<div class="paragraph">
<p>Disks, optical drives and passed-through storage USB devices will be directly
booted from, NICs will load PXE, and PCIe devices will either behave like disks
(e.g. NVMe) or load an option ROM (e.g. RAID controller, hardware NIC).</p></div>
<div class="paragraph">
<p>Note that only devices in this list will be marked as bootable and thus loaded
by the guest firmware (BIOS/UEFI). If you require multiple disks for booting
(e.g. software-raid), you need to specify all of them here.</p></div>
<div class="paragraph">
<p>Overrides the deprecated <em>legacy=[acdn]*</em> value when given.</p></div>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">bootdisk</span>: <span class="monospaced">(ide|sata|scsi|virtio)\d+</span> 
</dt>
<dd>
<p>
Enable booting from specified disk. Deprecated: Use <em>boot: order=foo;bar</em> instead.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">cdrom</span>: <span class="monospaced">&lt;volume&gt;</span> 
</dt>
<dd>
<p>
This is an alias for option -ide2
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">cicustom</span>: <span class="monospaced">[meta=&lt;volume&gt;] [,network=&lt;volume&gt;] [,user=&lt;volume&gt;] [,vendor=&lt;volume&gt;]</span> 
</dt>
<dd>
<p>
cloud-init: Specify custom files to replace the automatically generated ones at start.
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">meta</span>=<span class="monospaced">&lt;volume&gt;</span> 
</dt>
<dd>
<p>
Specify a custom file containing all meta data passed to the VM via"
            ." cloud-init. This is provider specific meaning configdrive2 and nocloud differ.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">network</span>=<span class="monospaced">&lt;volume&gt;</span> 
</dt>
<dd>
<p>
To pass a custom file containing all network data to the VM via cloud-init.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">user</span>=<span class="monospaced">&lt;volume&gt;</span> 
</dt>
<dd>
<p>
To pass a custom file containing all user data to the VM via cloud-init.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">vendor</span>=<span class="monospaced">&lt;volume&gt;</span> 
</dt>
<dd>
<p>
To pass a custom file containing all vendor data to the VM via cloud-init.
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">cipassword</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
cloud-init: Password to assign the user. Using this is generally not recommended. Use ssh keys instead. Also note that older cloud-init versions do not support hashed passwords.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">citype</span>: <span class="monospaced">&lt;configdrive2 | nocloud | opennebula&gt;</span> 
</dt>
<dd>
<p>
Specifies the cloud-init configuration format. The default depends on the configured operating system type (<span class="monospaced">ostype</span>. We use the <span class="monospaced">nocloud</span> format for Linux, and <span class="monospaced">configdrive2</span> for windows.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">ciupgrade</span>: <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
cloud-init: do an automatic package upgrade after the first boot.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">ciuser</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
cloud-init: User name to change ssh keys and password for instead of the image’s configured default user.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">cores</span>: <span class="monospaced">&lt;integer&gt; (1 - N)</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
The number of cores per socket.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">cpu</span>: <span class="monospaced">[[cputype=]&lt;string&gt;] [,flags=&lt;+FLAG[;-FLAG...]&gt;] [,hidden=&lt;1|0&gt;] [,hv-vendor-id=&lt;vendor-id&gt;] [,phys-bits=&lt;8-64|host&gt;] [,reported-model=&lt;enum&gt;]</span> 
</dt>
<dd>
<p>
Emulated CPU type.
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">cputype</span>=<span class="monospaced">&lt;string&gt;</span> (<em>default =</em> <span class="monospaced">kvm64</span>)
</dt>
<dd>
<p>
Emulated CPU type. Can be default or custom name (custom model names must be prefixed with <em>custom-</em>).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">flags</span>=<span class="monospaced">&lt;+FLAG[;-FLAG...]&gt;</span> 
</dt>
<dd>
<p>
List of additional CPU flags separated by <em>;</em>. Use <em>+FLAG</em> to enable, <em>-FLAG</em> to disable a flag. Custom CPU models can specify any flag supported by QEMU/KVM, VM-specific flags must be from the following set for security reasons: pcid, spec-ctrl, ibpb, ssbd, virt-ssbd, amd-ssbd, amd-no-ssb, pdpe1gb, md-clear, hv-tlbflush, hv-evmcs, aes
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">hidden</span>=<span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Do not identify as a KVM virtual machine.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">hv-vendor-id</span>=<span class="monospaced">&lt;vendor-id&gt;</span> 
</dt>
<dd>
<p>
The Hyper-V vendor ID. Some drivers or programs inside Windows guests need a specific ID.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">phys-bits</span>=<span class="monospaced">&lt;8-64|host&gt;</span> 
</dt>
<dd>
<p>
The physical memory address bits that are reported to the guest OS. Should be smaller or equal to the host’s. Set to <em>host</em> to use value from host CPU, but note that doing so will break live migration to CPUs with other values.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">reported-model</span>=<span class="monospaced">&lt;486 | Broadwell | Broadwell-IBRS | Broadwell-noTSX | Broadwell-noTSX-IBRS | Cascadelake-Server | Cascadelake-Server-noTSX | Cascadelake-Server-v2 | Cascadelake-Server-v4 | Cascadelake-Server-v5 | Conroe | Cooperlake | Cooperlake-v2 | EPYC | EPYC-Genoa | EPYC-IBPB | EPYC-Milan | EPYC-Milan-v2 | EPYC-Rome | EPYC-Rome-v2 | EPYC-Rome-v3 | EPYC-Rome-v4 | EPYC-v3 | EPYC-v4 | GraniteRapids | Haswell | Haswell-IBRS | Haswell-noTSX | Haswell-noTSX-IBRS | Icelake-Client | Icelake-Client-noTSX | Icelake-Server | Icelake-Server-noTSX | Icelake-Server-v3 | Icelake-Server-v4 | Icelake-Server-v5 | Icelake-Server-v6 | IvyBridge | IvyBridge-IBRS | KnightsMill | Nehalem | Nehalem-IBRS | Opteron_G1 | Opteron_G2 | Opteron_G3 | Opteron_G4 | Opteron_G5 | Penryn | SandyBridge | SandyBridge-IBRS | SapphireRapids | SapphireRapids-v2 | Skylake-Client | Skylake-Client-IBRS | Skylake-Client-noTSX-IBRS | Skylake-Client-v4 | Skylake-Server | Skylake-Server-IBRS | Skylake-Server-noTSX-IBRS | Skylake-Server-v4 | Skylake-Server-v5 | Westmere | Westmere-IBRS | athlon | core2duo | coreduo | host | kvm32 | kvm64 | max | pentium | pentium2 | pentium3 | phenom | qemu32 | qemu64&gt;</span> (<em>default =</em> <span class="monospaced">kvm64</span>)
</dt>
<dd>
<p>
CPU model and vendor to report to the guest. Must be a QEMU/KVM supported model. Only valid for custom CPU model definitions, default models will always report themselves to the guest OS.
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">cpulimit</span>: <span class="monospaced">&lt;number&gt; (0 - 128)</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Limit of CPU usage.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">If the computer has 2 CPUs, it has total of <em>2</em> CPU time. Value <em>0</em> indicates no CPU limit.</td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">cpuunits</span>: <span class="monospaced">&lt;integer&gt; (1 - 262144)</span> (<em>default =</em> <span class="monospaced">cgroup v1: 1024, cgroup v2: 100</span>)
</dt>
<dd>
<p>
CPU weight for a VM. Argument is used in the kernel fair scheduler. The larger the number is, the more CPU time this VM gets. Number is relative to weights of all the other running VMs.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">description</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Description for the VM. Shown in the web-interface VM’s summary. This is saved as comment inside the configuration file.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">efidisk0</span>: <span class="monospaced">[file=]&lt;volume&gt; [,efitype=&lt;2m|4m&gt;] [,format=&lt;enum&gt;] [,pre-enrolled-keys=&lt;1|0&gt;] [,size=&lt;DiskSize&gt;]</span> 
</dt>
<dd>
<p>
Configure a disk for storing EFI vars.
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">efitype</span>=<span class="monospaced">&lt;2m | 4m&gt;</span> (<em>default =</em> <span class="monospaced">2m</span>)
</dt>
<dd>
<p>
Size and type of the OVMF EFI vars. <em>4m</em> is newer and recommended, and required for Secure Boot. For backwards compatibility, <em>2m</em> is used if not otherwise specified. Ignored for VMs with arch=aarch64 (ARM).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">file</span>=<span class="monospaced">&lt;volume&gt;</span> 
</dt>
<dd>
<p>
The drive’s backing volume.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">format</span>=<span class="monospaced">&lt;cloop | cow | qcow | qcow2 | qed | raw | vmdk&gt;</span> 
</dt>
<dd>
<p>
The drive’s backing file’s data format.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">pre-enrolled-keys</span>=<span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Use am EFI vars template with distribution-specific and Microsoft Standard keys enrolled, if used with <em>efitype=4m</em>. Note that this will enable Secure Boot by default, though it can still be turned off from within the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">size</span>=<span class="monospaced">&lt;DiskSize&gt;</span> 
</dt>
<dd>
<p>
Disk size. This is purely informational and has no effect.
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">freeze</span>: <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Freeze CPU at startup (use <em>c</em> monitor command to start execution).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">hookscript</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Script that will be executed during various steps in the vms lifetime.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">hostpci[n]</span>: <span class="monospaced">[[host=]&lt;HOSTPCIID[;HOSTPCIID2...]&gt;] [,device-id=&lt;hex id&gt;] [,legacy-igd=&lt;1|0&gt;] [,mapping=&lt;mapping-id&gt;] [,mdev=&lt;string&gt;] [,pcie=&lt;1|0&gt;] [,rombar=&lt;1|0&gt;] [,romfile=&lt;string&gt;] [,sub-device-id=&lt;hex id&gt;] [,sub-vendor-id=&lt;hex id&gt;] [,vendor-id=&lt;hex id&gt;] [,x-vga=&lt;1|0&gt;]</span> 
</dt>
<dd>
<p>
Map host PCI devices into guest.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">This option allows direct access to host hardware. So it is no longer
possible to migrate such machines - use with special care.</td>
</tr></tbody></table>
</div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Caution" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKdUlEQVRoge1Ze1AV1x3+zt279wEi
DWCYGzRVktqa1MEmmtbWR22ncXxUrTDWV/5IqG2wUAUfmUwSkk6V+EAJCEQC6figwZBqZtRxqukf
tebRZBpFG1S0hiRErwiaKK977+6eX//YPXt37+UiIJlMZnpmdnb33PP4vt97z2VEhG9yc3zdAO60
/Z/A192+8QScX8Wifr+fWltbzffU1FT4fD72Vew15ASampqovr4eBw8eNPvGjRuHzMxMmj9//tCT
IKIhu958801yuVwEoNdr48aNNJT7EdHQEdi/fz/JshwTvLiKioqGlMSQLFJfX0+MMRvQsWNSqXLz
H2jez78fRWLTpk1DRuKOF6irqyOn02kDOP8XGdTz+VFSP91Hatu79NRvZ0SR2LJly5CQuKPJtbW1
JEmSDdivHp1AwctHSblYSsqFElLOF5PavIvWZU+LIrF169Y7JjHoibt3744CnzXrIQpeOUbKhVJS
mkpIPV9MyrnNpJwtIuVSNa15/MdRJLZt23ZHJBjRwIu5N954gxYvXgzOudm3aO5E1L5SBNZ1Hoxr
INLAyH6HJxVP/WkXtu9+37ZeSUkJVq9ePagQO+BMXFNTEwV+ybxHsPeVIjg6zwOkgSWNhuOuUUBC
KhxJY0AggDSgpwWbn1mO1csftq1ZUFCA0tLSQZXFA0pkVVVVlJOTA6vWsmY9hF07N0AS4OU48LgR
WLO+FPFxboxMS8G5hlPYmPMI4jwM1P0Jtj67DJyrKHvtNADdjPPz8+FwOCgvL29gmuivrVVUVESF
yqxZD1PoyjFSL+7QHbapmJTml+n9EzVRtl676ZekNKwl5VQ+KSfzKHR2E+UutodYxhiVl5cPyCf6
ZUJlZWWUm5trk/wTi6bgtVdf1M2Gq2DEwUgDNBWhUDBqjfSRwwGoIOEP3c3Y/uxirFw0zibMvLw8
VFZW9tucbkugoqKCVq1aZQO/YslUVJU+D9bVBIDroLgK4hygEDweybYGY8AD6d8yftfASL+j+2O8
9NwiPJk51kYiNzcXO3fu7BeJPgkUFxdTbm6ure/JZdNRub0QrOMcGFcBrgLgAHEAGqD2ID7OZZsz
Ji0RcR6HAV4DkaqPJxXouoSywkz8buF9NhIrV65EVVXVbUnEJHDixAlat26drS9n+U+xo/hZMMNh
yZA+uAbAIKOGMCJlmG3eA/clAYaJEamGBlRdC6QBXZewo3AhVswfbSORk5ODDz/8sE8SvRLw+/1U
XV0dBb5MgOe6LUNIEqpOwgCUlOCESw4HuAfG3KX7B2kAGaTJ0Jro77yI8ucX4DfzRtlIHD58GH6/
PyaJXgk0NjZi79695vuUiffrkr91Vt8cHAwc4FZJGiZEKqDcwsi0ZHP+d749POy8MS7iGtBxARWF
8/CDsQnm3BdeeAHWj6N+EYhspSVbgI5zYYkZGmAwzIEbwA0tUM9NfO+7aeb89HuGRQBWLcSNZxj3
zvOYMzk5NpjBEHi99mXD5lXdjqGDJh42h7CEOXhnK3448X5z/shUb1jSpIEM8yFuzOUcjAsNcnR0
KXdG4MEHH8TEiRPN9z0H3sfRE41m+GNcs9i0LjkGTTcrUoGem5g7/V5z/ohElw4O+jxGYQ3qJqmB
jN/+cbINtcfCJvPYY4+hr3qtVwI+n49lZmaa71fbbuH3hX/FycbPQKSCoIIMQEQcehjVo5Gw8/F3
3cSvH03H2HuHY1icQzcvYWKGIMiITMwQwjtnruPpqk9w/ZZq7p2RkYEJEyaAYrCIWY36/X7KyMhA
W1ub2XfP3Qmo3ZKFaQ+P0gmYIdSoOkU4lWSwxGRAdqHL/xm86k27w0blAw3/PncD6yo/xrsfdZj7
paSkoLGxESNGjAgDZsxWK8X0AZ/Px44fP460tLAzXrnWgUX5r+PYuxf1mG/Gf80sofWy2YuWbg/+
uONtPLHhPRw+0WIHL8hbwG95rcUGPiEhAYcOHUJSUhI456YZRWqiz+8BIqLm5mZMmjQJN27cMPsT
E9zYtWE25k4bYzh3mABIBRwOHPgPx+LH/6yDiXPi6pHZcEpk+I5qmtGpCzdQecCPPUevmevLsowj
R45gypQpcDgc5mVowKaJmBoQTEePHo2GhgaMGxcuum52BLHsqUOoP3oWgLB9Sz7QgpgxJoThw/SS
IutnaXBKZEpcgD998Uvs+3ubDbzT6URNTQ0mTZoEVVWhaRo459A0TeCy4etVA+JH8RvnHK2trZgz
Zw5Onz5tjnPJDrxSOAPLZt1nSVIqwDmIVJw6dx1nLt3A3J/cjZThkhE29cvf3o2KA59ja91lcz3G
GDZv3oylS5fC7XbD6XTC6XRCkiRIktS7Jm5HQLDXNA3Xr19HVlYWPvjgA4vEGLav+RHmTxsJX4rb
8AsO4kbOIG6GzTD4HvzlmB/PVH9q2zc/Px/Z2dnwer1wu92QZRkulwuSJEGWZZMEY8wkEGVCVvCC
AOcciqIgPj4e+/btw9SpU83xqkpYteU9HHm7Bf62rnC2RjikisTFSMPV9m4ceucannvVDn7p0qVY
uHAhgsEggsEgQqFQlAlZnVm0PjOxIKFpGogIqqrC5XKhuroa06dPt4wDcl78F+qPNeNKe6cRoTjs
ZYOKK+3dON7wBQrKm2H5pMbMmTOxZMkSBAIB9PT0oKenB4FAAIqiQFEUU4hRRyq3I2AlIZ4553A6
nSgvL8fs2bNtY9eVNaDub58YmhDZWpe+v70bJ5tu4clt/4WihqU4efJkrFixAkRkAg4Gg1AUBaqq
muDF/pGtX7WQNXeIZ0mSUFRUhAULFtjGPl3ZiJdev4QrbZ16zUQqrrZ342JLN5ZvaEJ3ICz68ePH
IycnB5qmmZIW4EOhUJTZWEKo+Rx1KsEYY9Zk4XA4IEmSKXnOuRkVJElCYWEhvF4v6urqzDVK9n2M
zu4Qnl5+L9q/7MHltgBWbr9kA5+eno7c3FzIsoyI5GrTeCzgosU8VmGMmSGLiEwSsixH2WNBQQEA
2EhUH/wc7330BaaOH479/2zHtS/CFabP58OaNWvg9Xp1EE4nZFmGLMtwu93wer1wuVy2ECpJko2I
iTNWJrZGI+HEwi6FisUVCAQQCASwZ88eVFZW9lk9JicnY/369fD5fPB4PCbo+Ph4k4TH44HL5YLX
64XX6zXH9EKE9aUBRkTEGLNJP1Kd4nI4HMjOzkZ8fDyKi4ttJ3eiJSYmYu3atRg1apQJzuv1wuPx
wOPxwOFwmEBFHnA6nXC5XGYSiywlbns2SnoDANN0hMNZnS0UCpkaOnPmDN566y20tLSY2ktLS8OM
GTOQkpICt9uNuLg4M2EJ8CJZSZJkZmFBwkrAWpH2+3BXEBEkRIhTVdVMOCL5qKqKy5cvQ1EUaJoG
VVWRnKx/Jlpt3eVy2QCKu9C68AFr9o0spwd0Oh1ZYggCIlNaiQktiXdhZgKQABcp3ci7LMtM7B0J
fsAErJoAYItG1mdr+hfvVlACtHDIXkploYXbHvQOikDEe6/ZmjFmAhf9ZvJxOk2Qol88W4kYz32S
GNQfHLG+TyMTkAVEr88xQfVnkBg7GAKixSIykDYQsL21/wFkW/B5QqT9lwAAAABJRU5ErkJggg==">
</td>
<td class="content">Experimental! User reported problems with this option.</td>
</tr></tbody></table>
</div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">device-id</span>=<span class="monospaced">&lt;hex id&gt;</span> 
</dt>
<dd>
<p>
Override PCI device ID visible to guest
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">host</span>=<span class="monospaced">&lt;HOSTPCIID[;HOSTPCIID2...]&gt;</span> 
</dt>
<dd>
<p>
Host PCI device pass through. The PCI ID of a host’s PCI device or a list
of PCI virtual functions of the host. HOSTPCIID syntax is:
</p>
<div class="paragraph">
<p><em>bus:dev.func</em> (hexadecimal numbers)</p></div>
<div class="paragraph">
<p>You can us the <em>lspci</em> command to list existing PCI devices.</p></div>
<div class="paragraph">
<p>Either this or the <em>mapping</em> key must be set.</p></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">legacy-igd</span>=<span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Pass this device in legacy IGD mode, making it the primary and exclusive graphics device in the VM. Requires <em>pc-i440fx</em> machine type and VGA set to <em>none</em>.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mapping</span>=<span class="monospaced">&lt;mapping-id&gt;</span> 
</dt>
<dd>
<p>
The ID of a cluster wide mapping. Either this or the default-key <em>host</em> must be set.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mdev</span>=<span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The type of mediated device to use.
An instance of this type will be created on startup of the VM and
will be cleaned up when the VM stops.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">pcie</span>=<span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Choose the PCI-express bus (needs the <em>q35</em> machine model).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">rombar</span>=<span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Specify whether or not the device’s ROM will be visible in the guest’s memory map.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">romfile</span>=<span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Custom pci device rom filename (must be located in /usr/share/kvm/).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">sub-device-id</span>=<span class="monospaced">&lt;hex id&gt;</span> 
</dt>
<dd>
<p>
Override PCI subsystem device ID visible to guest
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">sub-vendor-id</span>=<span class="monospaced">&lt;hex id&gt;</span> 
</dt>
<dd>
<p>
Override PCI subsystem vendor ID visible to guest
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">vendor-id</span>=<span class="monospaced">&lt;hex id&gt;</span> 
</dt>
<dd>
<p>
Override PCI vendor ID visible to guest
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">x-vga</span>=<span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Enable vfio-vga device support.
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">hotplug</span>: <span class="monospaced">&lt;string&gt;</span> (<em>default =</em> <span class="monospaced">network,disk,usb</span>)
</dt>
<dd>
<p>
Selectively enable hotplug features. This is a comma separated list of hotplug features: <em>network</em>, <em>disk</em>, <em>cpu</em>, <em>memory</em>, <em>usb</em> and <em>cloudinit</em>. Use <em>0</em> to disable hotplug completely. Using <em>1</em> as value is an alias for the default <span class="monospaced">network,disk,usb</span>. USB hotplugging is possible for guests with machine version &gt;= 7.1 and ostype l26 or windows &gt; 7.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">hugepages</span>: <span class="monospaced">&lt;1024 | 2 | any&gt;</span> 
</dt>
<dd>
<p>
Enable/disable hugepages memory.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">ide[n]</span>: <span class="monospaced">[file=]&lt;volume&gt; [,aio=&lt;native|threads|io_uring&gt;] [,backup=&lt;1|0&gt;] [,bps=&lt;bps&gt;] [,bps_max_length=&lt;seconds&gt;] [,bps_rd=&lt;bps&gt;] [,bps_rd_max_length=&lt;seconds&gt;] [,bps_wr=&lt;bps&gt;] [,bps_wr_max_length=&lt;seconds&gt;] [,cache=&lt;enum&gt;] [,cyls=&lt;integer&gt;] [,detect_zeroes=&lt;1|0&gt;] [,discard=&lt;ignore|on&gt;] [,format=&lt;enum&gt;] [,heads=&lt;integer&gt;] [,iops=&lt;iops&gt;] [,iops_max=&lt;iops&gt;] [,iops_max_length=&lt;seconds&gt;] [,iops_rd=&lt;iops&gt;] [,iops_rd_max=&lt;iops&gt;] [,iops_rd_max_length=&lt;seconds&gt;] [,iops_wr=&lt;iops&gt;] [,iops_wr_max=&lt;iops&gt;] [,iops_wr_max_length=&lt;seconds&gt;] [,mbps=&lt;mbps&gt;] [,mbps_max=&lt;mbps&gt;] [,mbps_rd=&lt;mbps&gt;] [,mbps_rd_max=&lt;mbps&gt;] [,mbps_wr=&lt;mbps&gt;] [,mbps_wr_max=&lt;mbps&gt;] [,media=&lt;cdrom|disk&gt;] [,model=&lt;model&gt;] [,replicate=&lt;1|0&gt;] [,rerror=&lt;ignore|report|stop&gt;] [,secs=&lt;integer&gt;] [,serial=&lt;serial&gt;] [,shared=&lt;1|0&gt;] [,size=&lt;DiskSize&gt;] [,snapshot=&lt;1|0&gt;] [,ssd=&lt;1|0&gt;] [,trans=&lt;none|lba|auto&gt;] [,werror=&lt;enum&gt;] [,wwn=&lt;wwn&gt;]</span> 
</dt>
<dd>
<p>
Use volume as IDE hard disk or CD-ROM (n is 0 to 3).
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">aio</span>=<span class="monospaced">&lt;io_uring | native | threads&gt;</span> 
</dt>
<dd>
<p>
AIO type to use.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">backup</span>=<span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Whether the drive should be included when making backups.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">bps</span>=<span class="monospaced">&lt;bps&gt;</span> 
</dt>
<dd>
<p>
Maximum r/w speed in bytes per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">bps_max_length</span>=<span class="monospaced">&lt;seconds&gt;</span> 
</dt>
<dd>
<p>
Maximum length of I/O bursts in seconds.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">bps_rd</span>=<span class="monospaced">&lt;bps&gt;</span> 
</dt>
<dd>
<p>
Maximum read speed in bytes per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">bps_rd_max_length</span>=<span class="monospaced">&lt;seconds&gt;</span> 
</dt>
<dd>
<p>
Maximum length of read I/O bursts in seconds.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">bps_wr</span>=<span class="monospaced">&lt;bps&gt;</span> 
</dt>
<dd>
<p>
Maximum write speed in bytes per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">bps_wr_max_length</span>=<span class="monospaced">&lt;seconds&gt;</span> 
</dt>
<dd>
<p>
Maximum length of write I/O bursts in seconds.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">cache</span>=<span class="monospaced">&lt;directsync | none | unsafe | writeback | writethrough&gt;</span> 
</dt>
<dd>
<p>
The drive’s cache mode
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">cyls</span>=<span class="monospaced">&lt;integer&gt;</span> 
</dt>
<dd>
<p>
Force the drive’s physical geometry to have a specific cylinder count.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">detect_zeroes</span>=<span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Controls whether to detect and try to optimize writes of zeroes.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">discard</span>=<span class="monospaced">&lt;ignore | on&gt;</span> 
</dt>
<dd>
<p>
Controls whether to pass discard/trim requests to the underlying storage.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">file</span>=<span class="monospaced">&lt;volume&gt;</span> 
</dt>
<dd>
<p>
The drive’s backing volume.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">format</span>=<span class="monospaced">&lt;cloop | cow | qcow | qcow2 | qed | raw | vmdk&gt;</span> 
</dt>
<dd>
<p>
The drive’s backing file’s data format.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">heads</span>=<span class="monospaced">&lt;integer&gt;</span> 
</dt>
<dd>
<p>
Force the drive’s physical geometry to have a specific head count.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">iops</span>=<span class="monospaced">&lt;iops&gt;</span> 
</dt>
<dd>
<p>
Maximum r/w I/O in operations per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">iops_max</span>=<span class="monospaced">&lt;iops&gt;</span> 
</dt>
<dd>
<p>
Maximum unthrottled r/w I/O pool in operations per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">iops_max_length</span>=<span class="monospaced">&lt;seconds&gt;</span> 
</dt>
<dd>
<p>
Maximum length of I/O bursts in seconds.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">iops_rd</span>=<span class="monospaced">&lt;iops&gt;</span> 
</dt>
<dd>
<p>
Maximum read I/O in operations per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">iops_rd_max</span>=<span class="monospaced">&lt;iops&gt;</span> 
</dt>
<dd>
<p>
Maximum unthrottled read I/O pool in operations per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">iops_rd_max_length</span>=<span class="monospaced">&lt;seconds&gt;</span> 
</dt>
<dd>
<p>
Maximum length of read I/O bursts in seconds.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">iops_wr</span>=<span class="monospaced">&lt;iops&gt;</span> 
</dt>
<dd>
<p>
Maximum write I/O in operations per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">iops_wr_max</span>=<span class="monospaced">&lt;iops&gt;</span> 
</dt>
<dd>
<p>
Maximum unthrottled write I/O pool in operations per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">iops_wr_max_length</span>=<span class="monospaced">&lt;seconds&gt;</span> 
</dt>
<dd>
<p>
Maximum length of write I/O bursts in seconds.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mbps</span>=<span class="monospaced">&lt;mbps&gt;</span> 
</dt>
<dd>
<p>
Maximum r/w speed in megabytes per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mbps_max</span>=<span class="monospaced">&lt;mbps&gt;</span> 
</dt>
<dd>
<p>
Maximum unthrottled r/w pool in megabytes per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mbps_rd</span>=<span class="monospaced">&lt;mbps&gt;</span> 
</dt>
<dd>
<p>
Maximum read speed in megabytes per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mbps_rd_max</span>=<span class="monospaced">&lt;mbps&gt;</span> 
</dt>
<dd>
<p>
Maximum unthrottled read pool in megabytes per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mbps_wr</span>=<span class="monospaced">&lt;mbps&gt;</span> 
</dt>
<dd>
<p>
Maximum write speed in megabytes per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mbps_wr_max</span>=<span class="monospaced">&lt;mbps&gt;</span> 
</dt>
<dd>
<p>
Maximum unthrottled write pool in megabytes per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">media</span>=<span class="monospaced">&lt;cdrom | disk&gt;</span> (<em>default =</em> <span class="monospaced">disk</span>)
</dt>
<dd>
<p>
The drive’s media type.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">model</span>=<span class="monospaced">&lt;model&gt;</span> 
</dt>
<dd>
<p>
The drive’s reported model name, url-encoded, up to 40 bytes long.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">replicate</span>=<span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Whether the drive should considered for replication jobs.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">rerror</span>=<span class="monospaced">&lt;ignore | report | stop&gt;</span> 
</dt>
<dd>
<p>
Read error action.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">secs</span>=<span class="monospaced">&lt;integer&gt;</span> 
</dt>
<dd>
<p>
Force the drive’s physical geometry to have a specific sector count.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">serial</span>=<span class="monospaced">&lt;serial&gt;</span> 
</dt>
<dd>
<p>
The drive’s reported serial number, url-encoded, up to 20 bytes long.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">shared</span>=<span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Mark this locally-managed volume as available on all nodes.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">This option does not share the volume automatically, it assumes it is shared already!</td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">size</span>=<span class="monospaced">&lt;DiskSize&gt;</span> 
</dt>
<dd>
<p>
Disk size. This is purely informational and has no effect.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">snapshot</span>=<span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Controls qemu’s snapshot mode feature. If activated, changes made to the disk are temporary and will be discarded when the VM is shutdown.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">ssd</span>=<span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Whether to expose this drive as an SSD, rather than a rotational hard disk.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">trans</span>=<span class="monospaced">&lt;auto | lba | none&gt;</span> 
</dt>
<dd>
<p>
Force disk geometry bios translation mode.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">werror</span>=<span class="monospaced">&lt;enospc | ignore | report | stop&gt;</span> 
</dt>
<dd>
<p>
Write error action.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">wwn</span>=<span class="monospaced">&lt;wwn&gt;</span> 
</dt>
<dd>
<p>
The drive’s worldwide name, encoded as 16 bytes hex string, prefixed by <em>0x</em>.
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">ipconfig[n]</span>: <span class="monospaced">[gw=&lt;GatewayIPv4&gt;] [,gw6=&lt;GatewayIPv6&gt;] [,ip=&lt;IPv4Format/CIDR&gt;] [,ip6=&lt;IPv6Format/CIDR&gt;]</span> 
</dt>
<dd>
<p>
cloud-init: Specify IP addresses and gateways for the corresponding interface.
</p>
<div class="paragraph">
<p>IP addresses use CIDR notation, gateways are optional but need an IP of the same type specified.</p></div>
<div class="paragraph">
<p>The special string <em>dhcp</em> can be used for IP addresses to use DHCP, in which case no explicit
gateway should be provided.
For IPv6 the special string <em>auto</em> can be used to use stateless autoconfiguration. This requires
cloud-init 19.4 or newer.</p></div>
<div class="paragraph">
<p>If cloud-init is enabled and neither an IPv4 nor an IPv6 address is specified, it defaults to using
dhcp on IPv4.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">gw</span>=<span class="monospaced">&lt;GatewayIPv4&gt;</span> 
</dt>
<dd>
<p>
Default gateway for IPv4 traffic.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Requires option(s): <span class="monospaced">ip</span></td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">gw6</span>=<span class="monospaced">&lt;GatewayIPv6&gt;</span> 
</dt>
<dd>
<p>
Default gateway for IPv6 traffic.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Requires option(s): <span class="monospaced">ip6</span></td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">ip</span>=<span class="monospaced">&lt;IPv4Format/CIDR&gt;</span> (<em>default =</em> <span class="monospaced">dhcp</span>)
</dt>
<dd>
<p>
IPv4 address in CIDR format.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">ip6</span>=<span class="monospaced">&lt;IPv6Format/CIDR&gt;</span> (<em>default =</em> <span class="monospaced">dhcp</span>)
</dt>
<dd>
<p>
IPv6 address in CIDR format.
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">ivshmem</span>: <span class="monospaced">size=&lt;integer&gt; [,name=&lt;string&gt;]</span> 
</dt>
<dd>
<p>
Inter-VM shared memory. Useful for direct communication between VMs, or to the host.
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">name</span>=<span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The name of the file. Will be prefixed with <em>pve-shm-</em>. Default is the VMID. Will be deleted when the VM is stopped.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">size</span>=<span class="monospaced">&lt;integer&gt; (1 - N)</span> 
</dt>
<dd>
<p>
The size of the file in MB.
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">keephugepages</span>: <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Use together with hugepages. If enabled, hugepages will not not be deleted after VM shutdown and can be used for subsequent starts.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">keyboard</span>: <span class="monospaced">&lt;da | de | de-ch | en-gb | en-us | es | fi | fr | fr-be | fr-ca | fr-ch | hu | is | it | ja | lt | mk | nl | no | pl | pt | pt-br | sl | sv | tr&gt;</span> 
</dt>
<dd>
<p>
Keyboard layout for VNC server. This option is generally not required and is often better handled from within the guest OS.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">kvm</span>: <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Enable/disable KVM hardware virtualization.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">localtime</span>: <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Set the real time clock (RTC) to local time. This is enabled by default if the <span class="monospaced">ostype</span> indicates a Microsoft Windows OS.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">lock</span>: <span class="monospaced">&lt;backup | clone | create | migrate | rollback | snapshot | snapshot-delete | suspended | suspending&gt;</span> 
</dt>
<dd>
<p>
Lock/unlock the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">machine</span>: <span class="monospaced">[[type=]&lt;machine type&gt;] [,viommu=&lt;intel|virtio&gt;]</span> 
</dt>
<dd>
<p>
Specify the QEMU machine.
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">type</span>=<span class="monospaced">&lt;machine type&gt;</span> 
</dt>
<dd>
<p>
Specifies the QEMU machine type.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">viommu</span>=<span class="monospaced">&lt;intel | virtio&gt;</span> 
</dt>
<dd>
<p>
Enable and set guest vIOMMU variant (Intel vIOMMU needs q35 to be set as machine type).
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">memory</span>: <span class="monospaced">[current=]&lt;integer&gt;</span> 
</dt>
<dd>
<p>
Memory properties.
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">current</span>=<span class="monospaced">&lt;integer&gt; (16 - N)</span> (<em>default =</em> <span class="monospaced">512</span>)
</dt>
<dd>
<p>
Current amount of online RAM for the VM in MiB. This is the maximum available memory when you use the balloon device.
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">migrate_downtime</span>: <span class="monospaced">&lt;number&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">0.1</span>)
</dt>
<dd>
<p>
Set maximum tolerated downtime (in seconds) for migrations. Should the migration not be able to converge in the very end, because too much newly dirtied RAM needs to be transferred, the limit will be increased automatically step-by-step until migration can converge.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">migrate_speed</span>: <span class="monospaced">&lt;integer&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Set maximum speed (in MB/s) for migrations. Value 0 is no limit.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">name</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Set a name for the VM. Only used on the configuration web interface.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">nameserver</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
cloud-init: Sets DNS server IP address for a container. Create will automatically use the setting from the host if neither searchdomain nor nameserver are set.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">net[n]</span>: <span class="monospaced">[model=]&lt;enum&gt; [,bridge=&lt;bridge&gt;] [,firewall=&lt;1|0&gt;] [,link_down=&lt;1|0&gt;] [,macaddr=&lt;XX:XX:XX:XX:XX:XX&gt;] [,mtu=&lt;integer&gt;] [,queues=&lt;integer&gt;] [,rate=&lt;number&gt;] [,tag=&lt;integer&gt;] [,trunks=&lt;vlanid[;vlanid...]&gt;] [,&lt;model&gt;=&lt;macaddr&gt;]</span> 
</dt>
<dd>
<p>
Specify network devices.
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">bridge</span>=<span class="monospaced">&lt;bridge&gt;</span> 
</dt>
<dd>
<p>
Bridge to attach the network device to. The Proxmox VE standard bridge
is called <em>vmbr0</em>.
</p>
<div class="paragraph">
<p>If you do not specify a bridge, we create a kvm user (NATed) network
device, which provides DHCP and DNS services. The following addresses
are used:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>10.0.2.2   Gateway
10.0.2.3   DNS Server
10.0.2.4   SMB Server</pre>
</div></div>
<div class="paragraph">
<p>The DHCP server assign addresses to the guest starting from 10.0.2.15.</p></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">firewall</span>=<span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Whether this interface should be protected by the firewall.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">link_down</span>=<span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Whether this interface should be disconnected (like pulling the plug).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">macaddr</span>=<span class="monospaced">&lt;XX:XX:XX:XX:XX:XX&gt;</span> 
</dt>
<dd>
<p>
A common MAC address with the I/G (Individual/Group) bit not set.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">model</span>=<span class="monospaced">&lt;e1000 | e1000-82540em | e1000-82544gc | e1000-82545em | e1000e | i82551 | i82557b | i82559er | ne2k_isa | ne2k_pci | pcnet | rtl8139 | virtio | vmxnet3&gt;</span> 
</dt>
<dd>
<p>
Network Card Model. The <em>virtio</em> model provides the best performance with very low CPU overhead. If your guest does not support this driver, it is usually best to use <em>e1000</em>.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mtu</span>=<span class="monospaced">&lt;integer&gt; (1 - 65520)</span> 
</dt>
<dd>
<p>
Force MTU, for VirtIO only. Set to <em>1</em> to use the bridge MTU
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">queues</span>=<span class="monospaced">&lt;integer&gt; (0 - 64)</span> 
</dt>
<dd>
<p>
Number of packet queues to be used on the device.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">rate</span>=<span class="monospaced">&lt;number&gt; (0 - N)</span> 
</dt>
<dd>
<p>
Rate limit in mbps (megabytes per second) as floating point number.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">tag</span>=<span class="monospaced">&lt;integer&gt; (1 - 4094)</span> 
</dt>
<dd>
<p>
VLAN tag to apply to packets on this interface.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">trunks</span>=<span class="monospaced">&lt;vlanid[;vlanid...]&gt;</span> 
</dt>
<dd>
<p>
VLAN trunks to pass through this interface.
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">numa</span>: <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Enable/disable NUMA.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">numa[n]</span>: <span class="monospaced">cpus=&lt;id[-id];...&gt; [,hostnodes=&lt;id[-id];...&gt;] [,memory=&lt;number&gt;] [,policy=&lt;preferred|bind|interleave&gt;]</span> 
</dt>
<dd>
<p>
NUMA topology.
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">cpus</span>=<span class="monospaced">&lt;id[-id];...&gt;</span> 
</dt>
<dd>
<p>
CPUs accessing this NUMA node.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">hostnodes</span>=<span class="monospaced">&lt;id[-id];...&gt;</span> 
</dt>
<dd>
<p>
Host NUMA nodes to use.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">memory</span>=<span class="monospaced">&lt;number&gt;</span> 
</dt>
<dd>
<p>
Amount of memory this NUMA node provides.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">policy</span>=<span class="monospaced">&lt;bind | interleave | preferred&gt;</span> 
</dt>
<dd>
<p>
NUMA allocation policy.
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">onboot</span>: <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Specifies whether a VM will be started during system bootup.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">ostype</span>: <span class="monospaced">&lt;l24 | l26 | other | solaris | w2k | w2k3 | w2k8 | win10 | win11 | win7 | win8 | wvista | wxp&gt;</span> 
</dt>
<dd>
<p>
Specify guest operating system. This is used to enable special
optimization/features for specific operating systems:
</p>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
other
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
unspecified OS
</p>
</td>
</tr>
<tr>
<td class="hdlist1">
wxp
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Microsoft Windows XP
</p>
</td>
</tr>
<tr>
<td class="hdlist1">
w2k
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Microsoft Windows 2000
</p>
</td>
</tr>
<tr>
<td class="hdlist1">
w2k3
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Microsoft Windows 2003
</p>
</td>
</tr>
<tr>
<td class="hdlist1">
w2k8
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Microsoft Windows 2008
</p>
</td>
</tr>
<tr>
<td class="hdlist1">
wvista
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Microsoft Windows Vista
</p>
</td>
</tr>
<tr>
<td class="hdlist1">
win7
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Microsoft Windows 7
</p>
</td>
</tr>
<tr>
<td class="hdlist1">
win8
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Microsoft Windows 8/2012/2012r2
</p>
</td>
</tr>
<tr>
<td class="hdlist1">
win10
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Microsoft Windows 10/2016/2019
</p>
</td>
</tr>
<tr>
<td class="hdlist1">
win11
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Microsoft Windows 11/2022/2025
</p>
</td>
</tr>
<tr>
<td class="hdlist1">
l24
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Linux 2.4 Kernel
</p>
</td>
</tr>
<tr>
<td class="hdlist1">
l26
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Linux 2.6 - 6.X Kernel
</p>
</td>
</tr>
<tr>
<td class="hdlist1">
solaris
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Solaris/OpenSolaris/OpenIndiania kernel
</p>
</td>
</tr>
</tbody></table></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">parallel[n]</span>: <span class="monospaced">/dev/parport\d+|/dev/usb/lp\d+</span> 
</dt>
<dd>
<p>
Map host parallel devices (n is 0 to 2).
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">This option allows direct access to host hardware. So it is no longer possible to migrate such
machines - use with special care.</td>
</tr></tbody></table>
</div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Caution" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKdUlEQVRoge1Ze1AV1x3+zt279wEi
DWCYGzRVktqa1MEmmtbWR22ncXxUrTDWV/5IqG2wUAUfmUwSkk6V+EAJCEQC6figwZBqZtRxqukf
tebRZBpFG1S0hiRErwiaKK977+6eX//YPXt37+UiIJlMZnpmdnb33PP4vt97z2VEhG9yc3zdAO60
/Z/A192+8QScX8Wifr+fWltbzffU1FT4fD72Vew15ASampqovr4eBw8eNPvGjRuHzMxMmj9//tCT
IKIhu958801yuVwEoNdr48aNNJT7EdHQEdi/fz/JshwTvLiKioqGlMSQLFJfX0+MMRvQsWNSqXLz
H2jez78fRWLTpk1DRuKOF6irqyOn02kDOP8XGdTz+VFSP91Hatu79NRvZ0SR2LJly5CQuKPJtbW1
JEmSDdivHp1AwctHSblYSsqFElLOF5PavIvWZU+LIrF169Y7JjHoibt3744CnzXrIQpeOUbKhVJS
mkpIPV9MyrnNpJwtIuVSNa15/MdRJLZt23ZHJBjRwIu5N954gxYvXgzOudm3aO5E1L5SBNZ1Hoxr
INLAyH6HJxVP/WkXtu9+37ZeSUkJVq9ePagQO+BMXFNTEwV+ybxHsPeVIjg6zwOkgSWNhuOuUUBC
KhxJY0AggDSgpwWbn1mO1csftq1ZUFCA0tLSQZXFA0pkVVVVlJOTA6vWsmY9hF07N0AS4OU48LgR
WLO+FPFxboxMS8G5hlPYmPMI4jwM1P0Jtj67DJyrKHvtNADdjPPz8+FwOCgvL29gmuivrVVUVESF
yqxZD1PoyjFSL+7QHbapmJTml+n9EzVRtl676ZekNKwl5VQ+KSfzKHR2E+UutodYxhiVl5cPyCf6
ZUJlZWWUm5trk/wTi6bgtVdf1M2Gq2DEwUgDNBWhUDBqjfSRwwGoIOEP3c3Y/uxirFw0zibMvLw8
VFZW9tucbkugoqKCVq1aZQO/YslUVJU+D9bVBIDroLgK4hygEDweybYGY8AD6d8yftfASL+j+2O8
9NwiPJk51kYiNzcXO3fu7BeJPgkUFxdTbm6ure/JZdNRub0QrOMcGFcBrgLgAHEAGqD2ID7OZZsz
Ji0RcR6HAV4DkaqPJxXouoSywkz8buF9NhIrV65EVVXVbUnEJHDixAlat26drS9n+U+xo/hZMMNh
yZA+uAbAIKOGMCJlmG3eA/clAYaJEamGBlRdC6QBXZewo3AhVswfbSORk5ODDz/8sE8SvRLw+/1U
XV0dBb5MgOe6LUNIEqpOwgCUlOCESw4HuAfG3KX7B2kAGaTJ0Jro77yI8ucX4DfzRtlIHD58GH6/
PyaJXgk0NjZi79695vuUiffrkr91Vt8cHAwc4FZJGiZEKqDcwsi0ZHP+d749POy8MS7iGtBxARWF
8/CDsQnm3BdeeAHWj6N+EYhspSVbgI5zYYkZGmAwzIEbwA0tUM9NfO+7aeb89HuGRQBWLcSNZxj3
zvOYMzk5NpjBEHi99mXD5lXdjqGDJh42h7CEOXhnK3448X5z/shUb1jSpIEM8yFuzOUcjAsNcnR0
KXdG4MEHH8TEiRPN9z0H3sfRE41m+GNcs9i0LjkGTTcrUoGem5g7/V5z/ohElw4O+jxGYQ3qJqmB
jN/+cbINtcfCJvPYY4+hr3qtVwI+n49lZmaa71fbbuH3hX/FycbPQKSCoIIMQEQcehjVo5Gw8/F3
3cSvH03H2HuHY1icQzcvYWKGIMiITMwQwjtnruPpqk9w/ZZq7p2RkYEJEyaAYrCIWY36/X7KyMhA
W1ub2XfP3Qmo3ZKFaQ+P0gmYIdSoOkU4lWSwxGRAdqHL/xm86k27w0blAw3/PncD6yo/xrsfdZj7
paSkoLGxESNGjAgDZsxWK8X0AZ/Px44fP460tLAzXrnWgUX5r+PYuxf1mG/Gf80sofWy2YuWbg/+
uONtPLHhPRw+0WIHL8hbwG95rcUGPiEhAYcOHUJSUhI456YZRWqiz+8BIqLm5mZMmjQJN27cMPsT
E9zYtWE25k4bYzh3mABIBRwOHPgPx+LH/6yDiXPi6pHZcEpk+I5qmtGpCzdQecCPPUevmevLsowj
R45gypQpcDgc5mVowKaJmBoQTEePHo2GhgaMGxcuum52BLHsqUOoP3oWgLB9Sz7QgpgxJoThw/SS
IutnaXBKZEpcgD998Uvs+3ubDbzT6URNTQ0mTZoEVVWhaRo459A0TeCy4etVA+JH8RvnHK2trZgz
Zw5Onz5tjnPJDrxSOAPLZt1nSVIqwDmIVJw6dx1nLt3A3J/cjZThkhE29cvf3o2KA59ja91lcz3G
GDZv3oylS5fC7XbD6XTC6XRCkiRIktS7Jm5HQLDXNA3Xr19HVlYWPvjgA4vEGLav+RHmTxsJX4rb
8AsO4kbOIG6GzTD4HvzlmB/PVH9q2zc/Px/Z2dnwer1wu92QZRkulwuSJEGWZZMEY8wkEGVCVvCC
AOcciqIgPj4e+/btw9SpU83xqkpYteU9HHm7Bf62rnC2RjikisTFSMPV9m4ceucannvVDn7p0qVY
uHAhgsEggsEgQqFQlAlZnVm0PjOxIKFpGogIqqrC5XKhuroa06dPt4wDcl78F+qPNeNKe6cRoTjs
ZYOKK+3dON7wBQrKm2H5pMbMmTOxZMkSBAIB9PT0oKenB4FAAIqiQFEUU4hRRyq3I2AlIZ4553A6
nSgvL8fs2bNtY9eVNaDub58YmhDZWpe+v70bJ5tu4clt/4WihqU4efJkrFixAkRkAg4Gg1AUBaqq
muDF/pGtX7WQNXeIZ0mSUFRUhAULFtjGPl3ZiJdev4QrbZ16zUQqrrZ342JLN5ZvaEJ3ICz68ePH
IycnB5qmmZIW4EOhUJTZWEKo+Rx1KsEYY9Zk4XA4IEmSKXnOuRkVJElCYWEhvF4v6urqzDVK9n2M
zu4Qnl5+L9q/7MHltgBWbr9kA5+eno7c3FzIsoyI5GrTeCzgosU8VmGMmSGLiEwSsixH2WNBQQEA
2EhUH/wc7330BaaOH479/2zHtS/CFabP58OaNWvg9Xp1EE4nZFmGLMtwu93wer1wuVy2ECpJko2I
iTNWJrZGI+HEwi6FisUVCAQQCASwZ88eVFZW9lk9JicnY/369fD5fPB4PCbo+Ph4k4TH44HL5YLX
64XX6zXH9EKE9aUBRkTEGLNJP1Kd4nI4HMjOzkZ8fDyKi4ttJ3eiJSYmYu3atRg1apQJzuv1wuPx
wOPxwOFwmEBFHnA6nXC5XGYSiywlbns2SnoDANN0hMNZnS0UCpkaOnPmDN566y20tLSY2ktLS8OM
GTOQkpICt9uNuLg4M2EJ8CJZSZJkZmFBwkrAWpH2+3BXEBEkRIhTVdVMOCL5qKqKy5cvQ1EUaJoG
VVWRnKx/Jlpt3eVy2QCKu9C68AFr9o0spwd0Oh1ZYggCIlNaiQktiXdhZgKQABcp3ci7LMtM7B0J
fsAErJoAYItG1mdr+hfvVlACtHDIXkploYXbHvQOikDEe6/ZmjFmAhf9ZvJxOk2Qol88W4kYz32S
GNQfHLG+TyMTkAVEr88xQfVnkBg7GAKixSIykDYQsL21/wFkW/B5QqT9lwAAAABJRU5ErkJggg==">
</td>
<td class="content">Experimental! User reported problems with this option.</td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">protection</span>: <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Sets the protection flag of the VM. This will disable the remove VM and remove disk operations.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">reboot</span>: <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Allow reboot. If set to <em>0</em> the VM exit on reboot.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">rng0</span>: <span class="monospaced">[source=]&lt;/dev/urandom|/dev/random|/dev/hwrng&gt; [,max_bytes=&lt;integer&gt;] [,period=&lt;integer&gt;]</span> 
</dt>
<dd>
<p>
Configure a VirtIO-based Random Number Generator.
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">max_bytes</span>=<span class="monospaced">&lt;integer&gt;</span> (<em>default =</em> <span class="monospaced">1024</span>)
</dt>
<dd>
<p>
Maximum bytes of entropy allowed to get injected into the guest every <em>period</em> milliseconds. Prefer a lower value when using <em>/dev/random</em> as source. Use <span class="monospaced">0</span> to disable limiting (potentially dangerous!).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">period</span>=<span class="monospaced">&lt;integer&gt;</span> (<em>default =</em> <span class="monospaced">1000</span>)
</dt>
<dd>
<p>
Every <em>period</em> milliseconds the entropy-injection quota is reset, allowing the guest to retrieve another <em>max_bytes</em> of entropy.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">source</span>=<span class="monospaced">&lt;/dev/hwrng | /dev/random | /dev/urandom&gt;</span> 
</dt>
<dd>
<p>
The file on the host to gather entropy from. In most cases <em>/dev/urandom</em> should be preferred over <em>/dev/random</em> to avoid entropy-starvation issues on the host. Using urandom does <strong>not</strong> decrease security in any meaningful way, as it’s still seeded from real entropy, and the bytes provided will most likely be mixed with real entropy on the guest as well. <em>/dev/hwrng</em> can be used to pass through a hardware RNG from the host.
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">sata[n]</span>: <span class="monospaced">[file=]&lt;volume&gt; [,aio=&lt;native|threads|io_uring&gt;] [,backup=&lt;1|0&gt;] [,bps=&lt;bps&gt;] [,bps_max_length=&lt;seconds&gt;] [,bps_rd=&lt;bps&gt;] [,bps_rd_max_length=&lt;seconds&gt;] [,bps_wr=&lt;bps&gt;] [,bps_wr_max_length=&lt;seconds&gt;] [,cache=&lt;enum&gt;] [,cyls=&lt;integer&gt;] [,detect_zeroes=&lt;1|0&gt;] [,discard=&lt;ignore|on&gt;] [,format=&lt;enum&gt;] [,heads=&lt;integer&gt;] [,iops=&lt;iops&gt;] [,iops_max=&lt;iops&gt;] [,iops_max_length=&lt;seconds&gt;] [,iops_rd=&lt;iops&gt;] [,iops_rd_max=&lt;iops&gt;] [,iops_rd_max_length=&lt;seconds&gt;] [,iops_wr=&lt;iops&gt;] [,iops_wr_max=&lt;iops&gt;] [,iops_wr_max_length=&lt;seconds&gt;] [,mbps=&lt;mbps&gt;] [,mbps_max=&lt;mbps&gt;] [,mbps_rd=&lt;mbps&gt;] [,mbps_rd_max=&lt;mbps&gt;] [,mbps_wr=&lt;mbps&gt;] [,mbps_wr_max=&lt;mbps&gt;] [,media=&lt;cdrom|disk&gt;] [,replicate=&lt;1|0&gt;] [,rerror=&lt;ignore|report|stop&gt;] [,secs=&lt;integer&gt;] [,serial=&lt;serial&gt;] [,shared=&lt;1|0&gt;] [,size=&lt;DiskSize&gt;] [,snapshot=&lt;1|0&gt;] [,ssd=&lt;1|0&gt;] [,trans=&lt;none|lba|auto&gt;] [,werror=&lt;enum&gt;] [,wwn=&lt;wwn&gt;]</span> 
</dt>
<dd>
<p>
Use volume as SATA hard disk or CD-ROM (n is 0 to 5).
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">aio</span>=<span class="monospaced">&lt;io_uring | native | threads&gt;</span> 
</dt>
<dd>
<p>
AIO type to use.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">backup</span>=<span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Whether the drive should be included when making backups.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">bps</span>=<span class="monospaced">&lt;bps&gt;</span> 
</dt>
<dd>
<p>
Maximum r/w speed in bytes per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">bps_max_length</span>=<span class="monospaced">&lt;seconds&gt;</span> 
</dt>
<dd>
<p>
Maximum length of I/O bursts in seconds.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">bps_rd</span>=<span class="monospaced">&lt;bps&gt;</span> 
</dt>
<dd>
<p>
Maximum read speed in bytes per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">bps_rd_max_length</span>=<span class="monospaced">&lt;seconds&gt;</span> 
</dt>
<dd>
<p>
Maximum length of read I/O bursts in seconds.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">bps_wr</span>=<span class="monospaced">&lt;bps&gt;</span> 
</dt>
<dd>
<p>
Maximum write speed in bytes per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">bps_wr_max_length</span>=<span class="monospaced">&lt;seconds&gt;</span> 
</dt>
<dd>
<p>
Maximum length of write I/O bursts in seconds.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">cache</span>=<span class="monospaced">&lt;directsync | none | unsafe | writeback | writethrough&gt;</span> 
</dt>
<dd>
<p>
The drive’s cache mode
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">cyls</span>=<span class="monospaced">&lt;integer&gt;</span> 
</dt>
<dd>
<p>
Force the drive’s physical geometry to have a specific cylinder count.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">detect_zeroes</span>=<span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Controls whether to detect and try to optimize writes of zeroes.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">discard</span>=<span class="monospaced">&lt;ignore | on&gt;</span> 
</dt>
<dd>
<p>
Controls whether to pass discard/trim requests to the underlying storage.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">file</span>=<span class="monospaced">&lt;volume&gt;</span> 
</dt>
<dd>
<p>
The drive’s backing volume.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">format</span>=<span class="monospaced">&lt;cloop | cow | qcow | qcow2 | qed | raw | vmdk&gt;</span> 
</dt>
<dd>
<p>
The drive’s backing file’s data format.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">heads</span>=<span class="monospaced">&lt;integer&gt;</span> 
</dt>
<dd>
<p>
Force the drive’s physical geometry to have a specific head count.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">iops</span>=<span class="monospaced">&lt;iops&gt;</span> 
</dt>
<dd>
<p>
Maximum r/w I/O in operations per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">iops_max</span>=<span class="monospaced">&lt;iops&gt;</span> 
</dt>
<dd>
<p>
Maximum unthrottled r/w I/O pool in operations per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">iops_max_length</span>=<span class="monospaced">&lt;seconds&gt;</span> 
</dt>
<dd>
<p>
Maximum length of I/O bursts in seconds.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">iops_rd</span>=<span class="monospaced">&lt;iops&gt;</span> 
</dt>
<dd>
<p>
Maximum read I/O in operations per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">iops_rd_max</span>=<span class="monospaced">&lt;iops&gt;</span> 
</dt>
<dd>
<p>
Maximum unthrottled read I/O pool in operations per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">iops_rd_max_length</span>=<span class="monospaced">&lt;seconds&gt;</span> 
</dt>
<dd>
<p>
Maximum length of read I/O bursts in seconds.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">iops_wr</span>=<span class="monospaced">&lt;iops&gt;</span> 
</dt>
<dd>
<p>
Maximum write I/O in operations per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">iops_wr_max</span>=<span class="monospaced">&lt;iops&gt;</span> 
</dt>
<dd>
<p>
Maximum unthrottled write I/O pool in operations per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">iops_wr_max_length</span>=<span class="monospaced">&lt;seconds&gt;</span> 
</dt>
<dd>
<p>
Maximum length of write I/O bursts in seconds.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mbps</span>=<span class="monospaced">&lt;mbps&gt;</span> 
</dt>
<dd>
<p>
Maximum r/w speed in megabytes per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mbps_max</span>=<span class="monospaced">&lt;mbps&gt;</span> 
</dt>
<dd>
<p>
Maximum unthrottled r/w pool in megabytes per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mbps_rd</span>=<span class="monospaced">&lt;mbps&gt;</span> 
</dt>
<dd>
<p>
Maximum read speed in megabytes per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mbps_rd_max</span>=<span class="monospaced">&lt;mbps&gt;</span> 
</dt>
<dd>
<p>
Maximum unthrottled read pool in megabytes per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mbps_wr</span>=<span class="monospaced">&lt;mbps&gt;</span> 
</dt>
<dd>
<p>
Maximum write speed in megabytes per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mbps_wr_max</span>=<span class="monospaced">&lt;mbps&gt;</span> 
</dt>
<dd>
<p>
Maximum unthrottled write pool in megabytes per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">media</span>=<span class="monospaced">&lt;cdrom | disk&gt;</span> (<em>default =</em> <span class="monospaced">disk</span>)
</dt>
<dd>
<p>
The drive’s media type.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">replicate</span>=<span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Whether the drive should considered for replication jobs.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">rerror</span>=<span class="monospaced">&lt;ignore | report | stop&gt;</span> 
</dt>
<dd>
<p>
Read error action.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">secs</span>=<span class="monospaced">&lt;integer&gt;</span> 
</dt>
<dd>
<p>
Force the drive’s physical geometry to have a specific sector count.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">serial</span>=<span class="monospaced">&lt;serial&gt;</span> 
</dt>
<dd>
<p>
The drive’s reported serial number, url-encoded, up to 20 bytes long.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">shared</span>=<span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Mark this locally-managed volume as available on all nodes.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">This option does not share the volume automatically, it assumes it is shared already!</td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">size</span>=<span class="monospaced">&lt;DiskSize&gt;</span> 
</dt>
<dd>
<p>
Disk size. This is purely informational and has no effect.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">snapshot</span>=<span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Controls qemu’s snapshot mode feature. If activated, changes made to the disk are temporary and will be discarded when the VM is shutdown.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">ssd</span>=<span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Whether to expose this drive as an SSD, rather than a rotational hard disk.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">trans</span>=<span class="monospaced">&lt;auto | lba | none&gt;</span> 
</dt>
<dd>
<p>
Force disk geometry bios translation mode.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">werror</span>=<span class="monospaced">&lt;enospc | ignore | report | stop&gt;</span> 
</dt>
<dd>
<p>
Write error action.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">wwn</span>=<span class="monospaced">&lt;wwn&gt;</span> 
</dt>
<dd>
<p>
The drive’s worldwide name, encoded as 16 bytes hex string, prefixed by <em>0x</em>.
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">scsi[n]</span>: <span class="monospaced">[file=]&lt;volume&gt; [,aio=&lt;native|threads|io_uring&gt;] [,backup=&lt;1|0&gt;] [,bps=&lt;bps&gt;] [,bps_max_length=&lt;seconds&gt;] [,bps_rd=&lt;bps&gt;] [,bps_rd_max_length=&lt;seconds&gt;] [,bps_wr=&lt;bps&gt;] [,bps_wr_max_length=&lt;seconds&gt;] [,cache=&lt;enum&gt;] [,cyls=&lt;integer&gt;] [,detect_zeroes=&lt;1|0&gt;] [,discard=&lt;ignore|on&gt;] [,format=&lt;enum&gt;] [,heads=&lt;integer&gt;] [,iops=&lt;iops&gt;] [,iops_max=&lt;iops&gt;] [,iops_max_length=&lt;seconds&gt;] [,iops_rd=&lt;iops&gt;] [,iops_rd_max=&lt;iops&gt;] [,iops_rd_max_length=&lt;seconds&gt;] [,iops_wr=&lt;iops&gt;] [,iops_wr_max=&lt;iops&gt;] [,iops_wr_max_length=&lt;seconds&gt;] [,iothread=&lt;1|0&gt;] [,mbps=&lt;mbps&gt;] [,mbps_max=&lt;mbps&gt;] [,mbps_rd=&lt;mbps&gt;] [,mbps_rd_max=&lt;mbps&gt;] [,mbps_wr=&lt;mbps&gt;] [,mbps_wr_max=&lt;mbps&gt;] [,media=&lt;cdrom|disk&gt;] [,product=&lt;product&gt;] [,queues=&lt;integer&gt;] [,replicate=&lt;1|0&gt;] [,rerror=&lt;ignore|report|stop&gt;] [,ro=&lt;1|0&gt;] [,scsiblock=&lt;1|0&gt;] [,secs=&lt;integer&gt;] [,serial=&lt;serial&gt;] [,shared=&lt;1|0&gt;] [,size=&lt;DiskSize&gt;] [,snapshot=&lt;1|0&gt;] [,ssd=&lt;1|0&gt;] [,trans=&lt;none|lba|auto&gt;] [,vendor=&lt;vendor&gt;] [,werror=&lt;enum&gt;] [,wwn=&lt;wwn&gt;]</span> 
</dt>
<dd>
<p>
Use volume as SCSI hard disk or CD-ROM (n is 0 to 30).
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">aio</span>=<span class="monospaced">&lt;io_uring | native | threads&gt;</span> 
</dt>
<dd>
<p>
AIO type to use.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">backup</span>=<span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Whether the drive should be included when making backups.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">bps</span>=<span class="monospaced">&lt;bps&gt;</span> 
</dt>
<dd>
<p>
Maximum r/w speed in bytes per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">bps_max_length</span>=<span class="monospaced">&lt;seconds&gt;</span> 
</dt>
<dd>
<p>
Maximum length of I/O bursts in seconds.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">bps_rd</span>=<span class="monospaced">&lt;bps&gt;</span> 
</dt>
<dd>
<p>
Maximum read speed in bytes per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">bps_rd_max_length</span>=<span class="monospaced">&lt;seconds&gt;</span> 
</dt>
<dd>
<p>
Maximum length of read I/O bursts in seconds.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">bps_wr</span>=<span class="monospaced">&lt;bps&gt;</span> 
</dt>
<dd>
<p>
Maximum write speed in bytes per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">bps_wr_max_length</span>=<span class="monospaced">&lt;seconds&gt;</span> 
</dt>
<dd>
<p>
Maximum length of write I/O bursts in seconds.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">cache</span>=<span class="monospaced">&lt;directsync | none | unsafe | writeback | writethrough&gt;</span> 
</dt>
<dd>
<p>
The drive’s cache mode
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">cyls</span>=<span class="monospaced">&lt;integer&gt;</span> 
</dt>
<dd>
<p>
Force the drive’s physical geometry to have a specific cylinder count.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">detect_zeroes</span>=<span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Controls whether to detect and try to optimize writes of zeroes.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">discard</span>=<span class="monospaced">&lt;ignore | on&gt;</span> 
</dt>
<dd>
<p>
Controls whether to pass discard/trim requests to the underlying storage.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">file</span>=<span class="monospaced">&lt;volume&gt;</span> 
</dt>
<dd>
<p>
The drive’s backing volume.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">format</span>=<span class="monospaced">&lt;cloop | cow | qcow | qcow2 | qed | raw | vmdk&gt;</span> 
</dt>
<dd>
<p>
The drive’s backing file’s data format.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">heads</span>=<span class="monospaced">&lt;integer&gt;</span> 
</dt>
<dd>
<p>
Force the drive’s physical geometry to have a specific head count.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">iops</span>=<span class="monospaced">&lt;iops&gt;</span> 
</dt>
<dd>
<p>
Maximum r/w I/O in operations per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">iops_max</span>=<span class="monospaced">&lt;iops&gt;</span> 
</dt>
<dd>
<p>
Maximum unthrottled r/w I/O pool in operations per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">iops_max_length</span>=<span class="monospaced">&lt;seconds&gt;</span> 
</dt>
<dd>
<p>
Maximum length of I/O bursts in seconds.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">iops_rd</span>=<span class="monospaced">&lt;iops&gt;</span> 
</dt>
<dd>
<p>
Maximum read I/O in operations per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">iops_rd_max</span>=<span class="monospaced">&lt;iops&gt;</span> 
</dt>
<dd>
<p>
Maximum unthrottled read I/O pool in operations per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">iops_rd_max_length</span>=<span class="monospaced">&lt;seconds&gt;</span> 
</dt>
<dd>
<p>
Maximum length of read I/O bursts in seconds.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">iops_wr</span>=<span class="monospaced">&lt;iops&gt;</span> 
</dt>
<dd>
<p>
Maximum write I/O in operations per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">iops_wr_max</span>=<span class="monospaced">&lt;iops&gt;</span> 
</dt>
<dd>
<p>
Maximum unthrottled write I/O pool in operations per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">iops_wr_max_length</span>=<span class="monospaced">&lt;seconds&gt;</span> 
</dt>
<dd>
<p>
Maximum length of write I/O bursts in seconds.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">iothread</span>=<span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Whether to use iothreads for this drive
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mbps</span>=<span class="monospaced">&lt;mbps&gt;</span> 
</dt>
<dd>
<p>
Maximum r/w speed in megabytes per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mbps_max</span>=<span class="monospaced">&lt;mbps&gt;</span> 
</dt>
<dd>
<p>
Maximum unthrottled r/w pool in megabytes per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mbps_rd</span>=<span class="monospaced">&lt;mbps&gt;</span> 
</dt>
<dd>
<p>
Maximum read speed in megabytes per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mbps_rd_max</span>=<span class="monospaced">&lt;mbps&gt;</span> 
</dt>
<dd>
<p>
Maximum unthrottled read pool in megabytes per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mbps_wr</span>=<span class="monospaced">&lt;mbps&gt;</span> 
</dt>
<dd>
<p>
Maximum write speed in megabytes per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mbps_wr_max</span>=<span class="monospaced">&lt;mbps&gt;</span> 
</dt>
<dd>
<p>
Maximum unthrottled write pool in megabytes per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">media</span>=<span class="monospaced">&lt;cdrom | disk&gt;</span> (<em>default =</em> <span class="monospaced">disk</span>)
</dt>
<dd>
<p>
The drive’s media type.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">product</span>=<span class="monospaced">&lt;product&gt;</span> 
</dt>
<dd>
<p>
The drive’s product name, up to 16 bytes long.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">queues</span>=<span class="monospaced">&lt;integer&gt; (2 - N)</span> 
</dt>
<dd>
<p>
Number of queues.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">replicate</span>=<span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Whether the drive should considered for replication jobs.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">rerror</span>=<span class="monospaced">&lt;ignore | report | stop&gt;</span> 
</dt>
<dd>
<p>
Read error action.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">ro</span>=<span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Whether the drive is read-only.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">scsiblock</span>=<span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
whether to use scsi-block for full passthrough of host block device
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">can lead to I/O errors in combination with low memory or high memory fragmentation on host</td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">secs</span>=<span class="monospaced">&lt;integer&gt;</span> 
</dt>
<dd>
<p>
Force the drive’s physical geometry to have a specific sector count.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">serial</span>=<span class="monospaced">&lt;serial&gt;</span> 
</dt>
<dd>
<p>
The drive’s reported serial number, url-encoded, up to 20 bytes long.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">shared</span>=<span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Mark this locally-managed volume as available on all nodes.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">This option does not share the volume automatically, it assumes it is shared already!</td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">size</span>=<span class="monospaced">&lt;DiskSize&gt;</span> 
</dt>
<dd>
<p>
Disk size. This is purely informational and has no effect.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">snapshot</span>=<span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Controls qemu’s snapshot mode feature. If activated, changes made to the disk are temporary and will be discarded when the VM is shutdown.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">ssd</span>=<span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Whether to expose this drive as an SSD, rather than a rotational hard disk.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">trans</span>=<span class="monospaced">&lt;auto | lba | none&gt;</span> 
</dt>
<dd>
<p>
Force disk geometry bios translation mode.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">vendor</span>=<span class="monospaced">&lt;vendor&gt;</span> 
</dt>
<dd>
<p>
The drive’s vendor name, up to 8 bytes long.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">werror</span>=<span class="monospaced">&lt;enospc | ignore | report | stop&gt;</span> 
</dt>
<dd>
<p>
Write error action.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">wwn</span>=<span class="monospaced">&lt;wwn&gt;</span> 
</dt>
<dd>
<p>
The drive’s worldwide name, encoded as 16 bytes hex string, prefixed by <em>0x</em>.
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">scsihw</span>: <span class="monospaced">&lt;lsi | lsi53c810 | megasas | pvscsi | virtio-scsi-pci | virtio-scsi-single&gt;</span> (<em>default =</em> <span class="monospaced">lsi</span>)
</dt>
<dd>
<p>
SCSI controller model
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">searchdomain</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
cloud-init: Sets DNS search domains for a container. Create will automatically use the setting from the host if neither searchdomain nor nameserver are set.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">serial[n]</span>: <span class="monospaced">(/dev/.+|socket)</span> 
</dt>
<dd>
<p>
Create a serial device inside the VM (n is 0 to 3), and pass through a
host serial device (i.e. /dev/ttyS0), or create a unix socket on the
host side (use <em>qm terminal</em> to open a terminal connection).
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">If you pass through a host serial device, it is no longer possible to migrate such machines -
use with special care.</td>
</tr></tbody></table>
</div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Caution" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKdUlEQVRoge1Ze1AV1x3+zt279wEi
DWCYGzRVktqa1MEmmtbWR22ncXxUrTDWV/5IqG2wUAUfmUwSkk6V+EAJCEQC6figwZBqZtRxqukf
tebRZBpFG1S0hiRErwiaKK977+6eX//YPXt37+UiIJlMZnpmdnb33PP4vt97z2VEhG9yc3zdAO60
/Z/A192+8QScX8Wifr+fWltbzffU1FT4fD72Vew15ASampqovr4eBw8eNPvGjRuHzMxMmj9//tCT
IKIhu958801yuVwEoNdr48aNNJT7EdHQEdi/fz/JshwTvLiKioqGlMSQLFJfX0+MMRvQsWNSqXLz
H2jez78fRWLTpk1DRuKOF6irqyOn02kDOP8XGdTz+VFSP91Hatu79NRvZ0SR2LJly5CQuKPJtbW1
JEmSDdivHp1AwctHSblYSsqFElLOF5PavIvWZU+LIrF169Y7JjHoibt3744CnzXrIQpeOUbKhVJS
mkpIPV9MyrnNpJwtIuVSNa15/MdRJLZt23ZHJBjRwIu5N954gxYvXgzOudm3aO5E1L5SBNZ1Hoxr
INLAyH6HJxVP/WkXtu9+37ZeSUkJVq9ePagQO+BMXFNTEwV+ybxHsPeVIjg6zwOkgSWNhuOuUUBC
KhxJY0AggDSgpwWbn1mO1csftq1ZUFCA0tLSQZXFA0pkVVVVlJOTA6vWsmY9hF07N0AS4OU48LgR
WLO+FPFxboxMS8G5hlPYmPMI4jwM1P0Jtj67DJyrKHvtNADdjPPz8+FwOCgvL29gmuivrVVUVESF
yqxZD1PoyjFSL+7QHbapmJTml+n9EzVRtl676ZekNKwl5VQ+KSfzKHR2E+UutodYxhiVl5cPyCf6
ZUJlZWWUm5trk/wTi6bgtVdf1M2Gq2DEwUgDNBWhUDBqjfSRwwGoIOEP3c3Y/uxirFw0zibMvLw8
VFZW9tucbkugoqKCVq1aZQO/YslUVJU+D9bVBIDroLgK4hygEDweybYGY8AD6d8yftfASL+j+2O8
9NwiPJk51kYiNzcXO3fu7BeJPgkUFxdTbm6ure/JZdNRub0QrOMcGFcBrgLgAHEAGqD2ID7OZZsz
Ji0RcR6HAV4DkaqPJxXouoSywkz8buF9NhIrV65EVVXVbUnEJHDixAlat26drS9n+U+xo/hZMMNh
yZA+uAbAIKOGMCJlmG3eA/clAYaJEamGBlRdC6QBXZewo3AhVswfbSORk5ODDz/8sE8SvRLw+/1U
XV0dBb5MgOe6LUNIEqpOwgCUlOCESw4HuAfG3KX7B2kAGaTJ0Jro77yI8ucX4DfzRtlIHD58GH6/
PyaJXgk0NjZi79695vuUiffrkr91Vt8cHAwc4FZJGiZEKqDcwsi0ZHP+d749POy8MS7iGtBxARWF
8/CDsQnm3BdeeAHWj6N+EYhspSVbgI5zYYkZGmAwzIEbwA0tUM9NfO+7aeb89HuGRQBWLcSNZxj3
zvOYMzk5NpjBEHi99mXD5lXdjqGDJh42h7CEOXhnK3448X5z/shUb1jSpIEM8yFuzOUcjAsNcnR0
KXdG4MEHH8TEiRPN9z0H3sfRE41m+GNcs9i0LjkGTTcrUoGem5g7/V5z/ohElw4O+jxGYQ3qJqmB
jN/+cbINtcfCJvPYY4+hr3qtVwI+n49lZmaa71fbbuH3hX/FycbPQKSCoIIMQEQcehjVo5Gw8/F3
3cSvH03H2HuHY1icQzcvYWKGIMiITMwQwjtnruPpqk9w/ZZq7p2RkYEJEyaAYrCIWY36/X7KyMhA
W1ub2XfP3Qmo3ZKFaQ+P0gmYIdSoOkU4lWSwxGRAdqHL/xm86k27w0blAw3/PncD6yo/xrsfdZj7
paSkoLGxESNGjAgDZsxWK8X0AZ/Px44fP460tLAzXrnWgUX5r+PYuxf1mG/Gf80sofWy2YuWbg/+
uONtPLHhPRw+0WIHL8hbwG95rcUGPiEhAYcOHUJSUhI456YZRWqiz+8BIqLm5mZMmjQJN27cMPsT
E9zYtWE25k4bYzh3mABIBRwOHPgPx+LH/6yDiXPi6pHZcEpk+I5qmtGpCzdQecCPPUevmevLsowj
R45gypQpcDgc5mVowKaJmBoQTEePHo2GhgaMGxcuum52BLHsqUOoP3oWgLB9Sz7QgpgxJoThw/SS
IutnaXBKZEpcgD998Uvs+3ubDbzT6URNTQ0mTZoEVVWhaRo459A0TeCy4etVA+JH8RvnHK2trZgz
Zw5Onz5tjnPJDrxSOAPLZt1nSVIqwDmIVJw6dx1nLt3A3J/cjZThkhE29cvf3o2KA59ja91lcz3G
GDZv3oylS5fC7XbD6XTC6XRCkiRIktS7Jm5HQLDXNA3Xr19HVlYWPvjgA4vEGLav+RHmTxsJX4rb
8AsO4kbOIG6GzTD4HvzlmB/PVH9q2zc/Px/Z2dnwer1wu92QZRkulwuSJEGWZZMEY8wkEGVCVvCC
AOcciqIgPj4e+/btw9SpU83xqkpYteU9HHm7Bf62rnC2RjikisTFSMPV9m4ceucannvVDn7p0qVY
uHAhgsEggsEgQqFQlAlZnVm0PjOxIKFpGogIqqrC5XKhuroa06dPt4wDcl78F+qPNeNKe6cRoTjs
ZYOKK+3dON7wBQrKm2H5pMbMmTOxZMkSBAIB9PT0oKenB4FAAIqiQFEUU4hRRyq3I2AlIZ4553A6
nSgvL8fs2bNtY9eVNaDub58YmhDZWpe+v70bJ5tu4clt/4WihqU4efJkrFixAkRkAg4Gg1AUBaqq
muDF/pGtX7WQNXeIZ0mSUFRUhAULFtjGPl3ZiJdev4QrbZ16zUQqrrZ342JLN5ZvaEJ3ICz68ePH
IycnB5qmmZIW4EOhUJTZWEKo+Rx1KsEYY9Zk4XA4IEmSKXnOuRkVJElCYWEhvF4v6urqzDVK9n2M
zu4Qnl5+L9q/7MHltgBWbr9kA5+eno7c3FzIsoyI5GrTeCzgosU8VmGMmSGLiEwSsixH2WNBQQEA
2EhUH/wc7330BaaOH479/2zHtS/CFabP58OaNWvg9Xp1EE4nZFmGLMtwu93wer1wuVy2ECpJko2I
iTNWJrZGI+HEwi6FisUVCAQQCASwZ88eVFZW9lk9JicnY/369fD5fPB4PCbo+Ph4k4TH44HL5YLX
64XX6zXH9EKE9aUBRkTEGLNJP1Kd4nI4HMjOzkZ8fDyKi4ttJ3eiJSYmYu3atRg1apQJzuv1wuPx
wOPxwOFwmEBFHnA6nXC5XGYSiywlbns2SnoDANN0hMNZnS0UCpkaOnPmDN566y20tLSY2ktLS8OM
GTOQkpICt9uNuLg4M2EJ8CJZSZJkZmFBwkrAWpH2+3BXEBEkRIhTVdVMOCL5qKqKy5cvQ1EUaJoG
VVWRnKx/Jlpt3eVy2QCKu9C68AFr9o0spwd0Oh1ZYggCIlNaiQktiXdhZgKQABcp3ci7LMtM7B0J
fsAErJoAYItG1mdr+hfvVlACtHDIXkploYXbHvQOikDEe6/ZmjFmAhf9ZvJxOk2Qol88W4kYz32S
GNQfHLG+TyMTkAVEr88xQfVnkBg7GAKixSIykDYQsL21/wFkW/B5QqT9lwAAAABJRU5ErkJggg==">
</td>
<td class="content">Experimental! User reported problems with this option.</td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">shares</span>: <span class="monospaced">&lt;integer&gt; (0 - 50000)</span> (<em>default =</em> <span class="monospaced">1000</span>)
</dt>
<dd>
<p>
Amount of memory shares for auto-ballooning. The larger the number is, the more memory this VM gets. Number is relative to weights of all other running VMs. Using zero disables auto-ballooning. Auto-ballooning is done by pvestatd.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">smbios1</span>: <span class="monospaced">[base64=&lt;1|0&gt;] [,family=&lt;Base64 encoded string&gt;] [,manufacturer=&lt;Base64 encoded string&gt;] [,product=&lt;Base64 encoded string&gt;] [,serial=&lt;Base64 encoded string&gt;] [,sku=&lt;Base64 encoded string&gt;] [,uuid=&lt;UUID&gt;] [,version=&lt;Base64 encoded string&gt;]</span> 
</dt>
<dd>
<p>
Specify SMBIOS type 1 fields.
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">base64</span>=<span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Flag to indicate that the SMBIOS values are base64 encoded
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">family</span>=<span class="monospaced">&lt;Base64 encoded string&gt;</span> 
</dt>
<dd>
<p>
Set SMBIOS1 family string.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">manufacturer</span>=<span class="monospaced">&lt;Base64 encoded string&gt;</span> 
</dt>
<dd>
<p>
Set SMBIOS1 manufacturer.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">product</span>=<span class="monospaced">&lt;Base64 encoded string&gt;</span> 
</dt>
<dd>
<p>
Set SMBIOS1 product ID.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">serial</span>=<span class="monospaced">&lt;Base64 encoded string&gt;</span> 
</dt>
<dd>
<p>
Set SMBIOS1 serial number.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">sku</span>=<span class="monospaced">&lt;Base64 encoded string&gt;</span> 
</dt>
<dd>
<p>
Set SMBIOS1 SKU string.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">uuid</span>=<span class="monospaced">&lt;UUID&gt;</span> 
</dt>
<dd>
<p>
Set SMBIOS1 UUID.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">version</span>=<span class="monospaced">&lt;Base64 encoded string&gt;</span> 
</dt>
<dd>
<p>
Set SMBIOS1 version.
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">smp</span>: <span class="monospaced">&lt;integer&gt; (1 - N)</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
The number of CPUs. Please use option -sockets instead.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">sockets</span>: <span class="monospaced">&lt;integer&gt; (1 - N)</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
The number of CPU sockets.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">spice_enhancements</span>: <span class="monospaced">[foldersharing=&lt;1|0&gt;] [,videostreaming=&lt;off|all|filter&gt;]</span> 
</dt>
<dd>
<p>
Configure additional enhancements for SPICE.
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">foldersharing</span>=<span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Enable folder sharing via SPICE. Needs Spice-WebDAV daemon installed in the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">videostreaming</span>=<span class="monospaced">&lt;all | filter | off&gt;</span> (<em>default =</em> <span class="monospaced">off</span>)
</dt>
<dd>
<p>
Enable video streaming. Uses compression for detected video streams.
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">sshkeys</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
cloud-init: Setup public SSH keys (one key per line, OpenSSH format).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">startdate</span>: <span class="monospaced">(now | YYYY-MM-DD | YYYY-MM-DDTHH:MM:SS)</span> (<em>default =</em> <span class="monospaced">now</span>)
</dt>
<dd>
<p>
Set the initial date of the real time clock. Valid format for date are:'now' or <em>2006-06-17T16:01:21</em> or <em>2006-06-17</em>.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">startup</span>: `[[order=]\d+] [,up=\d+] [,down=\d+] ` 
</dt>
<dd>
<p>
Startup and shutdown behavior. Order is a non-negative number defining the general startup order. Shutdown in done with reverse ordering. Additionally you can set the <em>up</em> or <em>down</em> delay in seconds, which specifies a delay to wait before the next VM is started or stopped.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">tablet</span>: <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Enable/disable the USB tablet device. This device is usually needed to allow absolute mouse positioning with VNC. Else the mouse runs out of sync with normal VNC clients. If you’re running lots of console-only guests on one host, you may consider disabling this to save some context switches. This is turned off by default if you use spice (<span class="monospaced">qm set &lt;vmid&gt; --vga qxl</span>).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">tags</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Tags of the VM. This is only meta information.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">tdf</span>: <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Enable/disable time drift fix.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">template</span>: <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Enable/disable Template.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">tpmstate0</span>: <span class="monospaced">[file=]&lt;volume&gt; [,size=&lt;DiskSize&gt;] [,version=&lt;v1.2|v2.0&gt;]</span> 
</dt>
<dd>
<p>
Configure a Disk for storing TPM state. The format is fixed to <em>raw</em>.
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">file</span>=<span class="monospaced">&lt;volume&gt;</span> 
</dt>
<dd>
<p>
The drive’s backing volume.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">size</span>=<span class="monospaced">&lt;DiskSize&gt;</span> 
</dt>
<dd>
<p>
Disk size. This is purely informational and has no effect.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">version</span>=<span class="monospaced">&lt;v1.2 | v2.0&gt;</span> (<em>default =</em> <span class="monospaced">v1.2</span>)
</dt>
<dd>
<p>
The TPM interface version. v2.0 is newer and should be preferred. Note that this cannot be changed later on.
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">unused[n]</span>: <span class="monospaced">[file=]&lt;volume&gt;</span> 
</dt>
<dd>
<p>
Reference to unused volumes. This is used internally, and should not be modified manually.
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">file</span>=<span class="monospaced">&lt;volume&gt;</span> 
</dt>
<dd>
<p>
The drive’s backing volume.
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">usb[n]</span>: <span class="monospaced">[[host=]&lt;HOSTUSBDEVICE|spice&gt;] [,mapping=&lt;mapping-id&gt;] [,usb3=&lt;1|0&gt;]</span> 
</dt>
<dd>
<p>
Configure an USB device (n is 0 to 4, for machine version &gt;= 7.1 and ostype l26 or windows &gt; 7, n can be up to 14).
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">host</span>=<span class="monospaced">&lt;HOSTUSBDEVICE|spice&gt;</span> 
</dt>
<dd>
<p>
The Host USB device or port or the value <em>spice</em>. HOSTUSBDEVICE syntax is:
</p>
<div class="literalblock">
<div class="content monospaced">
<pre>'bus-port(.port)*' (decimal numbers) or
'vendor_id:product_id' (hexadeciaml numbers) or
'spice'</pre>
</div></div>
<div class="paragraph">
<p>You can use the <em>lsusb -t</em> command to list existing usb devices.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">This option allows direct access to host hardware. So it is no longer possible to migrate such
machines - use with special care.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>The value <em>spice</em> can be used to add a usb redirection devices for spice.</p></div>
<div class="paragraph">
<p>Either this or the <em>mapping</em> key must be set.</p></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">mapping</span>=<span class="monospaced">&lt;mapping-id&gt;</span> 
</dt>
<dd>
<p>
The ID of a cluster wide mapping. Either this or the default-key <em>host</em> must be set.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">usb3</span>=<span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Specifies whether if given host option is a USB3 device or port. For modern guests (machine version &gt;= 7.1 and ostype l26 and windows &gt; 7), this flag is irrelevant (all devices are plugged into a xhci controller).
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">vcpus</span>: <span class="monospaced">&lt;integer&gt; (1 - N)</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Number of hotplugged vcpus.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">vga</span>: <span class="monospaced">[[type=]&lt;enum&gt;] [,clipboard=&lt;vnc&gt;] [,memory=&lt;integer&gt;]</span> 
</dt>
<dd>
<p>
Configure the VGA Hardware. If you want to use high resolution modes (&gt;= 1280x1024x16) you may need to increase the vga memory option. Since QEMU 2.9 the default VGA display type is <em>std</em> for all OS types besides some Windows versions (XP and older) which use <em>cirrus</em>. The <em>qxl</em> option enables the SPICE display server. For win* OS you can select how many independent displays you want, Linux guests can add displays them self.
You can also run without any graphic card, using a serial device as terminal.
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">clipboard</span>=<span class="monospaced">&lt;vnc&gt;</span> 
</dt>
<dd>
<p>
Enable a specific clipboard. If not set, depending on the display type the SPICE one will be added. Migration with VNC clipboard is not yet supported!
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">memory</span>=<span class="monospaced">&lt;integer&gt; (4 - 512)</span> 
</dt>
<dd>
<p>
Sets the VGA memory (in MiB). Has no effect with serial display.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">type</span>=<span class="monospaced">&lt;cirrus | none | qxl | qxl2 | qxl3 | qxl4 | serial0 | serial1 | serial2 | serial3 | std | virtio | virtio-gl | vmware&gt;</span> (<em>default =</em> <span class="monospaced">std</span>)
</dt>
<dd>
<p>
Select the VGA type. Using type <em>cirrus</em> is not recommended.
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">virtio[n]</span>: <span class="monospaced">[file=]&lt;volume&gt; [,aio=&lt;native|threads|io_uring&gt;] [,backup=&lt;1|0&gt;] [,bps=&lt;bps&gt;] [,bps_max_length=&lt;seconds&gt;] [,bps_rd=&lt;bps&gt;] [,bps_rd_max_length=&lt;seconds&gt;] [,bps_wr=&lt;bps&gt;] [,bps_wr_max_length=&lt;seconds&gt;] [,cache=&lt;enum&gt;] [,cyls=&lt;integer&gt;] [,detect_zeroes=&lt;1|0&gt;] [,discard=&lt;ignore|on&gt;] [,format=&lt;enum&gt;] [,heads=&lt;integer&gt;] [,iops=&lt;iops&gt;] [,iops_max=&lt;iops&gt;] [,iops_max_length=&lt;seconds&gt;] [,iops_rd=&lt;iops&gt;] [,iops_rd_max=&lt;iops&gt;] [,iops_rd_max_length=&lt;seconds&gt;] [,iops_wr=&lt;iops&gt;] [,iops_wr_max=&lt;iops&gt;] [,iops_wr_max_length=&lt;seconds&gt;] [,iothread=&lt;1|0&gt;] [,mbps=&lt;mbps&gt;] [,mbps_max=&lt;mbps&gt;] [,mbps_rd=&lt;mbps&gt;] [,mbps_rd_max=&lt;mbps&gt;] [,mbps_wr=&lt;mbps&gt;] [,mbps_wr_max=&lt;mbps&gt;] [,media=&lt;cdrom|disk&gt;] [,replicate=&lt;1|0&gt;] [,rerror=&lt;ignore|report|stop&gt;] [,ro=&lt;1|0&gt;] [,secs=&lt;integer&gt;] [,serial=&lt;serial&gt;] [,shared=&lt;1|0&gt;] [,size=&lt;DiskSize&gt;] [,snapshot=&lt;1|0&gt;] [,trans=&lt;none|lba|auto&gt;] [,werror=&lt;enum&gt;]</span> 
</dt>
<dd>
<p>
Use volume as VIRTIO hard disk (n is 0 to 15).
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">aio</span>=<span class="monospaced">&lt;io_uring | native | threads&gt;</span> 
</dt>
<dd>
<p>
AIO type to use.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">backup</span>=<span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Whether the drive should be included when making backups.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">bps</span>=<span class="monospaced">&lt;bps&gt;</span> 
</dt>
<dd>
<p>
Maximum r/w speed in bytes per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">bps_max_length</span>=<span class="monospaced">&lt;seconds&gt;</span> 
</dt>
<dd>
<p>
Maximum length of I/O bursts in seconds.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">bps_rd</span>=<span class="monospaced">&lt;bps&gt;</span> 
</dt>
<dd>
<p>
Maximum read speed in bytes per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">bps_rd_max_length</span>=<span class="monospaced">&lt;seconds&gt;</span> 
</dt>
<dd>
<p>
Maximum length of read I/O bursts in seconds.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">bps_wr</span>=<span class="monospaced">&lt;bps&gt;</span> 
</dt>
<dd>
<p>
Maximum write speed in bytes per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">bps_wr_max_length</span>=<span class="monospaced">&lt;seconds&gt;</span> 
</dt>
<dd>
<p>
Maximum length of write I/O bursts in seconds.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">cache</span>=<span class="monospaced">&lt;directsync | none | unsafe | writeback | writethrough&gt;</span> 
</dt>
<dd>
<p>
The drive’s cache mode
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">cyls</span>=<span class="monospaced">&lt;integer&gt;</span> 
</dt>
<dd>
<p>
Force the drive’s physical geometry to have a specific cylinder count.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">detect_zeroes</span>=<span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Controls whether to detect and try to optimize writes of zeroes.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">discard</span>=<span class="monospaced">&lt;ignore | on&gt;</span> 
</dt>
<dd>
<p>
Controls whether to pass discard/trim requests to the underlying storage.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">file</span>=<span class="monospaced">&lt;volume&gt;</span> 
</dt>
<dd>
<p>
The drive’s backing volume.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">format</span>=<span class="monospaced">&lt;cloop | cow | qcow | qcow2 | qed | raw | vmdk&gt;</span> 
</dt>
<dd>
<p>
The drive’s backing file’s data format.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">heads</span>=<span class="monospaced">&lt;integer&gt;</span> 
</dt>
<dd>
<p>
Force the drive’s physical geometry to have a specific head count.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">iops</span>=<span class="monospaced">&lt;iops&gt;</span> 
</dt>
<dd>
<p>
Maximum r/w I/O in operations per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">iops_max</span>=<span class="monospaced">&lt;iops&gt;</span> 
</dt>
<dd>
<p>
Maximum unthrottled r/w I/O pool in operations per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">iops_max_length</span>=<span class="monospaced">&lt;seconds&gt;</span> 
</dt>
<dd>
<p>
Maximum length of I/O bursts in seconds.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">iops_rd</span>=<span class="monospaced">&lt;iops&gt;</span> 
</dt>
<dd>
<p>
Maximum read I/O in operations per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">iops_rd_max</span>=<span class="monospaced">&lt;iops&gt;</span> 
</dt>
<dd>
<p>
Maximum unthrottled read I/O pool in operations per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">iops_rd_max_length</span>=<span class="monospaced">&lt;seconds&gt;</span> 
</dt>
<dd>
<p>
Maximum length of read I/O bursts in seconds.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">iops_wr</span>=<span class="monospaced">&lt;iops&gt;</span> 
</dt>
<dd>
<p>
Maximum write I/O in operations per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">iops_wr_max</span>=<span class="monospaced">&lt;iops&gt;</span> 
</dt>
<dd>
<p>
Maximum unthrottled write I/O pool in operations per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">iops_wr_max_length</span>=<span class="monospaced">&lt;seconds&gt;</span> 
</dt>
<dd>
<p>
Maximum length of write I/O bursts in seconds.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">iothread</span>=<span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Whether to use iothreads for this drive
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mbps</span>=<span class="monospaced">&lt;mbps&gt;</span> 
</dt>
<dd>
<p>
Maximum r/w speed in megabytes per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mbps_max</span>=<span class="monospaced">&lt;mbps&gt;</span> 
</dt>
<dd>
<p>
Maximum unthrottled r/w pool in megabytes per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mbps_rd</span>=<span class="monospaced">&lt;mbps&gt;</span> 
</dt>
<dd>
<p>
Maximum read speed in megabytes per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mbps_rd_max</span>=<span class="monospaced">&lt;mbps&gt;</span> 
</dt>
<dd>
<p>
Maximum unthrottled read pool in megabytes per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mbps_wr</span>=<span class="monospaced">&lt;mbps&gt;</span> 
</dt>
<dd>
<p>
Maximum write speed in megabytes per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mbps_wr_max</span>=<span class="monospaced">&lt;mbps&gt;</span> 
</dt>
<dd>
<p>
Maximum unthrottled write pool in megabytes per second.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">media</span>=<span class="monospaced">&lt;cdrom | disk&gt;</span> (<em>default =</em> <span class="monospaced">disk</span>)
</dt>
<dd>
<p>
The drive’s media type.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">replicate</span>=<span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Whether the drive should considered for replication jobs.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">rerror</span>=<span class="monospaced">&lt;ignore | report | stop&gt;</span> 
</dt>
<dd>
<p>
Read error action.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">ro</span>=<span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Whether the drive is read-only.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">secs</span>=<span class="monospaced">&lt;integer&gt;</span> 
</dt>
<dd>
<p>
Force the drive’s physical geometry to have a specific sector count.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">serial</span>=<span class="monospaced">&lt;serial&gt;</span> 
</dt>
<dd>
<p>
The drive’s reported serial number, url-encoded, up to 20 bytes long.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">shared</span>=<span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Mark this locally-managed volume as available on all nodes.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">This option does not share the volume automatically, it assumes it is shared already!</td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">size</span>=<span class="monospaced">&lt;DiskSize&gt;</span> 
</dt>
<dd>
<p>
Disk size. This is purely informational and has no effect.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">snapshot</span>=<span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Controls qemu’s snapshot mode feature. If activated, changes made to the disk are temporary and will be discarded when the VM is shutdown.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">trans</span>=<span class="monospaced">&lt;auto | lba | none&gt;</span> 
</dt>
<dd>
<p>
Force disk geometry bios translation mode.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">werror</span>=<span class="monospaced">&lt;enospc | ignore | report | stop&gt;</span> 
</dt>
<dd>
<p>
Write error action.
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">vmgenid</span>: <span class="monospaced">&lt;UUID&gt;</span> (<em>default =</em> <span class="monospaced">1 (autogenerated)</span>)
</dt>
<dd>
<p>
The VM generation ID (vmgenid) device exposes a 128-bit integer value identifier to the guest OS. This allows to notify the guest operating system when the virtual machine is executed with a different configuration (e.g. snapshot execution or creation from a template). The guest operating system notices the change, and is then able to react as appropriate by marking its copies of distributed databases as dirty, re-initializing its random number generator, etc.
Note that auto-creation only works when done through API/CLI create or update methods, but not when manually editing the config file.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">vmstatestorage</span>: <span class="monospaced">&lt;storage ID&gt;</span> 
</dt>
<dd>
<p>
Default storage for VM state volumes/files.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">watchdog</span>: <span class="monospaced">[[model=]&lt;i6300esb|ib700&gt;] [,action=&lt;enum&gt;]</span> 
</dt>
<dd>
<p>
Create a virtual hardware watchdog device. Once enabled (by a guest action), the watchdog must be periodically polled by an agent inside the guest or else the watchdog will reset the guest (or execute the respective action specified)
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">action</span>=<span class="monospaced">&lt;debug | none | pause | poweroff | reset | shutdown&gt;</span> 
</dt>
<dd>
<p>
The action to perform if after activation the guest fails to poll the watchdog in time.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">model</span>=<span class="monospaced">&lt;i6300esb | ib700&gt;</span> (<em>default =</em> <span class="monospaced">i6300esb</span>)
</dt>
<dd>
<p>
Watchdog type to emulate.
</p>
</dd>
</dl></div>
</dd>
</dl></div>
</div>
</div>
<div class="sect2">
<h3 id="_locks">
<span>10.15. Locks</span>
 <a class="headerlink" href="#_locks" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Online migrations, snapshots and backups (<span class="monospaced">vzdump</span>) set a lock to prevent
incompatible concurrent actions on the affected VMs. Sometimes you need to
remove such a lock manually (for example after a power failure).</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># qm unlock &lt;vmid&gt;</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Caution" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKdUlEQVRoge1Ze1AV1x3+zt279wEi
DWCYGzRVktqa1MEmmtbWR22ncXxUrTDWV/5IqG2wUAUfmUwSkk6V+EAJCEQC6figwZBqZtRxqukf
tebRZBpFG1S0hiRErwiaKK977+6eX//YPXt37+UiIJlMZnpmdnb33PP4vt97z2VEhG9yc3zdAO60
/Z/A192+8QScX8Wifr+fWltbzffU1FT4fD72Vew15ASampqovr4eBw8eNPvGjRuHzMxMmj9//tCT
IKIhu958801yuVwEoNdr48aNNJT7EdHQEdi/fz/JshwTvLiKioqGlMSQLFJfX0+MMRvQsWNSqXLz
H2jez78fRWLTpk1DRuKOF6irqyOn02kDOP8XGdTz+VFSP91Hatu79NRvZ0SR2LJly5CQuKPJtbW1
JEmSDdivHp1AwctHSblYSsqFElLOF5PavIvWZU+LIrF169Y7JjHoibt3744CnzXrIQpeOUbKhVJS
mkpIPV9MyrnNpJwtIuVSNa15/MdRJLZt23ZHJBjRwIu5N954gxYvXgzOudm3aO5E1L5SBNZ1Hoxr
INLAyH6HJxVP/WkXtu9+37ZeSUkJVq9ePagQO+BMXFNTEwV+ybxHsPeVIjg6zwOkgSWNhuOuUUBC
KhxJY0AggDSgpwWbn1mO1csftq1ZUFCA0tLSQZXFA0pkVVVVlJOTA6vWsmY9hF07N0AS4OU48LgR
WLO+FPFxboxMS8G5hlPYmPMI4jwM1P0Jtj67DJyrKHvtNADdjPPz8+FwOCgvL29gmuivrVVUVESF
yqxZD1PoyjFSL+7QHbapmJTml+n9EzVRtl676ZekNKwl5VQ+KSfzKHR2E+UutodYxhiVl5cPyCf6
ZUJlZWWUm5trk/wTi6bgtVdf1M2Gq2DEwUgDNBWhUDBqjfSRwwGoIOEP3c3Y/uxirFw0zibMvLw8
VFZW9tucbkugoqKCVq1aZQO/YslUVJU+D9bVBIDroLgK4hygEDweybYGY8AD6d8yftfASL+j+2O8
9NwiPJk51kYiNzcXO3fu7BeJPgkUFxdTbm6ure/JZdNRub0QrOMcGFcBrgLgAHEAGqD2ID7OZZsz
Ji0RcR6HAV4DkaqPJxXouoSywkz8buF9NhIrV65EVVXVbUnEJHDixAlat26drS9n+U+xo/hZMMNh
yZA+uAbAIKOGMCJlmG3eA/clAYaJEamGBlRdC6QBXZewo3AhVswfbSORk5ODDz/8sE8SvRLw+/1U
XV0dBb5MgOe6LUNIEqpOwgCUlOCESw4HuAfG3KX7B2kAGaTJ0Jro77yI8ucX4DfzRtlIHD58GH6/
PyaJXgk0NjZi79695vuUiffrkr91Vt8cHAwc4FZJGiZEKqDcwsi0ZHP+d749POy8MS7iGtBxARWF
8/CDsQnm3BdeeAHWj6N+EYhspSVbgI5zYYkZGmAwzIEbwA0tUM9NfO+7aeb89HuGRQBWLcSNZxj3
zvOYMzk5NpjBEHi99mXD5lXdjqGDJh42h7CEOXhnK3448X5z/shUb1jSpIEM8yFuzOUcjAsNcnR0
KXdG4MEHH8TEiRPN9z0H3sfRE41m+GNcs9i0LjkGTTcrUoGem5g7/V5z/ohElw4O+jxGYQ3qJqmB
jN/+cbINtcfCJvPYY4+hr3qtVwI+n49lZmaa71fbbuH3hX/FycbPQKSCoIIMQEQcehjVo5Gw8/F3
3cSvH03H2HuHY1icQzcvYWKGIMiITMwQwjtnruPpqk9w/ZZq7p2RkYEJEyaAYrCIWY36/X7KyMhA
W1ub2XfP3Qmo3ZKFaQ+P0gmYIdSoOkU4lWSwxGRAdqHL/xm86k27w0blAw3/PncD6yo/xrsfdZj7
paSkoLGxESNGjAgDZsxWK8X0AZ/Px44fP460tLAzXrnWgUX5r+PYuxf1mG/Gf80sofWy2YuWbg/+
uONtPLHhPRw+0WIHL8hbwG95rcUGPiEhAYcOHUJSUhI456YZRWqiz+8BIqLm5mZMmjQJN27cMPsT
E9zYtWE25k4bYzh3mABIBRwOHPgPx+LH/6yDiXPi6pHZcEpk+I5qmtGpCzdQecCPPUevmevLsowj
R45gypQpcDgc5mVowKaJmBoQTEePHo2GhgaMGxcuum52BLHsqUOoP3oWgLB9Sz7QgpgxJoThw/SS
IutnaXBKZEpcgD998Uvs+3ubDbzT6URNTQ0mTZoEVVWhaRo459A0TeCy4etVA+JH8RvnHK2trZgz
Zw5Onz5tjnPJDrxSOAPLZt1nSVIqwDmIVJw6dx1nLt3A3J/cjZThkhE29cvf3o2KA59ja91lcz3G
GDZv3oylS5fC7XbD6XTC6XRCkiRIktS7Jm5HQLDXNA3Xr19HVlYWPvjgA4vEGLav+RHmTxsJX4rb
8AsO4kbOIG6GzTD4HvzlmB/PVH9q2zc/Px/Z2dnwer1wu92QZRkulwuSJEGWZZMEY8wkEGVCVvCC
AOcciqIgPj4e+/btw9SpU83xqkpYteU9HHm7Bf62rnC2RjikisTFSMPV9m4ceucannvVDn7p0qVY
uHAhgsEggsEgQqFQlAlZnVm0PjOxIKFpGogIqqrC5XKhuroa06dPt4wDcl78F+qPNeNKe6cRoTjs
ZYOKK+3dON7wBQrKm2H5pMbMmTOxZMkSBAIB9PT0oKenB4FAAIqiQFEUU4hRRyq3I2AlIZ4553A6
nSgvL8fs2bNtY9eVNaDub58YmhDZWpe+v70bJ5tu4clt/4WihqU4efJkrFixAkRkAg4Gg1AUBaqq
muDF/pGtX7WQNXeIZ0mSUFRUhAULFtjGPl3ZiJdev4QrbZ16zUQqrrZ342JLN5ZvaEJ3ICz68ePH
IycnB5qmmZIW4EOhUJTZWEKo+Rx1KsEYY9Zk4XA4IEmSKXnOuRkVJElCYWEhvF4v6urqzDVK9n2M
zu4Qnl5+L9q/7MHltgBWbr9kA5+eno7c3FzIsoyI5GrTeCzgosU8VmGMmSGLiEwSsixH2WNBQQEA
2EhUH/wc7330BaaOH479/2zHtS/CFabP58OaNWvg9Xp1EE4nZFmGLMtwu93wer1wuVy2ECpJko2I
iTNWJrZGI+HEwi6FisUVCAQQCASwZ88eVFZW9lk9JicnY/369fD5fPB4PCbo+Ph4k4TH44HL5YLX
64XX6zXH9EKE9aUBRkTEGLNJP1Kd4nI4HMjOzkZ8fDyKi4ttJ3eiJSYmYu3atRg1apQJzuv1wuPx
wOPxwOFwmEBFHnA6nXC5XGYSiywlbns2SnoDANN0hMNZnS0UCpkaOnPmDN566y20tLSY2ktLS8OM
GTOQkpICt9uNuLg4M2EJ8CJZSZJkZmFBwkrAWpH2+3BXEBEkRIhTVdVMOCL5qKqKy5cvQ1EUaJoG
VVWRnKx/Jlpt3eVy2QCKu9C68AFr9o0spwd0Oh1ZYggCIlNaiQktiXdhZgKQABcp3ci7LMtM7B0J
fsAErJoAYItG1mdr+hfvVlACtHDIXkploYXbHvQOikDEe6/ZmjFmAhf9ZvJxOk2Qol88W4kYz32S
GNQfHLG+TyMTkAVEr88xQfVnkBg7GAKixSIykDYQsL21/wFkW/B5QqT9lwAAAABJRU5ErkJggg==">
</td>
<td class="content">Only do that if you are sure the action which set the lock is
no longer running.</td>
</tr></tbody></table>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="chapter_pct">
11. Proxmox Container Toolkit
 <a class="headerlink" href="#chapter_pct" title="Permalink to this heading"></a>
</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Containers are a lightweight alternative to fully virtualized machines (VMs).
They use the kernel of the host system that they run on, instead of emulating a
full operating system (OS). This means that containers can access resources on
the host system directly.</p></div>
<div class="paragraph">
<p>The runtime costs for containers is low, usually negligible. However, there are
some drawbacks that need be considered:</p></div>
<div class="ulist"><ul>
<li>
<p>
Only Linux distributions can be run in Proxmox Containers. It is not possible to run
  other operating systems like, for example, FreeBSD or Microsoft Windows
  inside a container.
</p>
</li>
<li>
<p>
For security reasons, access to host resources needs to be restricted.
  Therefore, containers run in their own separate namespaces. Additionally some
  syscalls (user space requests to the Linux kernel) are not allowed within containers.
</p>
</li>
</ul></div>
<div class="paragraph">
<p>Proxmox VE uses <a href="https://linuxcontainers.org/lxc/introduction/">Linux Containers (LXC)</a> as its underlying
container technology. The “Proxmox Container Toolkit” (<span class="monospaced">pct</span>) simplifies the
usage and management of LXC, by providing an interface that abstracts
complex tasks.</p></div>
<div class="paragraph">
<p>Containers are tightly integrated with Proxmox VE. This means that they are aware of
the cluster setup, and they can use the same network and storage resources as
virtual machines. You can also use the Proxmox VE firewall, or manage containers
using the HA framework.</p></div>
<div class="paragraph">
<p>Our primary goal is to offer an environment that provides the benefits of using a
VM, but without the additional overhead. This means that Proxmox Containers can
be categorized as “System Containers”, rather than “Application Containers”.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">If you want to run application containers, for example, <em>Docker</em> images, it
is recommended that you run them inside a Proxmox QEMU VM. This will give you
all the advantages of application containerization, while also providing the
benefits that VMs offer, such as strong isolation from the host and the ability
to live-migrate, which otherwise isn’t possible with containers.</td>
</tr></tbody></table>
</div>
<div class="sect2">
<h3 id="_technology_overview">
<span>11.1. Technology Overview</span>
 <a class="headerlink" href="#_technology_overview" title="Permalink to this heading"></a>
</h3>
<div class="ulist"><ul>
<li>
<p>
LXC (<a href="https://linuxcontainers.org/">https://linuxcontainers.org/</a>)
</p>
</li>
<li>
<p>
Integrated into Proxmox VE graphical web user interface (GUI)
</p>
</li>
<li>
<p>
Easy to use command-line tool <span class="monospaced">pct</span>
</p>
</li>
<li>
<p>
Access via Proxmox VE REST API
</p>
</li>
<li>
<p>
<em>lxcfs</em> to provide containerized /proc file system
</p>
</li>
<li>
<p>
Control groups (<em>cgroups</em>) for resource isolation and limitation
</p>
</li>
<li>
<p>
<em>AppArmor</em> and <em>seccomp</em> to improve security
</p>
</li>
<li>
<p>
Modern Linux kernels
</p>
</li>
<li>
<p>
Image based deployment (<a href="#pct_supported_distributions">templates</a>)
</p>
</li>
<li>
<p>
Uses Proxmox VE <a href="#chapter_storage">storage library</a>
</p>
</li>
<li>
<p>
Container setup from host (network, DNS, storage, etc.)
</p>
</li>
</ul></div>
</div>
<div class="sect2">
<h3 id="pct_supported_distributions">
<span>11.2. Supported Distributions</span>
 <a class="headerlink" href="#pct_supported_distributions" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>List of officially supported distributions can be found below.</p></div>
<div class="paragraph">
<p>Templates for the following distributions are available through our
repositories. You can use <a href="#pct_container_images">pveam</a> tool or the
Graphical User Interface to download them.</p></div>
<div class="sect3">
<h4 id="_alpine_linux">11.2.1. Alpine Linux
 <a class="headerlink" href="#_alpine_linux" title="Permalink to this heading"></a>
</h4>
<div class="quoteblock">
<div class="content">
<div class="paragraph">
<p>Alpine Linux is a security-oriented, lightweight Linux distribution based on
musl libc and busybox.</p></div>
</div>
<div class="attribution">
— <a href="https://alpinelinux.org">https://alpinelinux.org</a>
</div></div>
<div class="paragraph">
<p>For currently supported releases see:</p></div>
<div class="paragraph">
<p><a href="https://alpinelinux.org/releases/">https://alpinelinux.org/releases/</a></p></div>
</div>
<div class="sect3">
<h4 id="_arch_linux">11.2.2. Arch Linux
 <a class="headerlink" href="#_arch_linux" title="Permalink to this heading"></a>
</h4>
<div class="quoteblock">
<div class="content">
<div class="paragraph">
<p>Arch Linux, a lightweight and flexible Linux® distribution that tries to Keep It Simple.</p></div>
</div>
<div class="attribution">
— <a href="https://archlinux.org/">https://archlinux.org/</a>
</div></div>
<div class="paragraph">
<p>Arch Linux is using a rolling-release model, see its wiki for more details:</p></div>
<div class="paragraph">
<p><a href="https://wiki.archlinux.org/title/Arch_Linux">https://wiki.archlinux.org/title/Arch_Linux</a></p></div>
</div>
<div class="sect3">
<h4 id="_centos_almalinux_rocky_linux">11.2.3. CentOS, Almalinux, Rocky Linux
 <a class="headerlink" href="#_centos_almalinux_rocky_linux" title="Permalink to this heading"></a>
</h4>
<div class="sect4">
<h5 id="_centos_centos_stream">CentOS / CentOS Stream
 <a class="headerlink" href="#_centos_centos_stream" title="Permalink to this heading"></a>
</h5>
<div class="quoteblock">
<div class="content">
<div class="paragraph">
<p>The CentOS Linux distribution is a stable, predictable, manageable and
reproducible platform derived from the sources of Red Hat Enterprise Linux
(RHEL)</p></div>
</div>
<div class="attribution">
— <a href="https://centos.org">https://centos.org</a>
</div></div>
<div class="paragraph">
<p>For currently supported releases see:</p></div>
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/CentOS#End-of-support_schedule">https://en.wikipedia.org/wiki/CentOS#End-of-support_schedule</a></p></div>
</div>
<div class="sect4">
<h5 id="_almalinux">Almalinux
 <a class="headerlink" href="#_almalinux" title="Permalink to this heading"></a>
</h5>
<div class="quoteblock">
<div class="content">
<div class="paragraph">
<p>An Open Source, community owned and governed, forever-free enterprise Linux
distribution, focused on long-term stability, providing a robust
production-grade platform. AlmaLinux OS is 1:1 binary compatible with RHEL® and
pre-Stream CentOS.</p></div>
</div>
<div class="attribution">
— <a href="https://almalinux.org">https://almalinux.org</a>
</div></div>
<div class="paragraph">
<p>For currently supported releases see:</p></div>
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/AlmaLinux#Releases">https://en.wikipedia.org/wiki/AlmaLinux#Releases</a></p></div>
</div>
<div class="sect4">
<h5 id="_rocky_linux">Rocky Linux
 <a class="headerlink" href="#_rocky_linux" title="Permalink to this heading"></a>
</h5>
<div class="quoteblock">
<div class="content">
<div class="paragraph">
<p>Rocky Linux is a community enterprise operating system designed to be 100%
bug-for-bug compatible with America’s top enterprise Linux distribution now
that its downstream partner has shifted direction.</p></div>
</div>
<div class="attribution">
— <a href="https://rockylinux.org">https://rockylinux.org</a>
</div></div>
<div class="paragraph">
<p>For currently supported releases see:</p></div>
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/Rocky_Linux#Releases">https://en.wikipedia.org/wiki/Rocky_Linux#Releases</a></p></div>
</div>
</div>
<div class="sect3">
<h4 id="_debian">11.2.4. Debian
 <a class="headerlink" href="#_debian" title="Permalink to this heading"></a>
</h4>
<div class="quoteblock">
<div class="content">
<div class="paragraph">
<p>Debian is a free operating system, developed and maintained by the Debian
project. A free Linux distribution with thousands of applications to meet our
users' needs.</p></div>
</div>
<div class="attribution">
— <a href="https://www.debian.org/intro/index#software">https://www.debian.org/intro/index#software</a>
</div></div>
<div class="paragraph">
<p>For currently supported releases see:</p></div>
<div class="paragraph">
<p><a href="https://www.debian.org/releases/stable/releasenotes">https://www.debian.org/releases/stable/releasenotes</a></p></div>
</div>
<div class="sect3">
<h4 id="_devuan">11.2.5. Devuan
 <a class="headerlink" href="#_devuan" title="Permalink to this heading"></a>
</h4>
<div class="quoteblock">
<div class="content">
<div class="paragraph">
<p>Devuan GNU+Linux is a fork of Debian without systemd that allows users to
reclaim control over their system by avoiding unnecessary entanglements and
ensuring Init Freedom.</p></div>
</div>
<div class="attribution">
— <a href="https://www.devuan.org">https://www.devuan.org</a>
</div></div>
<div class="paragraph">
<p>For currently supported releases see:</p></div>
<div class="paragraph">
<p><a href="https://www.devuan.org/os/releases">https://www.devuan.org/os/releases</a></p></div>
</div>
<div class="sect3">
<h4 id="_fedora">11.2.6. Fedora
 <a class="headerlink" href="#_fedora" title="Permalink to this heading"></a>
</h4>
<div class="quoteblock">
<div class="content">
<div class="paragraph">
<p>Fedora creates an innovative, free, and open source platform for hardware,
clouds, and containers that enables software developers and community members
to build tailored solutions for their users.</p></div>
</div>
<div class="attribution">
— <a href="https://getfedora.org">https://getfedora.org</a>
</div></div>
<div class="paragraph">
<p>For currently supported releases see:</p></div>
<div class="paragraph">
<p><a href="https://fedoraproject.org/wiki/Releases">https://fedoraproject.org/wiki/Releases</a></p></div>
</div>
<div class="sect3">
<h4 id="_gentoo">11.2.7. Gentoo
 <a class="headerlink" href="#_gentoo" title="Permalink to this heading"></a>
</h4>
<div class="quoteblock">
<div class="content">
<div class="paragraph">
<p>a highly flexible, source-based Linux distribution.</p></div>
</div>
<div class="attribution">
— <a href="https://www.gentoo.org">https://www.gentoo.org</a>
</div></div>
<div class="paragraph">
<p>Gentoo is using a rolling-release model.</p></div>
</div>
<div class="sect3">
<h4 id="_opensuse">11.2.8. OpenSUSE
 <a class="headerlink" href="#_opensuse" title="Permalink to this heading"></a>
</h4>
<div class="quoteblock">
<div class="content">
<div class="paragraph">
<p>The makers' choice for sysadmins, developers and desktop users.</p></div>
</div>
<div class="attribution">
— <a href="https://www.opensuse.org">https://www.opensuse.org</a>
</div></div>
<div class="paragraph">
<p>For currently supported releases see:</p></div>
<div class="paragraph">
<p><a href="https://get.opensuse.org/leap/">https://get.opensuse.org/leap/</a></p></div>
</div>
<div class="sect3">
<h4 id="_ubuntu">11.2.9. Ubuntu
 <a class="headerlink" href="#_ubuntu" title="Permalink to this heading"></a>
</h4>
<div class="quoteblock">
<div class="content">
<div class="paragraph">
<p>Ubuntu is the modern, open source operating system on Linux for the enterprise
server, desktop, cloud, and IoT.</p></div>
</div>
<div class="attribution">
— <a href="https://ubuntu.com/">https://ubuntu.com/</a>
</div></div>
<div class="paragraph">
<p>For currently supported releases see:</p></div>
<div class="paragraph">
<p><a href="https://wiki.ubuntu.com/Releases">https://wiki.ubuntu.com/Releases</a></p></div>
</div>
</div>
<div class="sect2">
<h3 id="pct_container_images">
<span>11.3. Container Images</span>
 <a class="headerlink" href="#pct_container_images" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Container images, sometimes also referred to as “templates” or
“appliances”, are <span class="monospaced">tar</span> archives which contain everything to run a container.</p></div>
<div class="paragraph">
<p>Proxmox VE itself provides a variety of basic templates for the
<a href="#pct_supported_distributions">most common Linux distributions</a>.  They can be
downloaded using the GUI or the <span class="monospaced">pveam</span> (short for Proxmox VE Appliance Manager)
command-line utility.  Additionally, <a href="https://www.turnkeylinux.org/">TurnKey
Linux</a> container templates are also available to download.</p></div>
<div class="paragraph">
<p>The list of available templates is updated daily through the <em>pve-daily-update</em>
timer. You can also trigger an update manually by executing:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># pveam update</pre>
</div></div>
<div class="paragraph">
<p>To view the list of available images run:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># pveam available</pre>
</div></div>
<div class="paragraph">
<p>You can restrict this large list by specifying the <span class="monospaced">section</span> you are
interested in, for example basic <span class="monospaced">system</span> images:</p></div>
<div class="listingblock">
<div class="title">List available system images</div>
<div class="content monospaced">
<pre># pveam available --section system
system          alpine-3.12-default_20200823_amd64.tar.xz
system          alpine-3.13-default_20210419_amd64.tar.xz
system          alpine-3.14-default_20210623_amd64.tar.xz
system          archlinux-base_20210420-1_amd64.tar.gz
system          centos-7-default_20190926_amd64.tar.xz
system          centos-8-default_20201210_amd64.tar.xz
system          debian-9.0-standard_9.7-1_amd64.tar.gz
system          debian-10-standard_10.7-1_amd64.tar.gz
system          devuan-3.0-standard_3.0_amd64.tar.gz
system          fedora-33-default_20201115_amd64.tar.xz
system          fedora-34-default_20210427_amd64.tar.xz
system          gentoo-current-default_20200310_amd64.tar.xz
system          opensuse-15.2-default_20200824_amd64.tar.xz
system          ubuntu-16.04-standard_16.04.5-1_amd64.tar.gz
system          ubuntu-18.04-standard_18.04.1-1_amd64.tar.gz
system          ubuntu-20.04-standard_20.04-1_amd64.tar.gz
system          ubuntu-20.10-standard_20.10-1_amd64.tar.gz
system          ubuntu-21.04-standard_21.04-1_amd64.tar.gz</pre>
</div></div>
<div class="paragraph">
<p>Before you can use such a template, you need to download them into one of your
storages. If you’re unsure to which one, you can simply use the <span class="monospaced">local</span> named
storage for that purpose. For clustered installations, it is preferred to use a
shared storage so that all nodes can access those images.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># pveam download local debian-10.0-standard_10.0-1_amd64.tar.gz</pre>
</div></div>
<div class="paragraph">
<p>You are now ready to create containers using that image, and you can list all
downloaded images on storage <span class="monospaced">local</span> with:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># pveam list local
local:vztmpl/debian-10.0-standard_10.0-1_amd64.tar.gz  219.95MB</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">You can also use the Proxmox VE web interface GUI to download, list and delete
container templates.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p><span class="monospaced">pct</span> uses them to create a new container, for example:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># pct create 999 local:vztmpl/debian-10.0-standard_10.0-1_amd64.tar.gz</pre>
</div></div>
<div class="paragraph">
<p>The above command shows you the full Proxmox VE volume identifiers. They include the
storage name, and most other Proxmox VE commands can use them. For example you can
delete that image later with:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># pveam remove local:vztmpl/debian-10.0-standard_10.0-1_amd64.tar.gz</pre>
</div></div>
</div>
<div class="sect2">
<h3 id="pct_settings">
<span>11.4. Container Settings</span>
 <a class="headerlink" href="#pct_settings" title="Permalink to this heading"></a>
</h3>
<div class="sect3">
<h4 id="pct_general">11.4.1. General Settings
 <a class="headerlink" href="#pct_general" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-create-ct-general.png">
<img src="images/screenshot/gui-create-ct-general.png" alt="screenshot/gui-create-ct-general.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>General settings of a container include</p></div>
<div class="ulist"><ul>
<li>
<p>
the <strong>Node</strong> : the physical server on which the container will run
</p>
</li>
<li>
<p>
the <strong>CT ID</strong>: a unique number in this Proxmox VE installation used to identify your
  container
</p>
</li>
<li>
<p>
<strong>Hostname</strong>: the hostname of the container
</p>
</li>
<li>
<p>
<strong>Resource Pool</strong>: a logical group of containers and VMs
</p>
</li>
<li>
<p>
<strong>Password</strong>: the root password of the container
</p>
</li>
<li>
<p>
<strong>SSH Public Key</strong>: a public key for connecting to the root account over SSH
</p>
</li>
<li>
<p>
<strong>Unprivileged container</strong>: this option allows to choose at creation time
  if you want to create a privileged or unprivileged container.
</p>
</li>
</ul></div>
<div class="sect4">
<h5 id="_unprivileged_containers">Unprivileged Containers
 <a class="headerlink" href="#_unprivileged_containers" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Unprivileged containers use a new kernel feature called user namespaces.
The root UID 0 inside the container is mapped to an unprivileged user outside
the container. This means that most security issues (container escape, resource
abuse, etc.) in these containers will affect a random unprivileged user, and
would be a generic kernel security bug rather than an LXC issue. The LXC team
thinks unprivileged containers are safe by design.</p></div>
<div class="paragraph">
<p>This is the default option when creating a new container.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">If the container uses systemd as an init system, please be aware the
systemd version running inside the container should be equal to or greater than
220.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect4">
<h5 id="_privileged_containers">Privileged Containers
 <a class="headerlink" href="#_privileged_containers" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Security in containers is achieved by using mandatory access control <em>AppArmor</em>
restrictions, <em>seccomp</em> filters and Linux kernel namespaces. The LXC team
considers this kind of container as unsafe, and they will not consider new
container escape exploits to be security issues worthy of a CVE and quick fix.
That’s why privileged containers should only be used in trusted environments.</p></div>
</div>
</div>
<div class="sect3">
<h4 id="pct_cpu">11.4.2. CPU
 <a class="headerlink" href="#pct_cpu" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-create-ct-cpu.png">
<img src="images/screenshot/gui-create-ct-cpu.png" alt="screenshot/gui-create-ct-cpu.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>You can restrict the number of visible CPUs inside the container using the
<span class="monospaced">cores</span> option. This is implemented using the Linux <em>cpuset</em> cgroup
(<strong>c</strong>ontrol <strong>group</strong>).
A special task inside <span class="monospaced">pvestatd</span> tries to distribute running containers among
available CPUs periodically.
To view the assigned CPUs run the following command:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># pct cpusets
 ---------------------
 102:              6 7
 105:      2 3 4 5
 108:  0 1
 ---------------------</pre>
</div></div>
<div class="paragraph">
<p>Containers use the host kernel directly. All tasks inside a container are
handled by the host CPU scheduler. Proxmox VE uses the Linux <em>CFS</em> (<strong>C</strong>ompletely
<strong>F</strong>air <strong>S</strong>cheduler) scheduler by default, which has additional bandwidth
control options.</p></div>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<span class="monospaced">cpulimit</span>: 
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
You can use this option to further limit assigned CPU time.
Please note that this is a floating point number, so it is perfectly valid to
assign two cores to a container, but restrict overall CPU consumption to half a
core.
</p>
<div class="listingblock">
<div class="content monospaced">
<pre>cores: 2
cpulimit: 0.5</pre>
</div></div>
</td>
</tr>
<tr>
<td class="hdlist1">
<span class="monospaced">cpuunits</span>: 
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
This is a relative weight passed to the kernel scheduler. The
larger the number is, the more CPU time this container gets. Number is relative
to the weights of all the other running containers. The default is <span class="monospaced">100</span> (or
<span class="monospaced">1024</span> if the host uses legacy cgroup v1). You can use this setting to
prioritize some containers.
</p>
</td>
</tr>
</tbody></table></div>
</div>
<div class="sect3">
<h4 id="pct_memory">11.4.3. Memory
 <a class="headerlink" href="#pct_memory" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-create-ct-memory.png">
<img src="images/screenshot/gui-create-ct-memory.png" alt="screenshot/gui-create-ct-memory.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>Container memory is controlled using the cgroup memory controller.</p></div>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<span class="monospaced">memory</span>: 
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Limit overall memory usage. This corresponds to the
<span class="monospaced">memory.limit_in_bytes</span> cgroup setting.
</p>
</td>
</tr>
<tr>
<td class="hdlist1">
<span class="monospaced">swap</span>: 
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Allows the container to use additional swap memory from the host
swap space. This corresponds to the <span class="monospaced">memory.memsw.limit_in_bytes</span> cgroup
setting, which is set to the sum of both value (<span class="monospaced">memory + swap</span>).
</p>
</td>
</tr>
</tbody></table></div>
</div>
<div class="sect3">
<h4 id="pct_mount_points">11.4.4. Mount Points
 <a class="headerlink" href="#pct_mount_points" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-create-ct-root-disk.png">
<img src="images/screenshot/gui-create-ct-root-disk.png" alt="screenshot/gui-create-ct-root-disk.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>The root mount point is configured with the <span class="monospaced">rootfs</span> property. You can
configure up to 256 additional mount points. The corresponding options are
called <span class="monospaced">mp0</span> to <span class="monospaced">mp255</span>. They can contain the following settings:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">rootfs</span>: <span class="monospaced">[volume=]&lt;volume&gt; [,acl=&lt;1|0&gt;] [,mountoptions=&lt;opt[;opt...]&gt;] [,quota=&lt;1|0&gt;] [,replicate=&lt;1|0&gt;] [,ro=&lt;1|0&gt;] [,shared=&lt;1|0&gt;] [,size=&lt;DiskSize&gt;]</span> 
</dt>
<dd>
<p>
Use volume as container root. See below for a detailed description of all options.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mp[n]</span>: <span class="monospaced">[volume=]&lt;volume&gt; ,mp=&lt;Path&gt; [,acl=&lt;1|0&gt;] [,backup=&lt;1|0&gt;] [,mountoptions=&lt;opt[;opt...]&gt;] [,quota=&lt;1|0&gt;] [,replicate=&lt;1|0&gt;] [,ro=&lt;1|0&gt;] [,shared=&lt;1|0&gt;] [,size=&lt;DiskSize&gt;]</span> 
</dt>
<dd>
<p>
Use volume as container mount point. Use the special syntax STORAGE_ID:SIZE_IN_GiB to allocate a new volume.
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">acl</span>=<span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Explicitly enable or disable ACL support.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">backup</span>=<span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Whether to include the mount point in backups (only used for volume mount points).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mountoptions</span>=<span class="monospaced">&lt;opt[;opt...]&gt;</span> 
</dt>
<dd>
<p>
Extra mount options for rootfs/mps.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mp</span>=<span class="monospaced">&lt;Path&gt;</span> 
</dt>
<dd>
<p>
Path to the mount point as seen from inside the container.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Must not contain any symlinks for security reasons.</td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">quota</span>=<span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Enable user quotas inside the container (not supported with zfs subvolumes)
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">replicate</span>=<span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Will include this volume to a storage replica job.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">ro</span>=<span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Read-only mount point
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">shared</span>=<span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Mark this non-volume mount point as available on all nodes.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">This option does not share the mount point automatically, it assumes it is shared already!</td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">size</span>=<span class="monospaced">&lt;DiskSize&gt;</span> 
</dt>
<dd>
<p>
Volume size (read only value).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">volume</span>=<span class="monospaced">&lt;volume&gt;</span> 
</dt>
<dd>
<p>
Volume, device or directory to mount into the container.
</p>
</dd>
</dl></div>
</dd>
</dl></div>
<div class="paragraph">
<p>Currently there are three types of mount points: storage backed mount points,
bind mounts, and device mounts.</p></div>
<div class="listingblock">
<div class="title">Typical container <span class="monospaced">rootfs</span> configuration</div>
<div class="content monospaced">
<pre>rootfs: thin1:base-100-disk-1,size=8G</pre>
</div></div>
<div class="sect4">
<h5 id="_storage_backed_mount_points">Storage Backed Mount Points
 <a class="headerlink" href="#_storage_backed_mount_points" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Storage backed mount points are managed by the Proxmox VE storage subsystem and come
in three different flavors:</p></div>
<div class="ulist"><ul>
<li>
<p>
Image based: these are raw images containing a single ext4 formatted file
  system.
</p>
</li>
<li>
<p>
ZFS subvolumes: these are technically bind mounts, but with managed storage,
  and thus allow resizing and snapshotting.
</p>
</li>
<li>
<p>
Directories: passing <span class="monospaced">size=0</span> triggers a special case where instead of a raw
  image a directory is created.
</p>
</li>
</ul></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">The special option syntax <span class="monospaced">STORAGE_ID:SIZE_IN_GB</span> for storage backed
mount point volumes will automatically allocate a volume of the specified size
on the specified storage. For example, calling</td>
</tr></tbody></table>
</div>
<div class="listingblock">
<div class="content monospaced">
<pre>pct set 100 -mp0 thin1:10,mp=/path/in/container</pre>
</div></div>
<div class="paragraph">
<p>will allocate a 10GB volume on the storage <span class="monospaced">thin1</span> and replace the volume ID
place holder <span class="monospaced">10</span> with the allocated volume ID, and setup the moutpoint in the
container at <span class="monospaced">/path/in/container</span></p></div>
</div>
<div class="sect4">
<h5 id="_bind_mount_points">Bind Mount Points
 <a class="headerlink" href="#_bind_mount_points" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Bind mounts allow you to access arbitrary directories from your Proxmox VE host
inside a container. Some potential use cases are:</p></div>
<div class="ulist"><ul>
<li>
<p>
Accessing your home directory in the guest
</p>
</li>
<li>
<p>
Accessing an USB device directory in the guest
</p>
</li>
<li>
<p>
Accessing an NFS mount from the host in the guest
</p>
</li>
</ul></div>
<div class="paragraph">
<p>Bind mounts are considered to not be managed by the storage subsystem, so you
cannot make snapshots or deal with quotas from inside the container. With
unprivileged containers you might run into permission problems caused by the
user mapping and cannot use ACLs.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">The contents of bind mount points are not backed up when using <span class="monospaced">vzdump</span>.</td>
</tr></tbody></table>
</div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">For security reasons, bind mounts should only be established using
source directories especially reserved for this purpose, e.g., a directory
hierarchy under <span class="monospaced">/mnt/bindmounts</span>. Never bind mount system directories like
<span class="monospaced">/</span>, <span class="monospaced">/var</span> or <span class="monospaced">/etc</span> into a container - this poses a great security risk.</td>
</tr></tbody></table>
</div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">The bind mount source path must not contain any symlinks.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>For example, to make the directory <span class="monospaced">/mnt/bindmounts/shared</span> accessible in the
container with ID <span class="monospaced">100</span> under the path <span class="monospaced">/shared</span>, add a configuration line such as:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>mp0: /mnt/bindmounts/shared,mp=/shared</pre>
</div></div>
<div class="paragraph">
<p>into <span class="monospaced">/etc/pve/lxc/100.conf</span>.</p></div>
<div class="paragraph">
<p>Or alternatively use the <span class="monospaced">pct</span> tool:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>pct set 100 -mp0 /mnt/bindmounts/shared,mp=/shared</pre>
</div></div>
<div class="paragraph">
<p>to achieve the same result.</p></div>
</div>
<div class="sect4">
<h5 id="_device_mount_points">Device Mount Points
 <a class="headerlink" href="#_device_mount_points" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Device mount points allow to mount block devices of the host directly into the
container. Similar to bind mounts, device mounts are not managed by Proxmox VE’s
storage subsystem, but the <span class="monospaced">quota</span> and <span class="monospaced">acl</span> options will be honored.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Device mount points should only be used under special circumstances. In
most cases a storage backed mount point offers the same performance and a lot
more features.</td>
</tr></tbody></table>
</div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">The contents of device mount points are not backed up when using
<span class="monospaced">vzdump</span>.</td>
</tr></tbody></table>
</div>
</div>
</div>
<div class="sect3">
<h4 id="pct_container_network">11.4.5. Network
 <a class="headerlink" href="#pct_container_network" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-create-ct-network.png">
<img src="images/screenshot/gui-create-ct-network.png" alt="screenshot/gui-create-ct-network.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>You can configure up to 10 network interfaces for a single container.
The corresponding options are called <span class="monospaced">net0</span> to <span class="monospaced">net9</span>, and they can contain the
following setting:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">net[n]</span>: <span class="monospaced">name=&lt;string&gt; [,bridge=&lt;bridge&gt;] [,firewall=&lt;1|0&gt;] [,gw=&lt;GatewayIPv4&gt;] [,gw6=&lt;GatewayIPv6&gt;] [,hwaddr=&lt;XX:XX:XX:XX:XX:XX&gt;] [,ip=&lt;(IPv4/CIDR|dhcp|manual)&gt;] [,ip6=&lt;(IPv6/CIDR|auto|dhcp|manual)&gt;] [,link_down=&lt;1|0&gt;] [,mtu=&lt;integer&gt;] [,rate=&lt;mbps&gt;] [,tag=&lt;integer&gt;] [,trunks=&lt;vlanid[;vlanid...]&gt;] [,type=&lt;veth&gt;]</span> 
</dt>
<dd>
<p>
Specifies network interfaces for the container.
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">bridge</span>=<span class="monospaced">&lt;bridge&gt;</span> 
</dt>
<dd>
<p>
Bridge to attach the network device to.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">firewall</span>=<span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Controls whether this interface’s firewall rules should be used.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">gw</span>=<span class="monospaced">&lt;GatewayIPv4&gt;</span> 
</dt>
<dd>
<p>
Default gateway for IPv4 traffic.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">gw6</span>=<span class="monospaced">&lt;GatewayIPv6&gt;</span> 
</dt>
<dd>
<p>
Default gateway for IPv6 traffic.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">hwaddr</span>=<span class="monospaced">&lt;XX:XX:XX:XX:XX:XX&gt;</span> 
</dt>
<dd>
<p>
A common MAC address with the I/G (Individual/Group) bit not set.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">ip</span>=<span class="monospaced">&lt;(IPv4/CIDR|dhcp|manual)&gt;</span> 
</dt>
<dd>
<p>
IPv4 address in CIDR format.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">ip6</span>=<span class="monospaced">&lt;(IPv6/CIDR|auto|dhcp|manual)&gt;</span> 
</dt>
<dd>
<p>
IPv6 address in CIDR format.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">link_down</span>=<span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Whether this interface should be disconnected (like pulling the plug).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mtu</span>=<span class="monospaced">&lt;integer&gt; (64 - 65535)</span> 
</dt>
<dd>
<p>
Maximum transfer unit of the interface. (lxc.network.mtu)
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">name</span>=<span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Name of the network device as seen from inside the container. (lxc.network.name)
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">rate</span>=<span class="monospaced">&lt;mbps&gt;</span> 
</dt>
<dd>
<p>
Apply rate limiting to the interface
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">tag</span>=<span class="monospaced">&lt;integer&gt; (1 - 4094)</span> 
</dt>
<dd>
<p>
VLAN tag for this interface.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">trunks</span>=<span class="monospaced">&lt;vlanid[;vlanid...]&gt;</span> 
</dt>
<dd>
<p>
VLAN ids to pass through the interface
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">type</span>=<span class="monospaced">&lt;veth&gt;</span> 
</dt>
<dd>
<p>
Network interface type.
</p>
</dd>
</dl></div>
</dd>
</dl></div>
</div>
<div class="sect3">
<h4 id="pct_startup_and_shutdown">11.4.6. Automatic Start and Shutdown of Containers
 <a class="headerlink" href="#pct_startup_and_shutdown" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>To automatically start a container when the host system boots, select the
option <em>Start at boot</em> in the <em>Options</em> panel of the container in the web
interface or run the following command:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># pct set CTID -onboot 1</pre>
</div></div>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-qemu-edit-start-order.png">
<img src="images/screenshot/gui-qemu-edit-start-order.png" alt="screenshot/gui-qemu-edit-start-order.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<div class="title">Start and Shutdown Order</div><p>If you want to fine tune the boot order of your containers, you can use the
following parameters:</p></div>
<div class="ulist"><ul>
<li>
<p>
<strong>Start/Shutdown order</strong>: Defines the start order priority. For example, set it
  to 1 if you want the CT to be the first to be started. (We use the reverse
  startup order for shutdown, so a container with a start order of 1 would be
  the last to be shut down)
</p>
</li>
<li>
<p>
<strong>Startup delay</strong>: Defines the interval between this container start and
  subsequent containers starts. For example, set it to 240 if you want to wait
  240 seconds before starting other containers.
</p>
</li>
<li>
<p>
<strong>Shutdown timeout</strong>: Defines the duration in seconds Proxmox VE should wait
  for the container to be offline after issuing a shutdown command.
  By default this value is set to 60, which means that Proxmox VE will issue a
  shutdown request, wait 60s for the machine to be offline, and if after 60s
  the machine is still online will notify that the shutdown action failed.
</p>
</li>
</ul></div>
<div class="paragraph">
<p>Please note that containers without a Start/Shutdown order parameter will
always start after those where the parameter is set, and this parameter only
makes sense between the machines running locally on a host, and not
cluster-wide.</p></div>
<div class="paragraph">
<p>If you require a delay between the host boot and the booting of the first
container, see the section on
<a href="#first_guest_boot_delay">Proxmox VE Node Management</a>.</p></div>
</div>
<div class="sect3">
<h4 id="_hookscripts_2">11.4.7. Hookscripts
 <a class="headerlink" href="#_hookscripts_2" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>You can add a hook script to CTs with the config property <span class="monospaced">hookscript</span>.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># pct set 100 -hookscript local:snippets/hookscript.pl</pre>
</div></div>
<div class="paragraph">
<p>It will be called during various phases of the guests lifetime.  For an example
and documentation see the example script under
<span class="monospaced">/usr/share/pve-docs/examples/guest-example-hookscript.pl</span>.</p></div>
</div>
</div>
<div class="sect2">
<h3 id="_security_considerations">
<span>11.5. Security Considerations</span>
 <a class="headerlink" href="#_security_considerations" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Containers use the kernel of the host system. This exposes an attack surface
for malicious users. In general, full virtual machines provide better
isolation. This should be considered if containers are provided to unknown or
untrusted people.</p></div>
<div class="paragraph">
<p>To reduce the attack surface, LXC uses many security features like AppArmor,
CGroups and kernel namespaces.</p></div>
<div class="sect3">
<h4 id="_apparmor">11.5.1. AppArmor
 <a class="headerlink" href="#_apparmor" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>AppArmor profiles are used to restrict access to possibly dangerous actions.
Some system calls, i.e. <span class="monospaced">mount</span>, are prohibited from execution.</p></div>
<div class="paragraph">
<p>To trace AppArmor activity, use:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># dmesg | grep apparmor</pre>
</div></div>
<div class="paragraph">
<p>Although it is not recommended, AppArmor can be disabled for a container. This
brings security risks with it. Some syscalls can lead to privilege escalation
when executed within a container if the system is misconfigured or if a LXC or
Linux Kernel vulnerability exists.</p></div>
<div class="paragraph">
<p>To disable AppArmor for a container, add the following line to the container
configuration file located at <span class="monospaced">/etc/pve/lxc/CTID.conf</span>:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>lxc.apparmor.profile = unconfined</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">Please note that this is not recommended for production use.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="pct_cgroup">11.5.2. Control Groups (<em>cgroup</em>)
 <a class="headerlink" href="#pct_cgroup" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p><em>cgroup</em> is a kernel
mechanism used to hierarchically organize processes and distribute system
resources.</p></div>
<div class="paragraph">
<p>The main resources controlled via <em>cgroups</em> are CPU time, memory and swap
limits, and access to device nodes. <em>cgroups</em> are also used to "freeze" a
container before taking snapshots.</p></div>
<div class="paragraph">
<p>There are 2 versions of <em>cgroups</em> currently available,
<a href="https://www.kernel.org/doc/html/v5.11/admin-guide/cgroup-v1/index.html">legacy</a>
and
<a href="https://www.kernel.org/doc/html/v5.11/admin-guide/cgroup-v2.html"><em>cgroupv2</em></a>.</p></div>
<div class="paragraph">
<p>Since Proxmox VE 7.0, the default is a pure <em>cgroupv2</em> environment. Previously a
"hybrid" setup was used, where resource control was mainly done in <em>cgroupv1</em>
with an additional <em>cgroupv2</em> controller which could take over some subsystems
via the <em>cgroup_no_v1</em> kernel command-line parameter. (See the
<a href="https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html">kernel
parameter documentation</a> for details.)</p></div>
<div class="sect4">
<h5 id="pct_cgroup_compat">CGroup Version Compatibility
 <a class="headerlink" href="#pct_cgroup_compat" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>The main difference between pure <em>cgroupv2</em> and the old hybrid environments
regarding Proxmox VE is that with <em>cgroupv2</em> memory and swap are now controlled
independently. The memory and swap settings for containers can map directly to
these values, whereas previously only the memory limit and the limit of the
<strong>sum</strong> of memory and swap could be limited.</p></div>
<div class="paragraph">
<p>Another important difference is that the <em>devices</em> controller is configured in a
completely different way. Because of this, file system quotas are currently not
supported in a pure <em>cgroupv2</em> environment.</p></div>
<div class="paragraph">
<p><em>cgroupv2</em> support by the container’s OS is needed to run in a pure <em>cgroupv2</em>
environment. Containers running <em>systemd</em> version 231 or newer support
<em>cgroupv2</em> <span class="footnote" data-note="this includes all newest major versions of container
templates shipped by Proxmox VE">[<a id="_footnoteref_49" href="#_footnote_49" title="View footnote" class="footnote">49</a>]</span>, as do containers not using <em>systemd</em> as init
system <span class="footnote" data-note="for example Alpine Linux">[<a id="_footnoteref_50" href="#_footnote_50" title="View footnote" class="footnote">50</a>]</span>.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">
<div class="paragraph">
<p>CentOS 7 and Ubuntu 16.10 are two prominent Linux distributions releases,
which have a <em>systemd</em> version that is too old to run in a <em>cgroupv2</em>
environment, you can either</p></div>
<div class="ulist"><ul>
<li>
<p>
Upgrade the whole distribution to a newer release. For the examples above, that
  could be Ubuntu 18.04 or 20.04, and CentOS 8 (or RHEL/CentOS derivatives like
  AlmaLinux or Rocky Linux). This has the benefit to get the newest bug and
  security fixes, often also new features, and moving the EOL date in the future.
</p>
</li>
<li>
<p>
Upgrade the Containers systemd version. If the distribution provides a
  backports repository this can be an easy and quick stop-gap measurement.
</p>
</li>
<li>
<p>
Move the container, or its services, to a Virtual Machine. Virtual Machines
  have a much less interaction with the host, that’s why one can install
  decades old OS versions just fine there.
</p>
</li>
<li>
<p>
Switch back to the legacy <em>cgroup</em> controller. Note that while it can be a
  valid solution, it’s not a permanent one. Starting from Proxmox VE 9.0, the legacy
  controller will not be supported anymore.
</p>
</li>
</ul></div>
</td>
</tr></tbody></table>
</div>
</div>
<div class="sect4">
<h5 id="pct_cgroup_change_version">Changing CGroup Version
 <a class="headerlink" href="#pct_cgroup_change_version" title="Permalink to this heading"></a>
</h5>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">If file system quotas are not required and all containers support <em>cgroupv2</em>,
it is recommended to stick to the new default.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>To switch back to the previous version the following kernel command-line
parameter can be used:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>systemd.unified_cgroup_hierarchy=0</pre>
</div></div>
<div class="paragraph">
<p>See <a href="#sysboot_edit_kernel_cmdline">this section</a> on editing the kernel boot
command line on where to add the parameter.</p></div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_guest_operating_system_configuration">
<span>11.6. Guest Operating System Configuration</span>
 <a class="headerlink" href="#_guest_operating_system_configuration" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Proxmox VE tries to detect the Linux distribution in the container, and modifies
some files. Here is a short list of things done at container startup:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
set /etc/hostname
</dt>
<dd>
<p>
to set the container name
</p>
</dd>
<dt class="hdlist1">
modify /etc/hosts
</dt>
<dd>
<p>
to allow lookup of the local hostname
</p>
</dd>
<dt class="hdlist1">
network setup
</dt>
<dd>
<p>
pass the complete network setup to the container
</p>
</dd>
<dt class="hdlist1">
configure DNS
</dt>
<dd>
<p>
pass information about DNS servers
</p>
</dd>
<dt class="hdlist1">
adapt the init system
</dt>
<dd>
<p>
for example, fix the number of spawned getty processes
</p>
</dd>
<dt class="hdlist1">
set the root password
</dt>
<dd>
<p>
when creating a new container
</p>
</dd>
<dt class="hdlist1">
rewrite ssh_host_keys
</dt>
<dd>
<p>
so that each container has unique keys
</p>
</dd>
<dt class="hdlist1">
randomize crontab
</dt>
<dd>
<p>
so that cron does not start at the same time on all containers
</p>
</dd>
</dl></div>
<div class="paragraph">
<p>Changes made by Proxmox VE are enclosed by comment markers:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># --- BEGIN PVE ---
&lt;data&gt;
# --- END PVE ---</pre>
</div></div>
<div class="paragraph">
<p>Those markers will be inserted at a reasonable location in the file. If such a
section already exists, it will be updated in place and will not be moved.</p></div>
<div class="paragraph">
<p>Modification of a file can be prevented by adding a <span class="monospaced">.pve-ignore.</span> file for it.
For instance, if the file <span class="monospaced">/etc/.pve-ignore.hosts</span> exists then the <span class="monospaced">/etc/hosts</span>
file will not be touched. This can be a simple empty file created via:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># touch /etc/.pve-ignore.hosts</pre>
</div></div>
<div class="paragraph">
<p>Most modifications are OS dependent, so they differ between different
distributions and versions. You can completely disable modifications by
manually setting the <span class="monospaced">ostype</span> to <span class="monospaced">unmanaged</span>.</p></div>
<div class="paragraph">
<p>OS type detection is done by testing for certain files inside the
container. Proxmox VE first checks the <span class="monospaced">/etc/os-release</span> file
<span class="footnote" data-note="/etc/os-release replaces the multitude of per-distribution
release files <a href=&quot;https://manpages.debian.org/stable/systemd/os-release.5.en.html&quot;>https://manpages.debian.org/stable/systemd/os-release.5.en.html</a>">[<a id="_footnoteref_51" href="#_footnote_51" title="View footnote" class="footnote">51</a>]</span>.
If that file is not present, or it does not contain a clearly recognizable
distribution identifier the following distribution specific release files are
checked.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
Ubuntu
</dt>
<dd>
<p>
inspect /etc/lsb-release (<span class="monospaced">DISTRIB_ID=Ubuntu</span>)
</p>
</dd>
<dt class="hdlist1">
Debian
</dt>
<dd>
<p>
test /etc/debian_version
</p>
</dd>
<dt class="hdlist1">
Fedora
</dt>
<dd>
<p>
test /etc/fedora-release
</p>
</dd>
<dt class="hdlist1">
RedHat or CentOS
</dt>
<dd>
<p>
test /etc/redhat-release
</p>
</dd>
<dt class="hdlist1">
ArchLinux
</dt>
<dd>
<p>
test /etc/arch-release
</p>
</dd>
<dt class="hdlist1">
Alpine
</dt>
<dd>
<p>
test /etc/alpine-release
</p>
</dd>
<dt class="hdlist1">
Gentoo
</dt>
<dd>
<p>
test /etc/gentoo-release
</p>
</dd>
</dl></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Container start fails if the configured <span class="monospaced">ostype</span> differs from the auto
detected type.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect2">
<h3 id="pct_container_storage">
<span>11.7. Container Storage</span>
 <a class="headerlink" href="#pct_container_storage" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>The Proxmox VE LXC container storage model is more flexible than traditional
container storage models. A container can have multiple mount points. This
makes it possible to use the best suited storage for each application.</p></div>
<div class="paragraph">
<p>For example the root file system of the container can be on slow and cheap
storage while the database can be on fast and distributed storage via a second
mount point. See section <a href="#pct_mount_points">Mount Points</a> for further
details.</p></div>
<div class="paragraph">
<p>Any storage type supported by the Proxmox VE storage library can be used. This means
that containers can be stored on local (for example <span class="monospaced">lvm</span>, <span class="monospaced">zfs</span> or directory),
shared external (like <span class="monospaced">iSCSI</span>, <span class="monospaced">NFS</span>) or even distributed storage systems like
Ceph. Advanced storage features like snapshots or clones can be used if the
underlying storage supports them. The <span class="monospaced">vzdump</span> backup tool can use snapshots to
provide consistent container backups.</p></div>
<div class="paragraph">
<p>Furthermore, local devices or local directories can be mounted directly using
<em>bind mounts</em>. This gives access to local resources inside a container with
practically zero overhead. Bind mounts can be used as an easy way to share data
between containers.</p></div>
<div class="sect3">
<h4 id="_fuse_mounts">11.7.1. FUSE Mounts
 <a class="headerlink" href="#_fuse_mounts" title="Permalink to this heading"></a>
</h4>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">Because of existing issues in the Linux kernel’s freezer subsystem the
usage of FUSE mounts inside a container is strongly advised against, as
containers need to be frozen for suspend or snapshot mode backups.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>If FUSE mounts cannot be replaced by other mounting mechanisms or storage
technologies, it is possible to establish the FUSE mount on the Proxmox host
and use a bind mount point to make it accessible inside the container.</p></div>
</div>
<div class="sect3">
<h4 id="_using_quotas_inside_containers">11.7.2. Using Quotas Inside Containers
 <a class="headerlink" href="#_using_quotas_inside_containers" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Quotas allow to set limits inside a container for the amount of disk space that
each user can use.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">This currently requires the use of legacy <em>cgroups</em>.</td>
</tr></tbody></table>
</div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">This only works on ext4 image based storage types and currently only
works with privileged containers.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>Activating the <span class="monospaced">quota</span> option causes the following mount options to be used for
a mount point:
<span class="monospaced">usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0</span></p></div>
<div class="paragraph">
<p>This allows quotas to be used like on any other system. You can initialize the
<span class="monospaced">/aquota.user</span> and <span class="monospaced">/aquota.group</span> files by running:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># quotacheck -cmug /
# quotaon /</pre>
</div></div>
<div class="paragraph">
<p>Then edit the quotas using the <span class="monospaced">edquota</span> command. Refer to the documentation of
the distribution running inside the container for details.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">You need to run the above commands for every mount point by passing the
mount point’s path instead of just <span class="monospaced">/</span>.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="_using_acls_inside_containers">11.7.3. Using ACLs Inside Containers
 <a class="headerlink" href="#_using_acls_inside_containers" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The standard Posix <strong>A</strong>ccess <strong>C</strong>ontrol <strong>L</strong>ists are also available inside
containers. ACLs allow you to set more detailed file ownership than the
traditional user/group/others model.</p></div>
</div>
<div class="sect3">
<h4 id="_backup_of_container_mount_points">11.7.4. Backup of Container mount points
 <a class="headerlink" href="#_backup_of_container_mount_points" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>To include a mount point in backups, enable the <span class="monospaced">backup</span> option for it in the
container configuration. For an existing mount point <span class="monospaced">mp0</span></p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>mp0: guests:subvol-100-disk-1,mp=/root/files,size=8G</pre>
</div></div>
<div class="paragraph">
<p>add <span class="monospaced">backup=1</span> to enable it.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>mp0: guests:subvol-100-disk-1,mp=/root/files,size=8G,backup=1</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">When creating a new mount point in the GUI, this option is enabled by
default.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>To disable backups for a mount point, add <span class="monospaced">backup=0</span> in the way described
above, or uncheck the <strong>Backup</strong> checkbox on the GUI.</p></div>
</div>
<div class="sect3">
<h4 id="_replication_of_containers_mount_points">11.7.5. Replication of Containers mount points
 <a class="headerlink" href="#_replication_of_containers_mount_points" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>By default, additional mount points are replicated when the Root Disk is
replicated. If you want the Proxmox VE storage replication mechanism to skip a mount
point, you can set the <strong>Skip replication</strong> option for that mount point.
As of Proxmox VE 5.0, replication requires a storage of type <span class="monospaced">zfspool</span>. Adding a
mount point to a different type of storage when the container has replication
configured requires to have <strong>Skip replication</strong> enabled for that mount point.</p></div>
</div>
</div>
<div class="sect2">
<h3 id="_backup_and_restore">
<span>11.8. Backup and Restore</span>
 <a class="headerlink" href="#_backup_and_restore" title="Permalink to this heading"></a>
</h3>
<div class="sect3">
<h4 id="_container_backup">11.8.1. Container Backup
 <a class="headerlink" href="#_container_backup" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>It is possible to use the <span class="monospaced">vzdump</span> tool for container backup. Please refer to
the <span class="monospaced">vzdump</span> manual page for details.</p></div>
</div>
<div class="sect3">
<h4 id="_restoring_container_backups">11.8.2. Restoring Container Backups
 <a class="headerlink" href="#_restoring_container_backups" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Restoring container backups made with <span class="monospaced">vzdump</span> is possible using the <span class="monospaced">pct
restore</span> command. By default, <span class="monospaced">pct restore</span> will attempt to restore as much of
the backed up container configuration as possible. It is possible to override
the backed up configuration by manually setting container options on the
command line (see the <span class="monospaced">pct</span> manual page for details).</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content"><span class="monospaced">pvesm extractconfig</span> can be used to view the backed up configuration
contained in a vzdump archive.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>There are two basic restore modes, only differing by their handling of mount
points:</p></div>
<div class="sect4">
<h5 id="_8220_simple_8221_restore_mode">“Simple” Restore Mode
 <a class="headerlink" href="#_8220_simple_8221_restore_mode" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>If neither the <span class="monospaced">rootfs</span> parameter nor any of the optional <span class="monospaced">mpX</span> parameters are
explicitly set, the mount point configuration from the backed up configuration
file is restored using the following steps:</p></div>
<div class="olist arabic"><ol class="arabic">
<li>
<p>
Extract mount points and their options from backup
</p>
</li>
<li>
<p>
Create volumes for storage backed mount points on the storage provided with
  the <span class="monospaced">storage</span> parameter (default: <span class="monospaced">local</span>).
</p>
</li>
<li>
<p>
Extract files from backup archive
</p>
</li>
<li>
<p>
Add bind and device mount points to restored configuration (limited to root
  user)
</p>
</li>
</ol></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Since bind and device mount points are never backed up, no files are
restored in the last step, but only the configuration options. The assumption
is that such mount points are either backed up with another mechanism (e.g.,
NFS space that is bind mounted into many containers), or not intended to be
backed up at all.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>This simple mode is also used by the container restore operations in the web
interface.</p></div>
</div>
<div class="sect4">
<h5 id="_8220_advanced_8221_restore_mode">“Advanced” Restore Mode
 <a class="headerlink" href="#_8220_advanced_8221_restore_mode" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>By setting the <span class="monospaced">rootfs</span> parameter (and optionally, any combination of <span class="monospaced">mpX</span>
parameters), the <span class="monospaced">pct restore</span> command is automatically switched into an
advanced mode. This advanced mode completely ignores the <span class="monospaced">rootfs</span> and <span class="monospaced">mpX</span>
configuration options contained in the backup archive, and instead only uses
the options explicitly provided as parameters.</p></div>
<div class="paragraph">
<p>This mode allows flexible configuration of mount point settings at restore
time, for example:</p></div>
<div class="ulist"><ul>
<li>
<p>
Set target storages, volume sizes and other options for each mount point
  individually
</p>
</li>
<li>
<p>
Redistribute backed up files according to new mount point scheme
</p>
</li>
<li>
<p>
Restore to device and/or bind mount points (limited to root user)
</p>
</li>
</ul></div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_managing_containers_with_span_class_monospaced_pct_span">
<span>11.9. Managing Containers with <span class="monospaced">pct</span></span>
 <a class="headerlink" href="#_managing_containers_with_span_class_monospaced_pct_span" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>The “Proxmox Container Toolkit” (<span class="monospaced">pct</span>) is the command-line tool to manage
Proxmox VE containers. It enables you to create or destroy containers, as well as
control the container execution (start, stop, reboot, migrate, etc.). It can be
used to set parameters in the config file of a container, for example the
network configuration or memory limits.</p></div>
<div class="sect3">
<h4 id="_cli_usage_examples_2">11.9.1. CLI Usage Examples
 <a class="headerlink" href="#_cli_usage_examples_2" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Create a container based on a Debian template (provided you have already
downloaded the template via the web interface)</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># pct create 100 /var/lib/vz/template/cache/debian-10.0-standard_10.0-1_amd64.tar.gz</pre>
</div></div>
<div class="paragraph">
<p>Start container 100</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># pct start 100</pre>
</div></div>
<div class="paragraph">
<p>Start a login session via getty</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># pct console 100</pre>
</div></div>
<div class="paragraph">
<p>Enter the LXC namespace and run a shell as root user</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># pct enter 100</pre>
</div></div>
<div class="paragraph">
<p>Display the configuration</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># pct config 100</pre>
</div></div>
<div class="paragraph">
<p>Add a network interface called <span class="monospaced">eth0</span>, bridged to the host bridge <span class="monospaced">vmbr0</span>, set
the address and gateway, while it’s running</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># pct set 100 -net0 name=eth0,bridge=vmbr0,ip=192.168.15.147/24,gw=192.168.15.1</pre>
</div></div>
<div class="paragraph">
<p>Reduce the memory of the container to 512MB</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># pct set 100 -memory 512</pre>
</div></div>
<div class="paragraph">
<p>Destroying a container always removes it from Access Control Lists and it always
removes the firewall configuration of the container. You have to activate
<em>--purge</em>, if you want to additionally remove the container from replication jobs,
backup jobs and HA resource configurations.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># pct destroy 100 --purge</pre>
</div></div>
<div class="paragraph">
<p>Move a mount point volume to a different storage.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># pct move-volume 100 mp0 other-storage</pre>
</div></div>
<div class="paragraph">
<p>Reassign a volume to a different CT. This will remove the volume <span class="monospaced">mp0</span> from
the source CT and attaches it as <span class="monospaced">mp1</span> to the target CT. In the background
the volume is being renamed so that the name matches the new owner.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>#  pct move-volume 100 mp0 --target-vmid 200 --target-volume mp1</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_obtaining_debugging_logs">11.9.2. Obtaining Debugging Logs
 <a class="headerlink" href="#_obtaining_debugging_logs" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>In case <span class="monospaced">pct start</span> is unable to start a specific container, it might be
helpful to collect debugging output by passing the <span class="monospaced">--debug</span> flag (replace <span class="monospaced">CTID</span> with
the container’s CTID):</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># pct start CTID --debug</pre>
</div></div>
<div class="paragraph">
<p>Alternatively, you can use the following <span class="monospaced">lxc-start</span> command, which will save
the debug log to the file specified by the <span class="monospaced">-o</span> output option:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># lxc-start -n CTID -F -l DEBUG -o /tmp/lxc-CTID.log</pre>
</div></div>
<div class="paragraph">
<p>This command will attempt to start the container in foreground mode, to stop
the container run <span class="monospaced">pct shutdown CTID</span> or <span class="monospaced">pct stop CTID</span> in a second terminal.</p></div>
<div class="paragraph">
<p>The collected debug log is written to <span class="monospaced">/tmp/lxc-CTID.log</span>.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">If you have changed the container’s configuration since the last start
attempt with <span class="monospaced">pct start</span>, you need to run <span class="monospaced">pct start</span> at least once to also
update the configuration used by <span class="monospaced">lxc-start</span>.</td>
</tr></tbody></table>
</div>
</div>
</div>
<div class="sect2">
<h3 id="pct_migration">
<span>11.10. Migration</span>
 <a class="headerlink" href="#pct_migration" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>If you have a cluster, you can migrate your Containers with</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># pct migrate &lt;ctid&gt; &lt;target&gt;</pre>
</div></div>
<div class="paragraph">
<p>This works as long as your Container is offline. If it has local volumes or
mount points defined, the migration will copy the content over the network to
the target host if the same storage is defined there.</p></div>
<div class="paragraph">
<p>Running containers cannot live-migrated due to technical limitations. You can
do a restart migration, which shuts down, moves and then starts a container
again on the target node. As containers are very lightweight, this results
normally only in a downtime of some hundreds of milliseconds.</p></div>
<div class="paragraph">
<p>A restart migration can be done through the web interface or by using the
<span class="monospaced">--restart</span> flag with the <span class="monospaced">pct migrate</span> command.</p></div>
<div class="paragraph">
<p>A restart migration will shut down the Container and kill it after the
specified timeout (the default is 180 seconds). Then it will migrate the
Container like an offline migration and when finished, it starts the Container
on the target node.</p></div>
</div>
<div class="sect2">
<h3 id="pct_configuration">
<span>11.11. Configuration</span>
 <a class="headerlink" href="#pct_configuration" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>The <span class="monospaced">/etc/pve/lxc/&lt;CTID&gt;.conf</span> file stores container configuration, where
<span class="monospaced">&lt;CTID&gt;</span> is the numeric ID of the given container. Like all other files stored
inside <span class="monospaced">/etc/pve/</span>, they get automatically replicated to all other cluster
nodes.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">CTIDs &lt; 100 are reserved for internal purposes, and CTIDs need to be
unique cluster wide.</td>
</tr></tbody></table>
</div>
<div class="listingblock">
<div class="title">Example Container Configuration</div>
<div class="content monospaced">
<pre>ostype: debian
arch: amd64
hostname: www
memory: 512
swap: 512
net0: bridge=vmbr0,hwaddr=66:64:66:64:64:36,ip=dhcp,name=eth0,type=veth
rootfs: local:107/vm-107-disk-1.raw,size=7G</pre>
</div></div>
<div class="paragraph">
<p>The configuration files are simple text files. You can edit them using a normal
text editor, for example, <span class="monospaced">vi</span> or <span class="monospaced">nano</span>.
This is sometimes useful to do small corrections, but keep in mind that you
need to restart the container to apply such changes.</p></div>
<div class="paragraph">
<p>For that reason, it is usually better to use the <span class="monospaced">pct</span> command to generate and
modify those files, or do the whole thing using the GUI.
Our toolkit is smart enough to instantaneously apply most changes to running
containers. This feature is called “hot plug”, and there is no need to restart
the container in that case.</p></div>
<div class="paragraph">
<p>In cases where a change cannot be hot-plugged, it will be registered as a
pending change (shown in red color in the GUI).
They will only be applied after rebooting the container.</p></div>
<div class="sect3">
<h4 id="_file_format_2">11.11.1. File Format
 <a class="headerlink" href="#_file_format_2" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The container configuration file uses a simple colon separated key/value
format. Each line has the following format:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># this is a comment
OPTION: value</pre>
</div></div>
<div class="paragraph">
<p>Blank lines in those files are ignored, and lines starting with a <span class="monospaced">#</span> character
are treated as comments and are also ignored.</p></div>
<div class="paragraph">
<p>It is possible to add low-level, LXC style configuration directly, for example:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>lxc.init_cmd: /sbin/my_own_init</pre>
</div></div>
<div class="paragraph">
<p>or</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>lxc.init_cmd = /sbin/my_own_init</pre>
</div></div>
<div class="paragraph">
<p>The settings are passed directly to the LXC low-level tools.</p></div>
</div>
<div class="sect3">
<h4 id="pct_snapshots">11.11.2. Snapshots
 <a class="headerlink" href="#pct_snapshots" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>When you create a snapshot, <span class="monospaced">pct</span> stores the configuration at snapshot time
into a separate snapshot section within the same configuration file. For
example, after creating a snapshot called “testsnapshot”, your configuration
file will look like this:</p></div>
<div class="listingblock">
<div class="title">Container configuration with snapshot</div>
<div class="content monospaced">
<pre>memory: 512
swap: 512
parent: testsnaphot
...

[testsnaphot]
memory: 512
swap: 512
snaptime: 1457170803
...</pre>
</div></div>
<div class="paragraph">
<p>There are a few snapshot related properties like <span class="monospaced">parent</span> and <span class="monospaced">snaptime</span>. The
<span class="monospaced">parent</span> property is used to store the parent/child relationship between
snapshots. <span class="monospaced">snaptime</span> is the snapshot creation time stamp (Unix epoch).</p></div>
</div>
<div class="sect3">
<h4 id="pct_options">11.11.3. Options
 <a class="headerlink" href="#pct_options" title="Permalink to this heading"></a>
</h4>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">arch</span>: <span class="monospaced">&lt;amd64 | arm64 | armhf | i386 | riscv32 | riscv64&gt;</span> (<em>default =</em> <span class="monospaced">amd64</span>)
</dt>
<dd>
<p>
OS architecture type.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">cmode</span>: <span class="monospaced">&lt;console | shell | tty&gt;</span> (<em>default =</em> <span class="monospaced">tty</span>)
</dt>
<dd>
<p>
Console mode. By default, the console command tries to open a connection to one of the available tty devices. By setting cmode to <em>console</em> it tries to attach to /dev/console instead. If you set cmode to <em>shell</em>, it simply invokes a shell inside the container (no login).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">console</span>: <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Attach a console device (/dev/console) to the container.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">cores</span>: <span class="monospaced">&lt;integer&gt; (1 - 8192)</span> 
</dt>
<dd>
<p>
The number of cores assigned to the container. A container can use all available cores by default.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">cpulimit</span>: <span class="monospaced">&lt;number&gt; (0 - 8192)</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Limit of CPU usage.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">If the computer has 2 CPUs, it has a total of <em>2</em> CPU time. Value <em>0</em> indicates no CPU limit.</td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">cpuunits</span>: <span class="monospaced">&lt;integer&gt; (0 - 500000)</span> (<em>default =</em> <span class="monospaced">cgroup v1: 1024, cgroup v2: 100</span>)
</dt>
<dd>
<p>
CPU weight for a container. Argument is used in the kernel fair scheduler. The larger the number is, the more CPU time this container gets. Number is relative to the weights of all the other running guests.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">debug</span>: <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Try to be more verbose. For now this only enables debug log-level on start.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">description</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Description for the Container. Shown in the web-interface CT’s summary. This is saved as comment inside the configuration file.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">dev[n]</span>: <span class="monospaced">[[path=]&lt;Path&gt;] [,gid=&lt;integer&gt;] [,mode=&lt;Octal access mode&gt;] [,uid=&lt;integer&gt;]</span> 
</dt>
<dd>
<p>
Device to pass through to the container
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">gid</span>=<span class="monospaced">&lt;integer&gt; (0 - N)</span> 
</dt>
<dd>
<p>
Group ID to be assigned to the device node
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mode</span>=<span class="monospaced">&lt;Octal access mode&gt;</span> 
</dt>
<dd>
<p>
Access mode to be set on the device node
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">path</span>=<span class="monospaced">&lt;Path&gt;</span> 
</dt>
<dd>
<p>
Path to the device to pass through to the container
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">uid</span>=<span class="monospaced">&lt;integer&gt; (0 - N)</span> 
</dt>
<dd>
<p>
User ID to be assigned to the device node
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">features</span>: <span class="monospaced">[force_rw_sys=&lt;1|0&gt;] [,fuse=&lt;1|0&gt;] [,keyctl=&lt;1|0&gt;] [,mknod=&lt;1|0&gt;] [,mount=&lt;fstype;fstype;...&gt;] [,nesting=&lt;1|0&gt;]</span> 
</dt>
<dd>
<p>
Allow containers access to advanced features.
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">force_rw_sys</span>=<span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Mount /sys in unprivileged containers as <span class="monospaced">rw</span> instead of <span class="monospaced">mixed</span>. This can break networking under newer (&gt;= v245) systemd-network use.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">fuse</span>=<span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Allow using <em>fuse</em> file systems in a container. Note that interactions between fuse and the freezer cgroup can potentially cause I/O deadlocks.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">keyctl</span>=<span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
For unprivileged containers only: Allow the use of the keyctl() system call. This is required to use docker inside a container. By default unprivileged containers will see this system call as non-existent. This is mostly a workaround for systemd-networkd, as it will treat it as a fatal error when some keyctl() operations are denied by the kernel due to lacking permissions. Essentially, you can choose between running systemd-networkd or docker.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mknod</span>=<span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Allow unprivileged containers to use mknod() to add certain device nodes. This requires a kernel with seccomp trap to user space support (5.3 or newer). This is experimental.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mount</span>=<span class="monospaced">&lt;fstype;fstype;...&gt;</span> 
</dt>
<dd>
<p>
Allow mounting file systems of specific types. This should be a list of file system types as used with the mount command. Note that this can have negative effects on the container’s security. With access to a loop device, mounting a file can circumvent the mknod permission of the devices cgroup, mounting an NFS file system can block the host’s I/O completely and prevent it from rebooting, etc.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">nesting</span>=<span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Allow nesting. Best used with unprivileged containers with additional id mapping. Note that this will expose procfs and sysfs contents of the host to the guest.
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">hookscript</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Script that will be exectued during various steps in the containers lifetime.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">hostname</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Set a host name for the container.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">lock</span>: <span class="monospaced">&lt;backup | create | destroyed | disk | fstrim | migrate | mounted | rollback | snapshot | snapshot-delete&gt;</span> 
</dt>
<dd>
<p>
Lock/unlock the container.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">memory</span>: <span class="monospaced">&lt;integer&gt; (16 - N)</span> (<em>default =</em> <span class="monospaced">512</span>)
</dt>
<dd>
<p>
Amount of RAM for the container in MB.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mp[n]</span>: <span class="monospaced">[volume=]&lt;volume&gt; ,mp=&lt;Path&gt; [,acl=&lt;1|0&gt;] [,backup=&lt;1|0&gt;] [,mountoptions=&lt;opt[;opt...]&gt;] [,quota=&lt;1|0&gt;] [,replicate=&lt;1|0&gt;] [,ro=&lt;1|0&gt;] [,shared=&lt;1|0&gt;] [,size=&lt;DiskSize&gt;]</span> 
</dt>
<dd>
<p>
Use volume as container mount point. Use the special syntax STORAGE_ID:SIZE_IN_GiB to allocate a new volume.
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">acl</span>=<span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Explicitly enable or disable ACL support.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">backup</span>=<span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Whether to include the mount point in backups (only used for volume mount points).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mountoptions</span>=<span class="monospaced">&lt;opt[;opt...]&gt;</span> 
</dt>
<dd>
<p>
Extra mount options for rootfs/mps.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mp</span>=<span class="monospaced">&lt;Path&gt;</span> 
</dt>
<dd>
<p>
Path to the mount point as seen from inside the container.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Must not contain any symlinks for security reasons.</td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">quota</span>=<span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Enable user quotas inside the container (not supported with zfs subvolumes)
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">replicate</span>=<span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Will include this volume to a storage replica job.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">ro</span>=<span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Read-only mount point
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">shared</span>=<span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Mark this non-volume mount point as available on all nodes.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">This option does not share the mount point automatically, it assumes it is shared already!</td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">size</span>=<span class="monospaced">&lt;DiskSize&gt;</span> 
</dt>
<dd>
<p>
Volume size (read only value).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">volume</span>=<span class="monospaced">&lt;volume&gt;</span> 
</dt>
<dd>
<p>
Volume, device or directory to mount into the container.
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">nameserver</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Sets DNS server IP address for a container. Create will automatically use the setting from the host if you neither set searchdomain nor nameserver.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">net[n]</span>: <span class="monospaced">name=&lt;string&gt; [,bridge=&lt;bridge&gt;] [,firewall=&lt;1|0&gt;] [,gw=&lt;GatewayIPv4&gt;] [,gw6=&lt;GatewayIPv6&gt;] [,hwaddr=&lt;XX:XX:XX:XX:XX:XX&gt;] [,ip=&lt;(IPv4/CIDR|dhcp|manual)&gt;] [,ip6=&lt;(IPv6/CIDR|auto|dhcp|manual)&gt;] [,link_down=&lt;1|0&gt;] [,mtu=&lt;integer&gt;] [,rate=&lt;mbps&gt;] [,tag=&lt;integer&gt;] [,trunks=&lt;vlanid[;vlanid...]&gt;] [,type=&lt;veth&gt;]</span> 
</dt>
<dd>
<p>
Specifies network interfaces for the container.
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">bridge</span>=<span class="monospaced">&lt;bridge&gt;</span> 
</dt>
<dd>
<p>
Bridge to attach the network device to.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">firewall</span>=<span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Controls whether this interface’s firewall rules should be used.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">gw</span>=<span class="monospaced">&lt;GatewayIPv4&gt;</span> 
</dt>
<dd>
<p>
Default gateway for IPv4 traffic.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">gw6</span>=<span class="monospaced">&lt;GatewayIPv6&gt;</span> 
</dt>
<dd>
<p>
Default gateway for IPv6 traffic.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">hwaddr</span>=<span class="monospaced">&lt;XX:XX:XX:XX:XX:XX&gt;</span> 
</dt>
<dd>
<p>
A common MAC address with the I/G (Individual/Group) bit not set.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">ip</span>=<span class="monospaced">&lt;(IPv4/CIDR|dhcp|manual)&gt;</span> 
</dt>
<dd>
<p>
IPv4 address in CIDR format.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">ip6</span>=<span class="monospaced">&lt;(IPv6/CIDR|auto|dhcp|manual)&gt;</span> 
</dt>
<dd>
<p>
IPv6 address in CIDR format.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">link_down</span>=<span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Whether this interface should be disconnected (like pulling the plug).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mtu</span>=<span class="monospaced">&lt;integer&gt; (64 - 65535)</span> 
</dt>
<dd>
<p>
Maximum transfer unit of the interface. (lxc.network.mtu)
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">name</span>=<span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Name of the network device as seen from inside the container. (lxc.network.name)
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">rate</span>=<span class="monospaced">&lt;mbps&gt;</span> 
</dt>
<dd>
<p>
Apply rate limiting to the interface
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">tag</span>=<span class="monospaced">&lt;integer&gt; (1 - 4094)</span> 
</dt>
<dd>
<p>
VLAN tag for this interface.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">trunks</span>=<span class="monospaced">&lt;vlanid[;vlanid...]&gt;</span> 
</dt>
<dd>
<p>
VLAN ids to pass through the interface
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">type</span>=<span class="monospaced">&lt;veth&gt;</span> 
</dt>
<dd>
<p>
Network interface type.
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">onboot</span>: <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Specifies whether a container will be started during system bootup.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">ostype</span>: <span class="monospaced">&lt;alpine | archlinux | centos | debian | devuan | fedora | gentoo | nixos | opensuse | ubuntu | unmanaged&gt;</span> 
</dt>
<dd>
<p>
OS type. This is used to setup configuration inside the container, and corresponds to lxc setup scripts in /usr/share/lxc/config/&lt;ostype&gt;.common.conf. Value <em>unmanaged</em> can be used to skip and OS specific setup.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">protection</span>: <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Sets the protection flag of the container. This will prevent the CT or CT’s disk remove/update operation.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">rootfs</span>: <span class="monospaced">[volume=]&lt;volume&gt; [,acl=&lt;1|0&gt;] [,mountoptions=&lt;opt[;opt...]&gt;] [,quota=&lt;1|0&gt;] [,replicate=&lt;1|0&gt;] [,ro=&lt;1|0&gt;] [,shared=&lt;1|0&gt;] [,size=&lt;DiskSize&gt;]</span> 
</dt>
<dd>
<p>
Use volume as container root.
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">acl</span>=<span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Explicitly enable or disable ACL support.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mountoptions</span>=<span class="monospaced">&lt;opt[;opt...]&gt;</span> 
</dt>
<dd>
<p>
Extra mount options for rootfs/mps.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">quota</span>=<span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Enable user quotas inside the container (not supported with zfs subvolumes)
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">replicate</span>=<span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Will include this volume to a storage replica job.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">ro</span>=<span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Read-only mount point
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">shared</span>=<span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Mark this non-volume mount point as available on all nodes.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">This option does not share the mount point automatically, it assumes it is shared already!</td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">size</span>=<span class="monospaced">&lt;DiskSize&gt;</span> 
</dt>
<dd>
<p>
Volume size (read only value).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">volume</span>=<span class="monospaced">&lt;volume&gt;</span> 
</dt>
<dd>
<p>
Volume, device or directory to mount into the container.
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">searchdomain</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Sets DNS search domains for a container. Create will automatically use the setting from the host if you neither set searchdomain nor nameserver.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">startup</span>: `[[order=]\d+] [,up=\d+] [,down=\d+] ` 
</dt>
<dd>
<p>
Startup and shutdown behavior. Order is a non-negative number defining the general startup order. Shutdown in done with reverse ordering. Additionally you can set the <em>up</em> or <em>down</em> delay in seconds, which specifies a delay to wait before the next VM is started or stopped.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">swap</span>: <span class="monospaced">&lt;integer&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">512</span>)
</dt>
<dd>
<p>
Amount of SWAP for the container in MB.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">tags</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Tags of the Container. This is only meta information.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">template</span>: <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Enable/disable Template.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">timezone</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Time zone to use in the container. If option isn’t set, then nothing will be done. Can be set to <em>host</em> to match the host time zone, or an arbitrary time zone option from /usr/share/zoneinfo/zone.tab
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">tty</span>: <span class="monospaced">&lt;integer&gt; (0 - 6)</span> (<em>default =</em> <span class="monospaced">2</span>)
</dt>
<dd>
<p>
Specify the number of tty available to the container
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">unprivileged</span>: <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Makes the container run as unprivileged user. (Should not be modified manually.)
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">unused[n]</span>: <span class="monospaced">[volume=]&lt;volume&gt;</span> 
</dt>
<dd>
<p>
Reference to unused volumes. This is used internally, and should not be modified manually.
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">volume</span>=<span class="monospaced">&lt;volume&gt;</span> 
</dt>
<dd>
<p>
The volume that is not used currently.
</p>
</dd>
</dl></div>
</dd>
</dl></div>
</div>
</div>
<div class="sect2">
<h3 id="_locks_2">
<span>11.12. Locks</span>
 <a class="headerlink" href="#_locks_2" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Container migrations, snapshots and backups (<span class="monospaced">vzdump</span>) set a lock to prevent
incompatible concurrent actions on the affected container. Sometimes you need
to remove such a lock manually (e.g., after a power failure).</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># pct unlock &lt;CTID&gt;</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Caution" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKdUlEQVRoge1Ze1AV1x3+zt279wEi
DWCYGzRVktqa1MEmmtbWR22ncXxUrTDWV/5IqG2wUAUfmUwSkk6V+EAJCEQC6figwZBqZtRxqukf
tebRZBpFG1S0hiRErwiaKK977+6eX//YPXt37+UiIJlMZnpmdnb33PP4vt97z2VEhG9yc3zdAO60
/Z/A192+8QScX8Wifr+fWltbzffU1FT4fD72Vew15ASampqovr4eBw8eNPvGjRuHzMxMmj9//tCT
IKIhu958801yuVwEoNdr48aNNJT7EdHQEdi/fz/JshwTvLiKioqGlMSQLFJfX0+MMRvQsWNSqXLz
H2jez78fRWLTpk1DRuKOF6irqyOn02kDOP8XGdTz+VFSP91Hatu79NRvZ0SR2LJly5CQuKPJtbW1
JEmSDdivHp1AwctHSblYSsqFElLOF5PavIvWZU+LIrF169Y7JjHoibt3744CnzXrIQpeOUbKhVJS
mkpIPV9MyrnNpJwtIuVSNa15/MdRJLZt23ZHJBjRwIu5N954gxYvXgzOudm3aO5E1L5SBNZ1Hoxr
INLAyH6HJxVP/WkXtu9+37ZeSUkJVq9ePagQO+BMXFNTEwV+ybxHsPeVIjg6zwOkgSWNhuOuUUBC
KhxJY0AggDSgpwWbn1mO1csftq1ZUFCA0tLSQZXFA0pkVVVVlJOTA6vWsmY9hF07N0AS4OU48LgR
WLO+FPFxboxMS8G5hlPYmPMI4jwM1P0Jtj67DJyrKHvtNADdjPPz8+FwOCgvL29gmuivrVVUVESF
yqxZD1PoyjFSL+7QHbapmJTml+n9EzVRtl676ZekNKwl5VQ+KSfzKHR2E+UutodYxhiVl5cPyCf6
ZUJlZWWUm5trk/wTi6bgtVdf1M2Gq2DEwUgDNBWhUDBqjfSRwwGoIOEP3c3Y/uxirFw0zibMvLw8
VFZW9tucbkugoqKCVq1aZQO/YslUVJU+D9bVBIDroLgK4hygEDweybYGY8AD6d8yftfASL+j+2O8
9NwiPJk51kYiNzcXO3fu7BeJPgkUFxdTbm6ure/JZdNRub0QrOMcGFcBrgLgAHEAGqD2ID7OZZsz
Ji0RcR6HAV4DkaqPJxXouoSywkz8buF9NhIrV65EVVXVbUnEJHDixAlat26drS9n+U+xo/hZMMNh
yZA+uAbAIKOGMCJlmG3eA/clAYaJEamGBlRdC6QBXZewo3AhVswfbSORk5ODDz/8sE8SvRLw+/1U
XV0dBb5MgOe6LUNIEqpOwgCUlOCESw4HuAfG3KX7B2kAGaTJ0Jro77yI8ucX4DfzRtlIHD58GH6/
PyaJXgk0NjZi79695vuUiffrkr91Vt8cHAwc4FZJGiZEKqDcwsi0ZHP+d749POy8MS7iGtBxARWF
8/CDsQnm3BdeeAHWj6N+EYhspSVbgI5zYYkZGmAwzIEbwA0tUM9NfO+7aeb89HuGRQBWLcSNZxj3
zvOYMzk5NpjBEHi99mXD5lXdjqGDJh42h7CEOXhnK3448X5z/shUb1jSpIEM8yFuzOUcjAsNcnR0
KXdG4MEHH8TEiRPN9z0H3sfRE41m+GNcs9i0LjkGTTcrUoGem5g7/V5z/ohElw4O+jxGYQ3qJqmB
jN/+cbINtcfCJvPYY4+hr3qtVwI+n49lZmaa71fbbuH3hX/FycbPQKSCoIIMQEQcehjVo5Gw8/F3
3cSvH03H2HuHY1icQzcvYWKGIMiITMwQwjtnruPpqk9w/ZZq7p2RkYEJEyaAYrCIWY36/X7KyMhA
W1ub2XfP3Qmo3ZKFaQ+P0gmYIdSoOkU4lWSwxGRAdqHL/xm86k27w0blAw3/PncD6yo/xrsfdZj7
paSkoLGxESNGjAgDZsxWK8X0AZ/Px44fP460tLAzXrnWgUX5r+PYuxf1mG/Gf80sofWy2YuWbg/+
uONtPLHhPRw+0WIHL8hbwG95rcUGPiEhAYcOHUJSUhI456YZRWqiz+8BIqLm5mZMmjQJN27cMPsT
E9zYtWE25k4bYzh3mABIBRwOHPgPx+LH/6yDiXPi6pHZcEpk+I5qmtGpCzdQecCPPUevmevLsowj
R45gypQpcDgc5mVowKaJmBoQTEePHo2GhgaMGxcuum52BLHsqUOoP3oWgLB9Sz7QgpgxJoThw/SS
IutnaXBKZEpcgD998Uvs+3ubDbzT6URNTQ0mTZoEVVWhaRo459A0TeCy4etVA+JH8RvnHK2trZgz
Zw5Onz5tjnPJDrxSOAPLZt1nSVIqwDmIVJw6dx1nLt3A3J/cjZThkhE29cvf3o2KA59ja91lcz3G
GDZv3oylS5fC7XbD6XTC6XRCkiRIktS7Jm5HQLDXNA3Xr19HVlYWPvjgA4vEGLav+RHmTxsJX4rb
8AsO4kbOIG6GzTD4HvzlmB/PVH9q2zc/Px/Z2dnwer1wu92QZRkulwuSJEGWZZMEY8wkEGVCVvCC
AOcciqIgPj4e+/btw9SpU83xqkpYteU9HHm7Bf62rnC2RjikisTFSMPV9m4ceucannvVDn7p0qVY
uHAhgsEggsEgQqFQlAlZnVm0PjOxIKFpGogIqqrC5XKhuroa06dPt4wDcl78F+qPNeNKe6cRoTjs
ZYOKK+3dON7wBQrKm2H5pMbMmTOxZMkSBAIB9PT0oKenB4FAAIqiQFEUU4hRRyq3I2AlIZ4553A6
nSgvL8fs2bNtY9eVNaDub58YmhDZWpe+v70bJ5tu4clt/4WihqU4efJkrFixAkRkAg4Gg1AUBaqq
muDF/pGtX7WQNXeIZ0mSUFRUhAULFtjGPl3ZiJdev4QrbZ16zUQqrrZ342JLN5ZvaEJ3ICz68ePH
IycnB5qmmZIW4EOhUJTZWEKo+Rx1KsEYY9Zk4XA4IEmSKXnOuRkVJElCYWEhvF4v6urqzDVK9n2M
zu4Qnl5+L9q/7MHltgBWbr9kA5+eno7c3FzIsoyI5GrTeCzgosU8VmGMmSGLiEwSsixH2WNBQQEA
2EhUH/wc7330BaaOH479/2zHtS/CFabP58OaNWvg9Xp1EE4nZFmGLMtwu93wer1wuVy2ECpJko2I
iTNWJrZGI+HEwi6FisUVCAQQCASwZ88eVFZW9lk9JicnY/369fD5fPB4PCbo+Ph4k4TH44HL5YLX
64XX6zXH9EKE9aUBRkTEGLNJP1Kd4nI4HMjOzkZ8fDyKi4ttJ3eiJSYmYu3atRg1apQJzuv1wuPx
wOPxwOFwmEBFHnA6nXC5XGYSiywlbns2SnoDANN0hMNZnS0UCpkaOnPmDN566y20tLSY2ktLS8OM
GTOQkpICt9uNuLg4M2EJ8CJZSZJkZmFBwkrAWpH2+3BXEBEkRIhTVdVMOCL5qKqKy5cvQ1EUaJoG
VVWRnKx/Jlpt3eVy2QCKu9C68AFr9o0spwd0Oh1ZYggCIlNaiQktiXdhZgKQABcp3ci7LMtM7B0J
fsAErJoAYItG1mdr+hfvVlACtHDIXkploYXbHvQOikDEe6/ZmjFmAhf9ZvJxOk2Qol88W4kYz32S
GNQfHLG+TyMTkAVEr88xQfVnkBg7GAKixSIykDYQsL21/wFkW/B5QqT9lwAAAABJRU5ErkJggg==">
</td>
<td class="content">Only do this if you are sure the action which set the lock is no
longer running.</td>
</tr></tbody></table>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="chapter_pvesdn">
12. Software-Defined Network
 <a class="headerlink" href="#chapter_pvesdn" title="Permalink to this heading"></a>
</h2>
<div class="sectionbody">
<div class="paragraph">
<p>The <strong>S</strong>oftware-<strong>D</strong>efined <strong>N</strong>etwork (SDN) feature in Proxmox VE enables the
creation of virtual zones and networks (VNets). This functionality simplifies
advanced networking configurations and multitenancy setup.</p></div>
<div class="sect2">
<h3 id="pvesdn_overview">
<span>12.1. Introduction</span>
 <a class="headerlink" href="#pvesdn_overview" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>The Proxmox VE SDN allows for separation and fine-grained control of virtual guest
networks, using flexible, software-controlled configurations.</p></div>
<div class="paragraph">
<p>Separation is managed through <strong>zones</strong>, virtual networks (<strong>VNets</strong>), and
<strong>subnets</strong>.  A zone is its own virtually separated network area.  A VNet is a
virtual network that belongs to a zone. A subnet is an IP range inside a VNet.</p></div>
<div class="paragraph">
<p>Depending on the type of the zone, the network behaves differently and offers
specific features, advantages, and limitations.</p></div>
<div class="paragraph">
<p>Use cases for SDN range from an isolated private network on each individual node
to complex overlay networks across multiple PVE clusters on different locations.</p></div>
<div class="paragraph">
<p>After configuring an VNet in the cluster-wide datacenter SDN administration
interface, it is available as a common Linux bridge, locally on each node, to be
assigned to VMs and Containers.</p></div>
</div>
<div class="sect2">
<h3 id="pvesdn_support_status">
<span>12.2. Support Status</span>
 <a class="headerlink" href="#pvesdn_support_status" title="Permalink to this heading"></a>
</h3>
<div class="sect3">
<h4 id="_history">12.2.1. History
 <a class="headerlink" href="#_history" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The Proxmox VE SDN stack has been available as an experimental feature since 2019 and
has been continuously improved and tested by many developers and users.
With its integration into the web interface in Proxmox VE 6.2, a significant
milestone towards broader integration was achieved.
During the Proxmox VE 7 release cycle, numerous improvements and features were added.
Based on user feedback, it became apparent that the fundamental design choices
and their implementation were quite sound and stable. Consequently, labeling it
as ‘experimental’ did not do justice to the state of the SDN stack.
For Proxmox VE 8, a decision was made to lay the groundwork for full integration of
the SDN feature by elevating the management of networks and interfaces to a core
component in the Proxmox VE access control stack.
In Proxmox VE 8.1, two major milestones were achieved: firstly, DHCP integration was
added to the IP address management (IPAM) feature, and secondly, the SDN
integration is now installed by default.</p></div>
</div>
<div class="sect3">
<h4 id="_current_status">12.2.2. Current Status
 <a class="headerlink" href="#_current_status" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The current support status for the various layers of our SDN installation is as
follows:</p></div>
<div class="ulist"><ul>
<li>
<p>
Core SDN, which includes VNet management and its integration with the Proxmox VE
  stack, is fully supported.
</p>
</li>
<li>
<p>
IPAM, including DHCP management for virtual guests, is in tech preview.
</p>
</li>
<li>
<p>
Complex routing via FRRouting and controller integration are in tech preview.
</p>
</li>
</ul></div>
</div>
</div>
<div class="sect2">
<h3 id="pvesdn_installation">
<span>12.3. Installation</span>
 <a class="headerlink" href="#pvesdn_installation" title="Permalink to this heading"></a>
</h3>
<div class="sect3">
<h4 id="_sdn_core">12.3.1. SDN Core
 <a class="headerlink" href="#_sdn_core" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Since Proxmox VE 8.1 the core Software-Defined Network (SDN) packages are installed
by default.</p></div>
<div class="paragraph">
<p>If you upgrade from an older version, you need to install the
<span class="monospaced">libpve-network-perl</span> package on every node:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>apt update
apt install libpve-network-perl</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Proxmox VE version 7.0 and above have the <span class="monospaced">ifupdown2</span> package installed by
default. If you originally installed your system with an older version, you need
to explicitly install the <span class="monospaced">ifupdown2</span> package.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>After installation, you need to ensure that the following line is present at the
end of the <span class="monospaced">/etc/network/interfaces</span> configuration file on all nodes, so that
the SDN configuration gets included and activated.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>source /etc/network/interfaces.d/*</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="pvesdn_install_dhcp_ipam">12.3.2. DHCP IPAM
 <a class="headerlink" href="#pvesdn_install_dhcp_ipam" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The DHCP integration into the built-in <em>PVE</em> IP Address Management stack
currently uses <span class="monospaced">dnsmasq</span> for giving out DHCP leases. This is currently opt-in.</p></div>
<div class="paragraph">
<p>To use that feature you need to install the <span class="monospaced">dnsmasq</span> package on every node:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>apt update
apt install dnsmasq
# disable default instance
systemctl disable --now dnsmasq</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="pvesdn_install_frrouting">12.3.3. FRRouting
 <a class="headerlink" href="#pvesdn_install_frrouting" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The Proxmox VE SDN stack uses the <a href="https://frrouting.org/">FRRouting</a> project for
advanced setups. This is currently opt-in.</p></div>
<div class="paragraph">
<p>To use the SDN routing integration you need to install the <span class="monospaced">frr-pythontools</span>
package on all nodes:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>apt update
apt install frr-pythontools</pre>
</div></div>
</div>
</div>
<div class="sect2">
<h3 id="pvesdn_main_configuration">
<span>12.4. Configuration Overview</span>
 <a class="headerlink" href="#pvesdn_main_configuration" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Configuration is done at the web UI at datacenter level, separated into the
following sections:</p></div>
<div class="ulist"><ul>
<li>
<p>
SDN:: Here you get an overview of the current active SDN state, and you can
  apply all pending changes to the whole cluster.
</p>
</li>
<li>
<p>
<a href="#pvesdn_config_zone">Zones</a>: Create and manage the virtually separated
  network zones
</p>
</li>
<li>
<p>
<a href="#pvesdn_config_vnet">VNets</a> VNets: Create virtual network bridges and
  manage subnets
</p>
</li>
</ul></div>
<div class="paragraph">
<p>The Options category allows adding and managing additional services to be used
in your SDN setup.</p></div>
<div class="ulist"><ul>
<li>
<p>
<a href="#pvesdn_config_controllers">Controllers</a>: For controlling layer 3 routing
  in complex setups
</p>
</li>
<li>
<p>
DHCP: Define a DHCP server for a zone that automatically allocates IPs for
  guests in the IPAM and leases them to the guests via DHCP.
</p>
</li>
<li>
<p>
<a href="#pvesdn_config_ipam">IPAM</a>: Enables external for IP address management for
  guests
</p>
</li>
<li>
<p>
<a href="#pvesdn_config_dns">DNS</a>: Define a DNS server integration for registering
  virtual guests' hostname and IP addresses
</p>
</li>
</ul></div>
</div>
<div class="sect2">
<h3 id="pvesdn_tech_and_config_overview">
<span>12.5. Technology &amp; Configuration</span>
 <a class="headerlink" href="#pvesdn_tech_and_config_overview" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>The Proxmox VE Software-Defined Network implementation uses standard Linux networking
as much as possible. The reason for this is that modern Linux networking
provides almost all needs for a feature full SDN implementation and avoids adding
external dependencies and reduces the overall amount of components that can
break.</p></div>
<div class="paragraph">
<p>The Proxmox VE SDN configurations are located in <span class="monospaced">/etc/pve/sdn</span>, which is shared with
all other cluster nodes through the Proxmox VE <a href="#chapter_pmxcfs">configuration file system</a>.
Those configurations get translated to the respective configuration formats of
the tools that manage the underlying network stack (for example <span class="monospaced">ifupdown2</span> or
<span class="monospaced">frr</span>).</p></div>
<div class="paragraph">
<p>New changes are not immediately applied but recorded as pending first. You can
then apply a set of different changes all at once in the main <em>SDN</em> overview
panel on the web interface. This system allows to roll-out various changes as
single atomic one.</p></div>
<div class="paragraph">
<p>The SDN tracks the rolled-out state through the <em>.running-config</em> and <em>.version</em>
files located in <em>/etc/pve/sdn</em>.</p></div>
</div>
<div class="sect2">
<h3 id="pvesdn_config_zone">
<span>12.6. Zones</span>
 <a class="headerlink" href="#pvesdn_config_zone" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>A zone defines a virtually separated network. Zones are restricted to
specific nodes and assigned permissions, in order to restrict users to a certain
zone and its contained VNets.</p></div>
<div class="paragraph">
<p>Different technologies can be used for separation:</p></div>
<div class="ulist"><ul>
<li>
<p>
Simple: Isolated Bridge. A simple layer 3 routing bridge (NAT)
</p>
</li>
<li>
<p>
VLAN: Virtual LANs are the classic method of subdividing a LAN
</p>
</li>
<li>
<p>
QinQ: Stacked VLAN (formally known as <span class="monospaced">IEEE 802.1ad</span>)
</p>
</li>
<li>
<p>
VXLAN: Layer 2 VXLAN network via a UDP tunnel
</p>
</li>
<li>
<p>
EVPN (BGP EVPN): VXLAN with BGP to establish Layer 3 routing
</p>
</li>
</ul></div>
<div class="sect3">
<h4 id="pvesdn_config_common_options">12.6.1. Common Options
 <a class="headerlink" href="#pvesdn_config_common_options" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The following options are available for all zone types:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
Nodes
</dt>
<dd>
<p>
The nodes which the zone and associated VNets should be deployed on.
</p>
</dd>
<dt class="hdlist1">
IPAM
</dt>
<dd>
<p>
Use an IP Address Management (IPAM) tool to manage IPs in the
  zone. Optional, defaults to <span class="monospaced">pve</span>.
</p>
</dd>
<dt class="hdlist1">
DNS
</dt>
<dd>
<p>
DNS API server. Optional.
</p>
</dd>
<dt class="hdlist1">
ReverseDNS
</dt>
<dd>
<p>
Reverse DNS API server. Optional.
</p>
</dd>
<dt class="hdlist1">
DNSZone
</dt>
<dd>
<p>
DNS domain name. Used to register hostnames, such as
  <span class="monospaced">&lt;hostname&gt;.&lt;domain&gt;</span>. The DNS zone must already exist on the DNS server. Optional.
</p>
</dd>
</dl></div>
</div>
<div class="sect3">
<h4 id="pvesdn_zone_plugin_simple">12.6.2. Simple Zones
 <a class="headerlink" href="#pvesdn_zone_plugin_simple" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>This is the simplest plugin. It will create an isolated VNet bridge.  This
bridge is not linked to a physical interface, and VM traffic is only local on
each the node.
It can be used in NAT or routed setups.</p></div>
</div>
<div class="sect3">
<h4 id="pvesdn_zone_plugin_vlan">12.6.3. VLAN Zones
 <a class="headerlink" href="#pvesdn_zone_plugin_vlan" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The VLAN plugin uses an existing local Linux or OVS bridge to connect to the
node’s physical interface.  It uses VLAN tagging defined in the VNet to isolate
the network segments.  This allows connectivity of VMs between different nodes.</p></div>
<div class="paragraph">
<p>VLAN zone configuration options:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
Bridge
</dt>
<dd>
<p>
The local bridge or OVS switch, already configured on <strong>each</strong> node that
  allows node-to-node connection.
</p>
</dd>
</dl></div>
</div>
<div class="sect3">
<h4 id="pvesdn_zone_plugin_qinq">12.6.4. QinQ Zones
 <a class="headerlink" href="#pvesdn_zone_plugin_qinq" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>QinQ also known as VLAN stacking, that uses multiple layers of VLAN tags for
isolation.  The QinQ zone defines the outer VLAN tag (the <em>Service VLAN</em>)
whereas the inner VLAN tag is defined by the VNet.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Your physical network switches must support stacked VLANs for this
configuration.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>QinQ zone configuration options:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
Bridge
</dt>
<dd>
<p>
A local, VLAN-aware bridge that is already configured on each local
  node
</p>
</dd>
<dt class="hdlist1">
Service VLAN
</dt>
<dd>
<p>
The main VLAN tag of this zone
</p>
</dd>
<dt class="hdlist1">
Service VLAN Protocol
</dt>
<dd>
<p>
Allows you to choose between an 802.1q (default) or
  802.1ad service VLAN type.
</p>
</dd>
<dt class="hdlist1">
MTU
</dt>
<dd>
<p>
Due to the double stacking of tags, you need 4 more bytes for QinQ VLANs.
  For example, you must reduce the MTU to <span class="monospaced">1496</span> if you physical interface MTU is
  <span class="monospaced">1500</span>.
</p>
</dd>
</dl></div>
</div>
<div class="sect3">
<h4 id="pvesdn_zone_plugin_vxlan">12.6.5. VXLAN Zones
 <a class="headerlink" href="#pvesdn_zone_plugin_vxlan" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The VXLAN plugin establishes a tunnel (overlay) on top of an existing network
(underlay).  This encapsulates layer 2 Ethernet frames within layer 4 UDP
datagrams using the default destination port <span class="monospaced">4789</span>.</p></div>
<div class="paragraph">
<p>You have to configure the underlay network yourself to enable UDP connectivity
between all peers.</p></div>
<div class="paragraph">
<p>You can, for example, create a VXLAN overlay network on top of public internet,
appearing to the VMs as if they share the same local Layer 2 network.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">VXLAN on its own does does not provide any encryption. When joining
  multiple sites via VXLAN, make sure to establish a secure connection between
  the site, for example by using a site-to-site VPN.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>VXLAN zone configuration options:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
Peers Address List
</dt>
<dd>
<p>
A list of IP addresses of each node in the VXLAN zone. This
  can be external nodes reachable at this IP address.
  All nodes in the cluster need to be mentioned here.
</p>
</dd>
<dt class="hdlist1">
MTU
</dt>
<dd>
<p>
Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes
  lower than the outgoing physical interface.
</p>
</dd>
</dl></div>
</div>
<div class="sect3">
<h4 id="pvesdn_zone_plugin_evpn">12.6.6. EVPN Zones
 <a class="headerlink" href="#pvesdn_zone_plugin_evpn" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The EVPN zone creates a routable Layer 3 network, capable of spanning across
multiple clusters. This is achieved by establishing a VPN and utilizing BGP as
the routing protocol.</p></div>
<div class="paragraph">
<p>The VNet of EVPN can have an anycast IP address and/or MAC address. The bridge
IP is the same on each node, meaning a virtual guest can use this address as
gateway.</p></div>
<div class="paragraph">
<p>Routing can work across VNets from different zones through a VRF (Virtual
Routing and Forwarding) interface.</p></div>
<div class="paragraph">
<p>EVPN zone configuration options:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
VRF VXLAN ID
</dt>
<dd>
<p>
A VXLAN-ID used for dedicated routing interconnect between VNets.
  It must be different than the VXLAN-ID of the VNets.
</p>
</dd>
<dt class="hdlist1">
Controller
</dt>
<dd>
<p>
The EVPN-controller to use for this zone. (See controller plugins
  section).
</p>
</dd>
<dt class="hdlist1">
VNet MAC Address
</dt>
<dd>
<p>
Anycast MAC address that gets assigned to all VNets in this
  zone.  Will be auto-generated if not defined.
</p>
</dd>
<dt class="hdlist1">
Exit Nodes
</dt>
<dd>
<p>
Nodes that shall be configured as exit gateways from the EVPN
  network, through the real network.  The configured nodes will announce a
  default route in the EVPN network.  Optional.
</p>
</dd>
<dt class="hdlist1">
Primary Exit Node
</dt>
<dd>
<p>
If you use multiple exit nodes, force traffic through this
  primary exit node, instead of load-balancing on all nodes.  Optional but
  necessary if you want to use SNAT or if your upstream router doesn’t support
  ECMP.
</p>
</dd>
<dt class="hdlist1">
Exit Nodes Local Routing
</dt>
<dd>
<p>
This is a special option if you need to reach a VM/CT
  service from an exit node. (By default, the exit nodes only allow forwarding
  traffic between real network and EVPN network).  Optional.
</p>
</dd>
<dt class="hdlist1">
Advertise Subnets
</dt>
<dd>
<p>
Announce the full subnet in the EVPN network.
  If you have silent VMs/CTs (for example, if you have multiple IPs and the
  anycast gateway doesn’t see traffic from theses IPs, the IP addresses won’t be
  able to be reached inside the EVPN network).  Optional.
</p>
</dd>
<dt class="hdlist1">
Disable ARP ND Suppression
</dt>
<dd>
<p>
Don’t suppress ARP or ND (Neighbor Discovery)
  packets.  This is required if you use floating IPs in your VMs (IP and MAC
  addresses are being moved between systems).  Optional.
</p>
</dd>
<dt class="hdlist1">
Route-target Import
</dt>
<dd>
<p>
Allows you to import a list of external EVPN route
  targets. Used for cross-DC or different EVPN network interconnects.  Optional.
</p>
</dd>
<dt class="hdlist1">
MTU
</dt>
<dd>
<p>
Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes
  less than the maximal MTU of the outgoing physical interface.  Optional,
  defaults to 1450.
</p>
</dd>
</dl></div>
</div>
</div>
<div class="sect2">
<h3 id="pvesdn_config_vnet">
<span>12.7. VNets</span>
 <a class="headerlink" href="#pvesdn_config_vnet" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>After creating a virtual network (VNet) through the SDN GUI, a local network
interface with the same name is available on each node. To connect a guest to the
VNet, assign the interface to the guest and set the IP address accordingly.</p></div>
<div class="paragraph">
<p>Depending on the zone, these options have different meanings and are explained
in the respective zone section in this document.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">In the current state, some options may have no effect or won’t work in
certain zones.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>VNet configuration options:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
ID
</dt>
<dd>
<p>
An up to 8 character ID to identify a VNet
</p>
</dd>
<dt class="hdlist1">
Comment
</dt>
<dd>
<p>
More descriptive identifier. Assigned as an alias on the interface. Optional
</p>
</dd>
<dt class="hdlist1">
Zone
</dt>
<dd>
<p>
The associated zone for this VNet
</p>
</dd>
<dt class="hdlist1">
Tag
</dt>
<dd>
<p>
The unique VLAN or VXLAN ID
</p>
</dd>
<dt class="hdlist1">
VLAN Aware
</dt>
<dd>
<p>
Enables vlan-aware option on the interface, enabling configuration
  in the guest.
</p>
</dd>
</dl></div>
</div>
<div class="sect2">
<h3 id="pvesdn_config_subnet">
<span>12.8. Subnets</span>
 <a class="headerlink" href="#pvesdn_config_subnet" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>A subnet define a specific IP range, described by the CIDR network address.
Each VNet, can have one or more subnets.</p></div>
<div class="paragraph">
<p>A subnet can be used to:</p></div>
<div class="ulist"><ul>
<li>
<p>
Restrict the IP addresses you can define on a specific VNet
</p>
</li>
<li>
<p>
Assign routes/gateways on a VNet in layer 3 zones
</p>
</li>
<li>
<p>
Enable SNAT on a VNet in layer 3 zones
</p>
</li>
<li>
<p>
Auto assign IPs on virtual guests (VM or CT) through IPAM plugins
</p>
</li>
<li>
<p>
DNS registration through DNS plugins
</p>
</li>
</ul></div>
<div class="paragraph">
<p>If an IPAM server is associated with the subnet zone, the subnet prefix will be
automatically registered in the IPAM.</p></div>
<div class="paragraph">
<p>Subnet configuration options:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
ID
</dt>
<dd>
<p>
A CIDR network address, for example 10.0.0.0/8
</p>
</dd>
<dt class="hdlist1">
Gateway
</dt>
<dd>
<p>
The IP address of the network’s default gateway. On layer 3 zones
  (Simple/EVPN plugins), it will be deployed on the VNet.
</p>
</dd>
<dt class="hdlist1">
SNAT
</dt>
<dd>
<p>
Enable Source NAT which allows VMs from inside a
  VNet to connect to the outside network by forwarding the packets to the nodes
  outgoing interface. On EVPN zones, forwarding is done on EVPN gateway-nodes.
  Optional.
</p>
</dd>
<dt class="hdlist1">
DNS Zone Prefix
</dt>
<dd>
<p>
Add a prefix to the domain registration, like
  &lt;hostname&gt;.prefix.&lt;domain&gt;  Optional.
</p>
</dd>
</dl></div>
</div>
<div class="sect2">
<h3 id="pvesdn_config_controllers">
<span>12.9. Controllers</span>
 <a class="headerlink" href="#pvesdn_config_controllers" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Some zones implement a separated control and data plane that require an external
controller to manage the VNet’s control plane.</p></div>
<div class="paragraph">
<p>Currently, only the <span class="monospaced">EVPN</span> zone requires an external controller.</p></div>
<div class="sect3">
<h4 id="pvesdn_controller_plugin_evpn">12.9.1. EVPN Controller
 <a class="headerlink" href="#pvesdn_controller_plugin_evpn" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The <span class="monospaced">EVPN</span>, zone requires an external controller to manage the control plane.
The EVPN controller plugin configures the Free Range Routing (frr) router.</p></div>
<div class="paragraph">
<p>To enable the EVPN controller, you need to install frr on every node that shall
participate in the EVPN zone.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>apt install frr frr-pythontools</pre>
</div></div>
<div class="paragraph">
<p>EVPN controller configuration options:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
ASN #
</dt>
<dd>
<p>
A unique BGP ASN number. It’s highly recommended to use a private ASN
  number (64512 – 65534, 4200000000 – 4294967294), as otherwise you could end up
  breaking global routing by mistake.
</p>
</dd>
<dt class="hdlist1">
Peers
</dt>
<dd>
<p>
An IP list of all nodes that are part of the EVPN zone.  (could also be
  external nodes or route reflector servers)
</p>
</dd>
</dl></div>
</div>
<div class="sect3">
<h4 id="pvesdn_controller_plugin_BGP">12.9.2. BGP Controller
 <a class="headerlink" href="#pvesdn_controller_plugin_BGP" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The BGP controller is not used directly by a zone.
You can use it to configure FRR to manage BGP peers.</p></div>
<div class="paragraph">
<p>For BGP-EVPN, it can be used to define a different ASN by node, so doing EBGP.
It can also be used to export EVPN routes to an external BGP peer.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">By default, for a simple full mesh EVPN, you don’t need to define a BGP
controller.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>BGP controller configuration options:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
Node
</dt>
<dd>
<p>
The node of this BGP controller
</p>
</dd>
<dt class="hdlist1">
ASN #
</dt>
<dd>
<p>
A unique BGP ASN number. It’s highly recommended to use a private ASN
  number in the range (64512 - 65534) or (4200000000 - 4294967294), as otherwise
  you could break global routing by mistake.
</p>
</dd>
<dt class="hdlist1">
Peer
</dt>
<dd>
<p>
A list of peer IP addresses you want to communicate with using the
  underlying BGP network.
</p>
</dd>
<dt class="hdlist1">
EBGP
</dt>
<dd>
<p>
If your peer’s remote-AS is different, this enables EBGP.
</p>
</dd>
<dt class="hdlist1">
Loopback Interface
</dt>
<dd>
<p>
Use a loopback or dummy interface as the source of the EVPN network
  (for multipath).
</p>
</dd>
<dt class="hdlist1">
ebgp-mutltihop
</dt>
<dd>
<p>
Increase the number of hops to reach peers, in case they are
  not directly connected or they use loopback.
</p>
</dd>
<dt class="hdlist1">
bgp-multipath-as-path-relax
</dt>
<dd>
<p>
Allow ECMP if your peers have different ASN.
</p>
</dd>
</dl></div>
</div>
<div class="sect3">
<h4 id="pvesdn_controller_plugin_ISIS">12.9.3. ISIS Controller
 <a class="headerlink" href="#pvesdn_controller_plugin_ISIS" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The ISIS controller is not used directly by a zone.
You can use it to configure FRR to export EVPN routes to an ISIS domain.</p></div>
<div class="paragraph">
<p>ISIS controller configuration options:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
Node
</dt>
<dd>
<p>
The node of this ISIS controller.
</p>
</dd>
<dt class="hdlist1">
Domain
</dt>
<dd>
<p>
A unique ISIS domain.
</p>
</dd>
<dt class="hdlist1">
Network Entity Title
</dt>
<dd>
<p>
A Unique ISIS network address that identifies this node.
</p>
</dd>
<dt class="hdlist1">
Interfaces
</dt>
<dd>
<p>
A list of physical interface(s) used by ISIS.
</p>
</dd>
<dt class="hdlist1">
Loopback
</dt>
<dd>
<p>
Use a loopback or dummy interface as the source of the EVPN network
  (for multipath).
</p>
</dd>
</dl></div>
</div>
</div>
<div class="sect2">
<h3 id="pvesdn_config_ipam">
<span>12.10. IPAM</span>
 <a class="headerlink" href="#pvesdn_config_ipam" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>IP Address Management (IPAM) tools manage the IP addresses of clients on the
network. SDN in Proxmox VE uses IPAM for example to find free IP addresses for new
guests.</p></div>
<div class="paragraph">
<p>A single IPAM instance can be associated with one or more zones.</p></div>
<div class="sect3">
<h4 id="pvesdn_ipam_plugin_pveipam">12.10.1. PVE IPAM Plugin
 <a class="headerlink" href="#pvesdn_ipam_plugin_pveipam" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The default built-in IPAM for your Proxmox VE cluster.</p></div>
<div class="paragraph">
<p>You can inspect the current status of the PVE IPAM Plugin via the IPAM panel in
the SDN section of the datacenter configuration. This UI can be used to create,
update and delete IP mappings. This is particularly convenient in conjunction
with the <a href="#pvesdn_config_dhcp">DHCP feature</a>.</p></div>
<div class="paragraph">
<p>If you are using DHCP, you can use the IPAM panel to create or edit leases for
specific VMs, which enables you to change the IPs allocated via DHCP. When
editing an IP of a VM that is using DHCP you must make sure to force the guest
to acquire a new DHCP leases. This can usually be done by reloading the network
stack of the guest or rebooting it.</p></div>
</div>
<div class="sect3">
<h4 id="pvesdn_ipam_plugin_netbox">12.10.2. NetBox IPAM Plugin
 <a class="headerlink" href="#pvesdn_ipam_plugin_netbox" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p><a href="https://github.com/netbox-community/netbox">NetBox</a> is an open-source IP
Address Management (IPAM) and datacenter infrastructure management (DCIM) tool.</p></div>
<div class="paragraph">
<p>To integrate NetBox with Proxmox VE SDN, create an API token in NetBox as described
here: <a href="https://docs.netbox.dev/en/stable/integrations/rest-api/#tokens">https://docs.netbox.dev/en/stable/integrations/rest-api/#tokens</a></p></div>
<div class="paragraph">
<p>The NetBox configuration properties are:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
URL
</dt>
<dd>
<p>
The NetBox REST API endpoint: <span class="monospaced">http://yournetbox.domain.com/api</span>
</p>
</dd>
<dt class="hdlist1">
Token
</dt>
<dd>
<p>
An API access token
</p>
</dd>
</dl></div>
</div>
<div class="sect3">
<h4 id="pvesdn_ipam_plugin_phpipam">12.10.3. phpIPAM Plugin
 <a class="headerlink" href="#pvesdn_ipam_plugin_phpipam" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>In <a href="https://phpipam.net/">phpIPAM</a> you need to create an "application" and add
an API token with admin privileges to the application.</p></div>
<div class="paragraph">
<p>The phpIPAM configuration properties are:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
URL
</dt>
<dd>
<p>
The REST-API endpoint: <span class="monospaced">http://phpipam.domain.com/api/&lt;appname&gt;/</span>
</p>
</dd>
<dt class="hdlist1">
Token
</dt>
<dd>
<p>
An API access token
</p>
</dd>
<dt class="hdlist1">
Section
</dt>
<dd>
<p>
An integer ID. Sections are a group of subnets in phpIPAM. Default
  installations use <span class="monospaced">sectionid=1</span> for customers.
</p>
</dd>
</dl></div>
</div>
</div>
<div class="sect2">
<h3 id="pvesdn_config_dns">
<span>12.11. DNS</span>
 <a class="headerlink" href="#pvesdn_config_dns" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>The DNS plugin in Proxmox VE SDN is used to define a DNS API server for registration
of your hostname and IP address. A DNS configuration is associated with one or
more zones, to provide DNS registration for all the subnet IPs configured for
a zone.</p></div>
<div class="sect3">
<h4 id="pvesdn_dns_plugin_powerdns">12.11.1. PowerDNS Plugin
 <a class="headerlink" href="#pvesdn_dns_plugin_powerdns" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p><a href="https://doc.powerdns.com/authoritative/http-api/index.html">https://doc.powerdns.com/authoritative/http-api/index.html</a></p></div>
<div class="paragraph">
<p>You need to enable the web server and the API in your PowerDNS config:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>api=yes
api-key=arandomgeneratedstring
webserver=yes
webserver-port=8081</pre>
</div></div>
<div class="paragraph">
<p>The PowerDNS configuration options are:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
url
</dt>
<dd>
<p>
The REST API endpoint: <a href="http://yourpowerdnserver.domain.com:8081/api/v1/servers/localhost">http://yourpowerdnserver.domain.com:8081/api/v1/servers/localhost</a>
</p>
</dd>
<dt class="hdlist1">
key
</dt>
<dd>
<p>
An API access key
</p>
</dd>
<dt class="hdlist1">
ttl
</dt>
<dd>
<p>
The default TTL for records
</p>
</dd>
</dl></div>
</div>
</div>
<div class="sect2">
<h3 id="pvesdn_config_dhcp">
<span>12.12. DHCP</span>
 <a class="headerlink" href="#pvesdn_config_dhcp" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>The DHCP plugin in Proxmox VE SDN can be used to automatically deploy a DHCP server
for a Zone. It provides DHCP for all Subnets in a Zone that have a DHCP range
configured. Currently the only available backend plugin for DHCP is the dnsmasq
plugin.</p></div>
<div class="paragraph">
<p>The DHCP plugin works by allocating an IP in the IPAM plugin configured in the
Zone when adding a new network interface to a VM/CT. You can find more
information on how to configure an IPAM in the
<a href="#pvesdn_config_ipam">respective section of our documentation</a>.</p></div>
<div class="paragraph">
<p>When the VM starts, a mapping for the MAC address and IP gets created in the DHCP
plugin of the zone. When the network interfaces is removed or the VM/CT are
destroyed, then the entry in the IPAM and the DHCP server are deleted as well.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Some features (adding/editing/removing IP mappings) are currently only
available when using the <a href="#pvesdn_ipam_plugin_pveipam">PVE IPAM plugin</a>.</td>
</tr></tbody></table>
</div>
<div class="sect3">
<h4 id="_configuration_15">12.12.1. Configuration
 <a class="headerlink" href="#_configuration_15" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>You can enable automatic DHCP for a zone in the Web UI via the Zones panel and
enabling DHCP in the advanced options of a zone.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Currently only Simple Zones have support for automatic DHCP</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>After automatic DHCP has been enabled for a Zone, DHCP Ranges need to be
configured for the subnets in a Zone. In order to that, go to the Vnets panel and
select the Subnet for which you want to configure DHCP ranges. In the edit
dialogue you can configure DHCP ranges in the respective Tab. Alternatively you
can set DHCP ranges for a Subnet via the following CLI command:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>pvesh set /cluster/sdn/vnets/&lt;vnet&gt;/subnets/&lt;subnet&gt;
 -dhcp-range start-address=10.0.1.100,end-address=10.0.1.200
 -dhcp-range start-address=10.0.2.100,end-address=10.0.2.200</pre>
</div></div>
<div class="paragraph">
<p>You also need to have a gateway configured for the subnet - otherwise
automatic DHCP will not work.</p></div>
<div class="paragraph">
<p>The DHCP plugin will then allocate IPs in the IPAM only in the configured
ranges.</p></div>
<div class="paragraph">
<p>Do not forget to follow the installation steps for the
<a href="#pvesdn_install_dhcp_ipam">dnsmasq DHCP plugin</a> as well.</p></div>
</div>
<div class="sect3">
<h4 id="_plugins">12.12.2. Plugins
 <a class="headerlink" href="#_plugins" title="Permalink to this heading"></a>
</h4>
<div class="sect4">
<h5 id="_dnsmasq_plugin">Dnsmasq Plugin
 <a class="headerlink" href="#_dnsmasq_plugin" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Currently this is the only DHCP plugin and therefore the plugin that gets used
when you enable DHCP for a zone.</p></div>
<div class="paragraph">
<div class="title">Installation</div><p>For installation see the <a href="#pvesdn_install_dhcp_ipam">DHCP IPAM</a> section.</p></div>
<div class="paragraph">
<div class="title">Configuration</div><p>The plugin will create a new systemd service for each zone that dnsmasq gets
deployed to. The name for the service is <span class="monospaced">dnsmasq@&lt;zone&gt;</span>. The lifecycle of this
service is managed by the DHCP plugin.</p></div>
<div class="paragraph">
<p>The plugin automatically generates the following configuration files in the
folder <span class="monospaced">/etc/dnsmasq.d/&lt;zone&gt;</span>:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">00-default.conf</span>
</dt>
<dd>
<p>
This contains the default global configuration for a dnsmasq instance.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">10-&lt;zone&gt;-&lt;subnet_cidr&gt;.conf</span>
</dt>
<dd>
<p>
This file configures specific options for a subnet, such as the DNS server that
should get configured via DHCP.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">10-&lt;zone&gt;-&lt;subnet_cidr&gt;.ranges.conf</span>
</dt>
<dd>
<p>
This file configures the DHCP ranges for the dnsmasq instance.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">ethers</span>
</dt>
<dd>
<p>
This file contains the MAC-address and IP mappings from the IPAM plugin. In
order to override those mappings, please use the respective IPAM plugin rather
than editing this file, as it will get overwritten by the dnsmasq plugin.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p>You must not edit any of the above files, since they are managed by the DHCP
plugin. In order to customize the dnsmasq configuration you can create
additional files (e.g. <span class="monospaced">90-custom.conf</span>) in the configuration folder - they will
not get changed by the dnsmasq DHCP plugin.</p></div>
<div class="paragraph">
<p>Configuration files are read in order, so you can control the order of the
configuration directives by naming your custom configuration files appropriately.</p></div>
<div class="paragraph">
<p>DHCP leases are stored in the file <span class="monospaced">/var/lib/misc/dnsmasq.&lt;zone&gt;.leases</span>.</p></div>
<div class="paragraph">
<p>When using the PVE IPAM plugin, you can update, create and delete DHCP leases.
For more information please consult the documentation of
<a href="#pvesdn_ipam_plugin_pveipam">the PVE IPAM plugin</a>. Changing DHCP leases is
currently not supported for the other IPAM plugins.</p></div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="pvesdn_setup_examples">
<span>12.13. Examples</span>
 <a class="headerlink" href="#pvesdn_setup_examples" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>This section presents multiple configuration examples tailored for common SDN
use cases. It aims to offer tangible implementations, providing additional
details to enhance comprehension of the available configuration options.</p></div>
<div class="sect3">
<h4 id="pvesdn_setup_example_simple">12.13.1. Simple Zone Example
 <a class="headerlink" href="#pvesdn_setup_example_simple" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Simple zone networks create an isolated network for guests on a single host to
connect to each other.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">connection between guests are possible if all guests reside on a same host
but cannot be reached on other nodes.</td>
</tr></tbody></table>
</div>
<div class="ulist"><ul>
<li>
<p>
Create a simple zone named <span class="monospaced">simple</span>.
</p>
</li>
<li>
<p>
Add a VNet names <span class="monospaced">vnet1</span>.
</p>
</li>
<li>
<p>
Create a Subnet with a gateway and the SNAT option enabled.
</p>
</li>
<li>
<p>
This creates a network bridge <span class="monospaced">vnet1</span> on the node. Assign this bridge to the
  guests that shall join the network and configure an IP address.
</p>
</li>
</ul></div>
<div class="paragraph">
<p>The network interface configuration in two VMs may look like this which allows
them to communicate via the 10.0.1.0/24 network.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>allow-hotplug ens19
iface ens19 inet static
        address 10.0.1.14/24</pre>
</div></div>
<div class="listingblock">
<div class="content monospaced">
<pre>allow-hotplug ens19
iface ens19 inet static
        address 10.0.1.15/24</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="pvesdn_setup_example_nat">12.13.2. Source NAT Example
 <a class="headerlink" href="#pvesdn_setup_example_nat" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>If you want to allow outgoing connections for guests in the simple network zone
the simple zone offers a Source NAT (SNAT) option.</p></div>
<div class="paragraph">
<p>Starting from the configuration <a href="#pvesdn_setup_example_simple">above</a>, Add a
Subnet to the VNet <span class="monospaced">vnet1</span>, set a gateway IP and enable the SNAT option.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>Subnet: 172.16.0.0/24
Gateway: 172.16.0.1
SNAT: checked</pre>
</div></div>
<div class="paragraph">
<p>In the guests configure the static IP address inside the subnet’s IP range.</p></div>
<div class="paragraph">
<p>The node itself will join this network with the Gateway IP <em>172.16.0.1</em> and
function as the NAT gateway for guests within the subnet range.</p></div>
</div>
<div class="sect3">
<h4 id="pvesdn_setup_example_vlan">12.13.3. VLAN Setup Example
 <a class="headerlink" href="#pvesdn_setup_example_vlan" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>When VMs on different nodes need to communicate through an isolated network, the
VLAN zone allows network level isolation using VLAN tags.</p></div>
<div class="paragraph">
<p>Create a VLAN zone named <span class="monospaced">myvlanzone</span>:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>ID: myvlanzone
Bridge: vmbr0</pre>
</div></div>
<div class="paragraph">
<p>Create a VNet named <span class="monospaced">myvnet1</span> with VLAN tag 10 and the previously created
<span class="monospaced">myvlanzone</span>.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>ID: myvnet1
Zone: myvlanzone
Tag: 10</pre>
</div></div>
<div class="paragraph">
<p>Apply the configuration through the main SDN panel, to create VNets locally on
each node.</p></div>
<div class="paragraph">
<p>Create a Debian-based virtual machine (<em>vm1</em>) on node1, with a vNIC on <span class="monospaced">myvnet1</span>.</p></div>
<div class="paragraph">
<p>Use the following network configuration for this VM:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>auto eth0
iface eth0 inet static
        address 10.0.3.100/24</pre>
</div></div>
<div class="paragraph">
<p>Create a second virtual machine (<em>vm2</em>) on node2, with a vNIC on the same VNet
<span class="monospaced">myvnet1</span> as vm1.</p></div>
<div class="paragraph">
<p>Use the following network configuration for this VM:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>auto eth0
iface eth0 inet static
        address 10.0.3.101/24</pre>
</div></div>
<div class="paragraph">
<p>Following this, you should be able to ping between both VMs using that network.</p></div>
</div>
<div class="sect3">
<h4 id="pvesdn_setup_example_qinq">12.13.4. QinQ Setup Example
 <a class="headerlink" href="#pvesdn_setup_example_qinq" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>This example configures two QinQ zones and adds two VMs to each zone to
demonstrate the additional layer of VLAN tags which allows the configuration of
more isolated VLANs.</p></div>
<div class="paragraph">
<p>A typical use case for this configuration is a hosting provider that provides an
isolated network to customers for VM communication but isolates the VMs from
other customers.</p></div>
<div class="paragraph">
<p>Create a QinQ zone named <span class="monospaced">qinqzone1</span> with service VLAN 20</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>ID: qinqzone1
Bridge: vmbr0
Service VLAN: 20</pre>
</div></div>
<div class="paragraph">
<p>Create another QinQ zone named <span class="monospaced">qinqzone2</span> with service VLAN 30</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>ID: qinqzone2
Bridge: vmbr0
Service VLAN: 30</pre>
</div></div>
<div class="paragraph">
<p>Create a VNet named <span class="monospaced">myvnet1</span> with VLAN-ID 100 on the previously created
<span class="monospaced">qinqzone1</span> zone.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>ID: qinqvnet1
Zone: qinqzone1
Tag: 100</pre>
</div></div>
<div class="paragraph">
<p>Create a <span class="monospaced">myvnet2</span> with VLAN-ID 100 on the  <span class="monospaced">qinqzone2</span> zone.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>ID: qinqvnet2
Zone: qinqzone2
Tag: 100</pre>
</div></div>
<div class="paragraph">
<p>Apply the configuration on the main SDN web interface panel to create VNets
locally on each node.</p></div>
<div class="paragraph">
<p>Create four Debian-bases virtual machines (vm1, vm2, vm3, vm4) and add network
interfaces to vm1 and vm2 with bridge <span class="monospaced">qinqvnet1</span> and vm3 and vm4 with bridge
<span class="monospaced">qinqvnet2</span>.</p></div>
<div class="paragraph">
<p>Inside the VM, configure the IP addresses of the interfaces, for example via
<span class="monospaced">/etc/network/interfaces</span>:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>auto eth0
iface eth0 inet static
        address 10.0.3.101/24</pre>
</div></div>
<div class="paragraph">
<p>Configure all four VMs to have IP addresses from the <em>10.0.3.101</em> to
<em>10.0.3.104</em> range.</p></div>
<div class="paragraph">
<p>Now you should be able to ping between the VMs <em>vm1</em> and <em>vm2</em>, as well as
between <em>vm3</em> and <em>vm4</em>. However, neither of VMs <em>vm1</em> or <em>vm2</em> can ping VMs
<em>vm3</em> or <em>vm4</em>, as they are on a different zone with a different service-VLAN.</p></div>
</div>
<div class="sect3">
<h4 id="pvesdn_setup_example_vxlan">12.13.5. VXLAN Setup Example
 <a class="headerlink" href="#pvesdn_setup_example_vxlan" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The example assumes a cluster with three nodes, with the node IP addresses
192.168.0.1, 192.168.0.2 and 192.168.0.3.</p></div>
<div class="paragraph">
<p>Create a VXLAN zone named <span class="monospaced">myvxlanzone</span> and add all IPs from the nodes to the
peer address list. Use the default MTU of 1450 or configure accordingly.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>ID: myvxlanzone
Peers Address List: 192.168.0.1,192.168.0.2,192.168.0.3</pre>
</div></div>
<div class="paragraph">
<p>Create a VNet named <span class="monospaced">vxvnet1</span> using the VXLAN zone <span class="monospaced">myvxlanzone</span> created
previously.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>ID: vxvnet1
Zone: myvxlanzone
Tag: 100000</pre>
</div></div>
<div class="paragraph">
<p>Apply the configuration on the main SDN web interface panel to create VNets
locally on each nodes.</p></div>
<div class="paragraph">
<p>Create a Debian-based virtual machine (<em>vm1</em>) on node1, with a vNIC on <span class="monospaced">vxvnet1</span>.</p></div>
<div class="paragraph">
<p>Use the following network configuration for this VM (note the lower MTU).</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>auto eth0
iface eth0 inet static
        address 10.0.3.100/24
        mtu 1450</pre>
</div></div>
<div class="paragraph">
<p>Create a second virtual machine (<em>vm2</em>) on node3, with a vNIC on the same VNet
<span class="monospaced">vxvnet1</span> as vm1.</p></div>
<div class="paragraph">
<p>Use the following network configuration for this VM:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>auto eth0
iface eth0 inet static
        address 10.0.3.101/24
        mtu 1450</pre>
</div></div>
<div class="paragraph">
<p>Then, you should be able to ping between between <em>vm1</em> and <em>vm2</em>.</p></div>
</div>
<div class="sect3">
<h4 id="pvesdn_setup_example_evpn">12.13.6. EVPN Setup Example
 <a class="headerlink" href="#pvesdn_setup_example_evpn" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The example assumes a cluster with three nodes (node1, node2, node3) with IP
addresses 192.168.0.1, 192.168.0.2 and 192.168.0.3.</p></div>
<div class="paragraph">
<p>Create an EVPN controller, using a private ASN number and the above node
addresses as peers.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>ID: myevpnctl
ASN#: 65000
Peers: 192.168.0.1,192.168.0.2,192.168.0.3</pre>
</div></div>
<div class="paragraph">
<p>Create an EVPN zone named <span class="monospaced">myevpnzone</span>, assign the previously created
EVPN-controller and define <em>node1</em> and <em>node2</em> as exit nodes.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>ID: myevpnzone
VRF VXLAN Tag: 10000
Controller: myevpnctl
MTU: 1450
VNet MAC Address: 32:F4:05:FE:6C:0A
Exit Nodes: node1,node2</pre>
</div></div>
<div class="paragraph">
<p>Create the first VNet named <span class="monospaced">myvnet1</span> using the EVPN zone <span class="monospaced">myevpnzone</span>.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>ID: myvnet1
Zone: myevpnzone
Tag: 11000</pre>
</div></div>
<div class="paragraph">
<p>Create a subnet on <span class="monospaced">myvnet1</span>:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>Subnet: 10.0.1.0/24
Gateway: 10.0.1.1</pre>
</div></div>
<div class="paragraph">
<p>Create the second VNet named <span class="monospaced">myvnet2</span> using the same EVPN zone <span class="monospaced">myevpnzone</span>.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>ID: myvnet2
Zone: myevpnzone
Tag: 12000</pre>
</div></div>
<div class="paragraph">
<p>Create a different subnet on <span class="monospaced">myvnet2`</span>:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>Subnet: 10.0.2.0/24
Gateway: 10.0.2.1</pre>
</div></div>
<div class="paragraph">
<p>Apply the configuration from the main SDN web interface panel to create VNets
locally on each node and generate the FRR configuration.</p></div>
<div class="paragraph">
<p>Create a Debian-based virtual machine (<em>vm1</em>) on node1, with a vNIC on <span class="monospaced">myvnet1</span>.</p></div>
<div class="paragraph">
<p>Use the following network configuration for <em>vm1</em>:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>auto eth0
iface eth0 inet static
        address 10.0.1.100/24
        gateway 10.0.1.1
        mtu 1450</pre>
</div></div>
<div class="paragraph">
<p>Create a second virtual machine (<em>vm2</em>) on node2, with a vNIC on the other VNet
<span class="monospaced">myvnet2</span>.</p></div>
<div class="paragraph">
<p>Use the following network configuration for <em>vm2</em>:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>auto eth0
iface eth0 inet static
        address 10.0.2.100/24
        gateway 10.0.2.1
        mtu 1450</pre>
</div></div>
<div class="paragraph">
<p>Now you should be able to ping vm2 from vm1, and vm1 from vm2.</p></div>
<div class="paragraph">
<p>If you ping an external IP from <em>vm2</em> on the non-gateway node3, the packet
will go to the configured <em>myvnet2</em> gateway, then will be routed to the exit
nodes (<em>node1</em> or <em>node2</em>) and from there it will leave those nodes over the
default gateway configured on node1 or node2.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">You need to add reverse routes for the <em>10.0.1.0/24</em> and <em>10.0.2.0/24</em>
networks to node1 and node2 on your external gateway, so that the public network
can reply back.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>If you have configured an external BGP router, the BGP-EVPN routes (10.0.1.0/24
and 10.0.2.0/24 in this example), will be announced dynamically.</p></div>
</div>
</div>
<div class="sect2">
<h3 id="pvesdn_notes">
<span>12.14. Notes</span>
 <a class="headerlink" href="#pvesdn_notes" title="Permalink to this heading"></a>
</h3>
<div class="sect3">
<h4 id="_multiple_evpn_exit_nodes">12.14.1. Multiple EVPN Exit Nodes
 <a class="headerlink" href="#_multiple_evpn_exit_nodes" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>If you have multiple gateway nodes, you should disable the <span class="monospaced">rp_filter</span> (Strict
Reverse Path Filter) option, because packets can arrive at one node but go out
from another node.</p></div>
<div class="paragraph">
<p>Add the following to <span class="monospaced">/etc/sysctl.conf</span>:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>net.ipv4.conf.default.rp_filter=0
net.ipv4.conf.all.rp_filter=0</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_vxlan_ipsec_encryption">12.14.2. VXLAN IPSEC Encryption
 <a class="headerlink" href="#_vxlan_ipsec_encryption" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>To add IPSEC encryption on top of a VXLAN, this example shows how to use
<span class="monospaced">strongswan</span>.</p></div>
<div class="paragraph">
<p>You`ll need to reduce the <em>MTU</em> by additional 60 bytes for IPv4 or 80 bytes for
IPv6 to handle encryption.</p></div>
<div class="paragraph">
<p>So with default real 1500 MTU, you need to use a MTU of 1370 (1370 + 80 (IPSEC)
+ 50 (VXLAN) == 1500).</p></div>
<div class="paragraph">
<p>Install strongswan on the host.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>apt install strongswan</pre>
</div></div>
<div class="paragraph">
<p>Add configuration to <span class="monospaced">/etc/ipsec.conf</span>. We only need to encrypt traffic from
the VXLAN UDP port <em>4789</em>.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>conn %default
    ike=aes256-sha1-modp1024!  # the fastest, but reasonably secure cipher on modern HW
    esp=aes256-sha1!
    leftfirewall=yes           # this is necessary when using Proxmox VE firewall rules

conn output
    rightsubnet=%dynamic[udp/4789]
    right=%any
    type=transport
    authby=psk
    auto=route

conn input
    leftsubnet=%dynamic[udp/4789]
    type=transport
    authby=psk
    auto=route</pre>
</div></div>
<div class="paragraph">
<p>Generate a pre-shared key with:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>openssl rand -base64 128</pre>
</div></div>
<div class="paragraph">
<p>and add the key to <span class="monospaced">/etc/ipsec.secrets</span>, so that the file contents looks like:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>: PSK &lt;generatedbase64key&gt;</pre>
</div></div>
<div class="paragraph">
<p>Copy the PSK and the configuration to all nodes participating in the VXLAN network.</p></div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="chapter_pve_firewall">
13. Proxmox VE Firewall
 <a class="headerlink" href="#chapter_pve_firewall" title="Permalink to this heading"></a>
</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Proxmox VE Firewall provides an easy way to protect your IT
infrastructure. You can setup firewall rules for all hosts
inside a cluster, or define rules for virtual machines and
containers. Features like firewall macros, security groups, IP sets
and aliases help to make that task easier.</p></div>
<div class="paragraph">
<p>While all configuration is stored on the cluster file system, the
<span class="monospaced">iptables</span>-based firewall service runs on each cluster node, and thus provides
full isolation between virtual machines. The distributed nature of
this system also provides much higher bandwidth than a central
firewall solution.</p></div>
<div class="paragraph">
<p>The firewall has full support for IPv4 and IPv6. IPv6 support is fully
transparent, and we filter traffic for both protocols by default. So
there is no need to maintain a different set of rules for IPv6.</p></div>
<div class="sect2">
<h3 id="_zones">
<span>13.1. Zones</span>
 <a class="headerlink" href="#_zones" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>The Proxmox VE firewall groups the network into the following logical zones:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
Host
</dt>
<dd>
<p>
Traffic from/to a cluster node
</p>
</dd>
<dt class="hdlist1">
VM
</dt>
<dd>
<p>
Traffic from/to a specific VM
</p>
</dd>
</dl></div>
<div class="paragraph">
<p>For each zone, you can define firewall rules for incoming and/or
outgoing traffic.</p></div>
</div>
<div class="sect2">
<h3 id="_configuration_files">
<span>13.2. Configuration Files</span>
 <a class="headerlink" href="#_configuration_files" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>All firewall related configuration is stored on the proxmox cluster
file system. So those files are automatically distributed to all
cluster nodes, and the <span class="monospaced">pve-firewall</span> service updates the underlying
<span class="monospaced">iptables</span> rules automatically on changes.</p></div>
<div class="paragraph">
<p>You can configure anything using the GUI (i.e. <strong>Datacenter</strong> → <strong>Firewall</strong>,
or on a <strong>Node</strong> → <strong>Firewall</strong>), or you can edit the configuration files
directly using your preferred editor.</p></div>
<div class="paragraph">
<p>Firewall configuration files contain sections of key-value
pairs. Lines beginning with a <span class="monospaced">#</span> and blank lines are considered
comments. Sections start with a header line containing the section
name enclosed in <span class="monospaced">[</span> and <span class="monospaced">]</span>.</p></div>
<div class="sect3">
<h4 id="pve_firewall_cluster_wide_setup">13.2.1. Cluster Wide Setup
 <a class="headerlink" href="#pve_firewall_cluster_wide_setup" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The cluster-wide firewall configuration is stored at:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>/etc/pve/firewall/cluster.fw</pre>
</div></div>
<div class="paragraph">
<p>The configuration can contain the following sections:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">[OPTIONS]</span>
</dt>
<dd>
<p>
This is used to set cluster-wide firewall options.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">ebtables</span>: <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Enable ebtables rules cluster wide.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">enable</span>: <span class="monospaced">&lt;integer&gt; (0 - N)</span> 
</dt>
<dd>
<p>
Enable or disable the firewall cluster wide.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">log_ratelimit</span>: <span class="monospaced">[enable=]&lt;1|0&gt; [,burst=&lt;integer&gt;] [,rate=&lt;rate&gt;]</span> 
</dt>
<dd>
<p>
Log ratelimiting settings
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">burst</span>=<span class="monospaced">&lt;integer&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">5</span>)
</dt>
<dd>
<p>
Initial burst of packages which will always get logged before the rate is applied
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">enable</span>=<span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Enable or disable log rate limiting
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">rate</span>=<span class="monospaced">&lt;rate&gt;</span> (<em>default =</em> <span class="monospaced">1/second</span>)
</dt>
<dd>
<p>
Frequency with which the burst bucket gets refilled
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">policy_in</span>: <span class="monospaced">&lt;ACCEPT | DROP | REJECT&gt;</span> 
</dt>
<dd>
<p>
Input policy.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">policy_out</span>: <span class="monospaced">&lt;ACCEPT | DROP | REJECT&gt;</span> 
</dt>
<dd>
<p>
Output policy.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">[RULES]</span>
</dt>
<dd>
<p>
This sections contains cluster-wide firewall rules for all nodes.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">[IPSET &lt;name&gt;]</span>
</dt>
<dd>
<p>
Cluster wide IP set definitions.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">[GROUP &lt;name&gt;]</span>
</dt>
<dd>
<p>
Cluster wide security group definitions.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">[ALIASES]</span>
</dt>
<dd>
<p>
Cluster wide Alias definitions.
</p>
</dd>
</dl></div>
<div class="sect4">
<h5 id="_enabling_the_firewall">Enabling the Firewall
 <a class="headerlink" href="#_enabling_the_firewall" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>The firewall is completely disabled by default, so you need to
set the enable option here:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>[OPTIONS]
# enable firewall (cluster-wide setting, default is disabled)
enable: 1</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Important" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAALa0lEQVRogdWZa2wc1RXHfzM7O/te
P9e1vXHSmEdjx3YeDkkaF6REKRRCEDSEFNkRjdSWSsgC2iqoRWqLQKiqIBg1NOQDiMeHtkQIQkRR
S9S4aWwgCQoUgl232KkT28J21l6vd3d2dx79sDuTXdtre03Uqlc62tl53Pv/n3PuOffcKxiGwf9z
E//XAL5sk65WR0a6AaDrOjMtKwgCoigiCAKCIAhXa9wvTcAEHhoc5KMjR7jw3nucf+MN67k/GCS4
YQOr77yTpl27kGUZm81mXC0ywlLnQDbwEwcO0P2b3yz4jV5RwZq2Nm5/5BH8fj+SJGGz2b4UiSUR
MAzD0DSNEwcO8MdHHln8d0AcsAUCfOe551h7yy04HA7sdvuSrVEwAV3XjdDgIC/v2cOl06dznqWA
JCAAMvn9MwlEgF1PP8037rsPt9uN3W5fkjUKIqDrujF+4QLP3XQTU0NDOc8UIAa4y8sJtrQQqK/H
7XQSfv99Pn/nHQRRxNB16/0EEAUatm9n76uv4vP5cDgcBZNYNIF84I0MEK20lIZvf5sNe/dSWlqK
1+vFxBEZHubT3/+evqNHmR4dJRWP55BevW0bra+8gt/vx+l0IknSokksikA+8ElgClixYwc3799P
ZWUlTqczbz+x0VFOPvEEQ6dPE/niCwxdt0jUbd1K68svU1xcXBCJBQnMBz5eWkrL/v2s++Y3KSsr
QxQXlxc7f/5zBjo7CQ8NoadSKKQn943t7dyyf79FYjHuNC8BwzAMVVV56Z57+OzNN3PAJ8vK+FZH
B2s2b55X6/nahRMnOP7TnxLOKEXJyO2/+hVf37uXoqIinE6nmfzyksirMjNUvvGjH+WAVwG1vJxb
Dhxg1bp1uFwuMwQWJCu3bWPN3r24SkoAcGbA/OXXv2bws8+IRqOkUqlZGX1RBAzDMHRd51+nTvHe
wYNX7gPTQMO+faxav56SkpIlgTdlU3s7TXv2IGUs6AGioRDH2tuZmJggFouhqirGPCzyEUBVVX7X
1pZzPwI0fP/7bNmzh7KysrzADh8+zP3338++ffs4ePBg3vdEUWTDD3/IsuZmRLsdW4bEpd5eOp99
lnA4jKIoaJqWl8QsAqb2Txw4kDNp40BxXR0bWlspLy/PC+qZZ55hdHQUSZIYHByko6ODxx57bBZw
cy3kq6xk6y9/ib+qCgQBmbQ7nT50iIs9PUSjUZLJJIZhzEliLgKMDQzwp5/9zLqnkQ51W3/xCwKB
AE6nMy+B3t5exsfH6ezspLe3l3g8zuHDh2cBz5bKpibWtrXhLi0FwA3owNsPP8zExATxeBxVVecy
QC4Bc+L+taNjluvU33031dddN6/2BUGgpKSE/v5+JicncweaA3i2tDz8MIHrr0eU0gsQBzDW10ff
yZNEIhESicScrpRDQNd1xgYG+OC3v7XuJQBXIMDWn/yEioqKeUGMjY0hCAKKoszS1MjIyLzfAmx/
4gmKli0DQcBJOmicfOopJicnicfjJoG5LWBq/+Szz+a8oABNe/bgdrvndR1BEKiqqiIejxOLxWYR
CAaDC0alqjVrqGxowFNWhkDalUKff07f3/5mWSFTLFksLAK6rjN+4UKO9lXAUV7O2tbWeaNOthZl
WZ4FHlh0aN3y4IPIHk+6L8AGvNfRQTgctuZCthVEU/u6rnPutddyBk0A9bt3W4us+QY2fbyurg6v
15vTzw033LBoAsHmZoLNzdgcDgBcQKi/n391dTE9PU0ymcyxgkmAVCrF+4cO5WjfFgiwfoGwOTOy
lJWVUVtba1mkUAKCILClvR1veTkA9owVPnzllZy8YFnA1H5/dzfTw8PWgziwcutWfD4fsiwvCNyU
xsZG3G43lZWVVl933XVXQQSq161j+ebNlhXcwBdnzxIOh60lhmkFEUDTNP7++uvWgBrp6mrNHBk3
H/BsAtFoFEdm8GAwyMaNGwsiIAgCjbt346uosKwgAp8cPWolNj1THInmsqHn6FGLQApY1thIxcqV
+Hy+WX6+0ETesWMHLpcLWZZ54IEHCgYvCAKrbrsNyem0JrQEXMzkhKzlBaKu61z+97+ZHhnJmbwr
tm2zJu5CWp8pra2tNDU1sXPnTtrb25dEQBAEvrplC57MXJCB8XPniEQixONxa6Uq6brOQHd3jvvo
QP2ttxZUpMxsL7zwwpK+AyxLXrt9Oxe6uxFEEXvGZc4fO0bVD35AMplM1wuapjGYtbugAp5AgKLq
aquu/W/JTEs37dqFqihWzSADk8PD1jzQNC3tQtmrThUINDbm+H4h0tfXx6OPPsodd9xBW1sbPT09
BQPPluu2b7fWRxIQ6usjFotZBCRN06wZbbZAfT1FRUWWKQtpzz//PMPDw/T09DA1NcXo6CjHjx+f
11Xma95AwHJjEdB0HUVRrhCYuRGrAiXBoFUqFtq6u7sZHx9nJBMUurq65uxnsX0Hm5v56A9/ANIJ
LXrpEolEwgqlUqZQyPlo6OOPlzx5ly1bxnBWQmxpackBW6hSBEC02axrpqasNZGmaek8kN0kYPDj
j1FVdUlz4Mknn6Surs4Cb5aUhYZiUy59+GGOgm3V1WiadiUPCILAV9autV5wAaM9PXQeOWJprBBp
aGjgxIkTpFIpOjs7aWxsXBJwUwZOnUKZmgLS2zl4vUjSlV1XURRFqtavv3IDKAVe/O53efPxx69a
SCxUBs+c4dV77+Vyfz+xy5eBdG3iX7ECm81m9S2JokiwuZmiujrCPT1AuqgOAqcef5zzL77Imt27
uemhhyipqVnYZ5cw8bPbR0eOcOall7g8MEB4aIhEJAJk9qOAwKZNyLJsHpQgRKNRIxQK0XPmDH/+
3vdQQ6G8na/YtIk199zDypYWVmzceFWAT1y8yMWzZzn/9tv0vfsudqeTlKIQHRvL2QSOAqU7d/K1
W2/l2muvpba2Nl3iJhIJIxKJMDw8TO/Zs3zw4x+jTUzkHVCw2fCUleHw+QiuXYu/upqqhgb8VVUY
hkFNczPFweCs7z556y3r+vyxYwiiyD+PHwdBwNB1krEYhqYRC4UwdB2D9LJmmnRtXHz77VTfeCPL
ly+ntraWmpqa9MaaqqqGoiiEQiGGhoYYHBzk00OHiHR2LkqDNlnG7nJhd7nQswqN+ZogiuipFMlY
DDWRgBl5KEF6woqAUFFByc03U756NZWVlSxfvpyamhoCgQAejwdB13VDVVVisRihUIiRkRGGh4cZ
+sc/GO3qIt7VhZGJAle7mVpWuVKDCBng8jXX4N68Gc8111BUVER5eTnV1dVUV1cTCASuFFqGYaDr
upFKpYjH40xMTDA+Ps7Y2Bjj4+OEw2EmenqYPneO1OAgZCWpxTY9S9QMcLhyHGUDRI8HqbISZ309
zoYGnE4nHo8Hv99PaWkpgUCAiooK6/DE4XCkI1HW2a6hqiqKojA9Pc3k5KQl4XCY6elpotFoetuk
v59UKIR66RLToRBJTcMYHUWIRvMSMPO6rbY2/SsIOFatQpYk5NWrkSQJWZZxOBy43W68Xi9+v5+i
oiJKSkooLi7G5/PhdruRZdnads85HzD3hpLJJIqiEIvFiEajRCIRIpEI0WiUWCxGPB631iOKopBK
pazUbi4Oc7Y+snKCzWbDZrMhSRKSJGG3262w6HK5cLlceDwevF4vPp8Pr9eL2+3G5XLhcDjMkxuE
TNibdcBhnv9qmkYqlSKZTJJIJFAUhXg8jqIoKIpCIpGwSCSTSVRVtWSuk3qTiAnebrfngHc4HDid
TpxOJy6Xy7qWZdk6wZzrxCbvCY1JRNd1NE3LAWgSM69ngp+PgCiK1gF3NhFTzHvmO+Y3wCzw8xLI
JpL5tYCZpEyw2f/nWt1ag2UtM0x3Mq1iAp1j+bH0M7KFCGWDnfmbd8AMnuxfUzL/C0rp/wFnFd4n
EQn3XQAAAABJRU5ErkJggg==">
</td>
<td class="content">If you enable the firewall, traffic to all hosts is blocked by
default. Only exceptions is WebGUI(8006) and ssh(22) from your local
network.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>If you want to administrate your Proxmox VE hosts from remote, you
need to create rules to allow traffic from those remote IPs to the web
GUI (port 8006). You may also want to allow ssh (port 22), and maybe
SPICE (port 3128).</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">Please open a SSH connection to one of your Proxmox VE hosts before
enabling the firewall. That way you still have access to the host if
something goes wrong .</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>To simplify that task, you can instead create an IPSet called
“management”, and add all remote IPs there. This creates all required
firewall rules to access the GUI from remote.</p></div>
</div>
</div>
<div class="sect3">
<h4 id="pve_firewall_host_specific_configuration">13.2.2. Host Specific Configuration
 <a class="headerlink" href="#pve_firewall_host_specific_configuration" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Host related configuration is read from:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>/etc/pve/nodes/&lt;nodename&gt;/host.fw</pre>
</div></div>
<div class="paragraph">
<p>This is useful if you want to overwrite rules from <span class="monospaced">cluster.fw</span>
config. You can also increase log verbosity, and set netfilter related
options. The configuration can contain the following sections:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">[OPTIONS]</span>
</dt>
<dd>
<p>
This is used to set host related firewall options.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">enable</span>: <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Enable host firewall rules.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">log_level_in</span>: <span class="monospaced">&lt;alert | crit | debug | emerg | err | info | nolog | notice | warning&gt;</span> 
</dt>
<dd>
<p>
Log level for incoming traffic.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">log_level_out</span>: <span class="monospaced">&lt;alert | crit | debug | emerg | err | info | nolog | notice | warning&gt;</span> 
</dt>
<dd>
<p>
Log level for outgoing traffic.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">log_nf_conntrack</span>: <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Enable logging of conntrack information.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">ndp</span>: <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Enable NDP (Neighbor Discovery Protocol).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">nf_conntrack_allow_invalid</span>: <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Allow invalid packets on connection tracking.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">nf_conntrack_helpers</span>: <span class="monospaced">&lt;string&gt;</span> (<em>default =</em> ``)
</dt>
<dd>
<p>
Enable conntrack helpers for specific protocols. Supported protocols: amanda, ftp, irc, netbios-ns, pptp, sane, sip, snmp, tftp
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">nf_conntrack_max</span>: <span class="monospaced">&lt;integer&gt; (32768 - N)</span> (<em>default =</em> <span class="monospaced">262144</span>)
</dt>
<dd>
<p>
Maximum number of tracked connections.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">nf_conntrack_tcp_timeout_established</span>: <span class="monospaced">&lt;integer&gt; (7875 - N)</span> (<em>default =</em> <span class="monospaced">432000</span>)
</dt>
<dd>
<p>
Conntrack established timeout.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">nf_conntrack_tcp_timeout_syn_recv</span>: <span class="monospaced">&lt;integer&gt; (30 - 60)</span> (<em>default =</em> <span class="monospaced">60</span>)
</dt>
<dd>
<p>
Conntrack syn recv timeout.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">nftables</span>: <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Enable nftables based firewall (tech preview)
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">nosmurfs</span>: <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Enable SMURFS filter.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">protection_synflood</span>: <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Enable synflood protection
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">protection_synflood_burst</span>: <span class="monospaced">&lt;integer&gt;</span> (<em>default =</em> <span class="monospaced">1000</span>)
</dt>
<dd>
<p>
Synflood protection rate burst by ip src.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">protection_synflood_rate</span>: <span class="monospaced">&lt;integer&gt;</span> (<em>default =</em> <span class="monospaced">200</span>)
</dt>
<dd>
<p>
Synflood protection rate syn/sec by ip src.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">smurf_log_level</span>: <span class="monospaced">&lt;alert | crit | debug | emerg | err | info | nolog | notice | warning&gt;</span> 
</dt>
<dd>
<p>
Log level for SMURFS filter.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">tcp_flags_log_level</span>: <span class="monospaced">&lt;alert | crit | debug | emerg | err | info | nolog | notice | warning&gt;</span> 
</dt>
<dd>
<p>
Log level for illegal tcp flags filter.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">tcpflags</span>: <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Filter illegal combinations of TCP flags.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">[RULES]</span>
</dt>
<dd>
<p>
This sections contains host specific firewall rules.
</p>
</dd>
</dl></div>
</div>
<div class="sect3">
<h4 id="pve_firewall_vm_container_configuration">13.2.3. VM/Container Configuration
 <a class="headerlink" href="#pve_firewall_vm_container_configuration" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>VM firewall configuration is read from:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>/etc/pve/firewall/&lt;VMID&gt;.fw</pre>
</div></div>
<div class="paragraph">
<p>and contains the following data:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">[OPTIONS]</span>
</dt>
<dd>
<p>
This is used to set VM/Container related firewall options.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">dhcp</span>: <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Enable DHCP.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">enable</span>: <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Enable/disable firewall rules.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">ipfilter</span>: <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Enable default IP filters. This is equivalent to adding an empty ipfilter-net&lt;id&gt; ipset for every interface. Such ipsets implicitly contain sane default restrictions such as restricting IPv6 link local addresses to the one derived from the interface’s MAC address. For containers the configured IP addresses will be implicitly added.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">log_level_in</span>: <span class="monospaced">&lt;alert | crit | debug | emerg | err | info | nolog | notice | warning&gt;</span> 
</dt>
<dd>
<p>
Log level for incoming traffic.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">log_level_out</span>: <span class="monospaced">&lt;alert | crit | debug | emerg | err | info | nolog | notice | warning&gt;</span> 
</dt>
<dd>
<p>
Log level for outgoing traffic.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">macfilter</span>: <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Enable/disable MAC address filter.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">ndp</span>: <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Enable NDP (Neighbor Discovery Protocol).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">policy_in</span>: <span class="monospaced">&lt;ACCEPT | DROP | REJECT&gt;</span> 
</dt>
<dd>
<p>
Input policy.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">policy_out</span>: <span class="monospaced">&lt;ACCEPT | DROP | REJECT&gt;</span> 
</dt>
<dd>
<p>
Output policy.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">radv</span>: <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Allow sending Router Advertisement.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">[RULES]</span>
</dt>
<dd>
<p>
This sections contains VM/Container firewall rules.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">[IPSET &lt;name&gt;]</span>
</dt>
<dd>
<p>
IP set definitions.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">[ALIASES]</span>
</dt>
<dd>
<p>
IP Alias definitions.
</p>
</dd>
</dl></div>
<div class="sect4">
<h5 id="_enabling_the_firewall_for_vms_and_containers">Enabling the Firewall for VMs and Containers
 <a class="headerlink" href="#_enabling_the_firewall_for_vms_and_containers" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Each virtual network device has its own firewall enable flag. So you
can selectively enable the firewall for each interface. This is
required in addition to the general firewall <span class="monospaced">enable</span> option.</p></div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_firewall_rules">
<span>13.3. Firewall Rules</span>
 <a class="headerlink" href="#_firewall_rules" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Firewall rules consists of a direction (<span class="monospaced">IN</span> or <span class="monospaced">OUT</span>) and an
action (<span class="monospaced">ACCEPT</span>, <span class="monospaced">DENY</span>, <span class="monospaced">REJECT</span>). You can also specify a macro
name. Macros contain predefined sets of rules and options. Rules can be
disabled by prefixing them with <span class="monospaced">|</span>.</p></div>
<div class="listingblock">
<div class="title">Firewall rules syntax</div>
<div class="content monospaced">
<pre>[RULES]

DIRECTION ACTION [OPTIONS]
|DIRECTION ACTION [OPTIONS] # disabled rule

DIRECTION MACRO(ACTION) [OPTIONS] # use predefined macro</pre>
</div></div>
<div class="paragraph">
<p>The following options can be used to refine rule matches.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--dest</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Restrict packet destination address. This can refer to a single IP address, an IP set (<em>+ipsetname</em>) or an IP alias definition. You can also specify an address range like <em>20.34.101.207-201.3.9.99</em>, or a list of IP addresses and networks (entries are separated by comma). Please do not mix IPv4 and IPv6 addresses inside such lists.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--dport</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Restrict TCP/UDP destination port. You can use service names or simple numbers (0-65535), as defined in <em>/etc/services</em>. Port ranges can be specified with <em>\d+:\d+</em>, for example <em>80:85</em>, and you can use comma separated list to match several ports or ranges.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--icmp-type</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Specify icmp-type. Only valid if proto equals <em>icmp</em> or <em>icmpv6</em>/<em>ipv6-icmp</em>.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--iface</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Network interface name. You have to use network configuration key names for VMs and containers (<em>net\d+</em>). Host related rules can use arbitrary strings.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--log</span> <span class="monospaced">&lt;alert | crit | debug | emerg | err | info | nolog | notice | warning&gt;</span> 
</dt>
<dd>
<p>
Log level for firewall rule.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--proto</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
IP protocol. You can use protocol names (<em>tcp</em>/<em>udp</em>) or simple numbers, as defined in <em>/etc/protocols</em>.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--source</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Restrict packet source address. This can refer to a single IP address, an IP set (<em>+ipsetname</em>) or an IP alias definition. You can also specify an address range like <em>20.34.101.207-201.3.9.99</em>, or a list of IP addresses and networks (entries are separated by comma). Please do not mix IPv4 and IPv6 addresses inside such lists.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--sport</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Restrict TCP/UDP source port. You can use service names or simple numbers (0-65535), as defined in <em>/etc/services</em>. Port ranges can be specified with <em>\d+:\d+</em>, for example <em>80:85</em>, and you can use comma separated list to match several ports or ranges.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p>Here are some examples:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>[RULES]
IN SSH(ACCEPT) -i net0
IN SSH(ACCEPT) -i net0 # a comment
IN SSH(ACCEPT) -i net0 -source 192.168.2.192 # only allow SSH from 192.168.2.192
IN SSH(ACCEPT) -i net0 -source 10.0.0.1-10.0.0.10 # accept SSH for IP range
IN SSH(ACCEPT) -i net0 -source 10.0.0.1,10.0.0.2,10.0.0.3 #accept ssh for IP list
IN SSH(ACCEPT) -i net0 -source +mynetgroup # accept ssh for ipset mynetgroup
IN SSH(ACCEPT) -i net0 -source myserveralias #accept ssh for alias myserveralias

|IN SSH(ACCEPT) -i net0 # disabled rule

IN  DROP # drop all incoming packages
OUT ACCEPT # accept all outgoing packages</pre>
</div></div>
</div>
<div class="sect2">
<h3 id="pve_firewall_security_groups">
<span>13.4. Security Groups</span>
 <a class="headerlink" href="#pve_firewall_security_groups" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>A security group is a collection of rules, defined at cluster level, which
can be used in all VMs' rules. For example you can define a group named
“webserver” with rules to open the <em>http</em> and <em>https</em> ports.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># /etc/pve/firewall/cluster.fw

[group webserver]
IN  ACCEPT -p tcp -dport 80
IN  ACCEPT -p tcp -dport 443</pre>
</div></div>
<div class="paragraph">
<p>Then, you can add this group to a VM’s firewall</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># /etc/pve/firewall/&lt;VMID&gt;.fw

[RULES]
GROUP webserver</pre>
</div></div>
</div>
<div class="sect2">
<h3 id="pve_firewall_ip_aliases">
<span>13.5. IP Aliases</span>
 <a class="headerlink" href="#pve_firewall_ip_aliases" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>IP Aliases allow you to associate IP addresses of networks with a
name. You can then refer to those names:</p></div>
<div class="ulist"><ul>
<li>
<p>
inside IP set definitions
</p>
</li>
<li>
<p>
in <span class="monospaced">source</span> and <span class="monospaced">dest</span> properties of firewall rules
</p>
</li>
</ul></div>
<div class="sect3">
<h4 id="_standard_ip_alias_span_class_monospaced_local_network_span">13.5.1. Standard IP Alias <span class="monospaced">local_network</span>
 <a class="headerlink" href="#_standard_ip_alias_span_class_monospaced_local_network_span" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>This alias is automatically defined. Please use the following command
to see assigned values:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># pve-firewall localnet
local hostname: example
local IP address: 192.168.2.100
network auto detect: 192.168.0.0/20
using detected local_network: 192.168.0.0/20</pre>
</div></div>
<div class="paragraph">
<p>The firewall automatically sets up rules to allow everything needed
for cluster communication (corosync, API, SSH) using this alias.</p></div>
<div class="paragraph">
<p>The user can overwrite these values in the <span class="monospaced">cluster.fw</span> alias
section. If you use a single host on a public network, it is better to
explicitly assign the local IP address</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>#  /etc/pve/firewall/cluster.fw
[ALIASES]
local_network 1.2.3.4 # use the single IP address</pre>
</div></div>
</div>
</div>
<div class="sect2">
<h3 id="pve_firewall_ip_sets">
<span>13.6. IP Sets</span>
 <a class="headerlink" href="#pve_firewall_ip_sets" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>IP sets can be used to define groups of networks and hosts. You can
refer to them with ‘+name` in the firewall rules’ <span class="monospaced">source</span> and <span class="monospaced">dest</span>
properties.</p></div>
<div class="paragraph">
<p>The following example allows HTTP traffic from the <span class="monospaced">management</span> IP
set.</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>IN HTTP(ACCEPT) -source +management</pre>
</div></div>
<div class="sect3">
<h4 id="_standard_ip_set_span_class_monospaced_management_span">13.6.1. Standard IP set <span class="monospaced">management</span>
 <a class="headerlink" href="#_standard_ip_set_span_class_monospaced_management_span" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>This IP set applies only to host firewalls (not VM firewalls).  Those
IPs are allowed to do normal management tasks (Proxmox VE GUI, VNC, SPICE,
SSH).</p></div>
<div class="paragraph">
<p>The local cluster network is automatically added to this IP set (alias
<span class="monospaced">cluster_network</span>), to enable inter-host cluster
communication. (multicast,ssh,…)</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># /etc/pve/firewall/cluster.fw

[IPSET management]
192.168.2.10
192.168.2.10/24</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_standard_ip_set_span_class_monospaced_blacklist_span">13.6.2. Standard IP set <span class="monospaced">blacklist</span>
 <a class="headerlink" href="#_standard_ip_set_span_class_monospaced_blacklist_span" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Traffic from these IPs is dropped by every host’s and VM’s firewall.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># /etc/pve/firewall/cluster.fw

[IPSET blacklist]
77.240.159.182
213.87.123.0/24</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="pve_firewall_ipfilter_section">13.6.3. Standard IP set <span class="monospaced">ipfilter-net*</span>
 <a class="headerlink" href="#pve_firewall_ipfilter_section" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>These filters belong to a VM’s network interface and are mainly used to prevent
IP spoofing. If such a set exists for an interface then any outgoing traffic
with a source IP not matching its interface’s corresponding ipfilter set will
be dropped.</p></div>
<div class="paragraph">
<p>For containers with configured IP addresses these sets, if they exist (or are
activated via the general <span class="monospaced">IP Filter</span> option in the VM’s firewall’s <strong>options</strong>
tab), implicitly contain the associated IP addresses.</p></div>
<div class="paragraph">
<p>For both virtual machines and containers they also implicitly contain the
standard MAC-derived IPv6 link-local address in order to allow the neighbor
discovery protocol to work.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>/etc/pve/firewall/&lt;VMID&gt;.fw

[IPSET ipfilter-net0] # only allow specified IPs on net0
192.168.2.10</pre>
</div></div>
</div>
</div>
<div class="sect2">
<h3 id="pve_firewall_services_commands">
<span>13.7. Services and Commands</span>
 <a class="headerlink" href="#pve_firewall_services_commands" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>The firewall runs two service daemons on each node:</p></div>
<div class="ulist"><ul>
<li>
<p>
pvefw-logger: NFLOG daemon (ulogd replacement).
</p>
</li>
<li>
<p>
pve-firewall: updates iptables rules
</p>
</li>
</ul></div>
<div class="paragraph">
<p>There is also a CLI command named <span class="monospaced">pve-firewall</span>, which can be used to
start and stop the firewall service:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># pve-firewall start
# pve-firewall stop</pre>
</div></div>
<div class="paragraph">
<p>To get the status use:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># pve-firewall status</pre>
</div></div>
<div class="paragraph">
<p>The above command reads and compiles all firewall rules, so you will
see warnings if your firewall configuration contains any errors.</p></div>
<div class="paragraph">
<p>If you want to see the generated iptables rules you can use:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># iptables-save</pre>
</div></div>
</div>
<div class="sect2">
<h3 id="pve_firewall_default_rules">
<span>13.8. Default firewall rules</span>
 <a class="headerlink" href="#pve_firewall_default_rules" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>The following traffic is filtered by the default firewall configuration:</p></div>
<div class="sect3">
<h4 id="_datacenter_incoming_outgoing_drop_reject">13.8.1. Datacenter incoming/outgoing DROP/REJECT
 <a class="headerlink" href="#_datacenter_incoming_outgoing_drop_reject" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>If the input or output policy for the firewall is set to DROP or REJECT, the
following traffic is still allowed for all Proxmox VE hosts in the cluster:</p></div>
<div class="ulist"><ul>
<li>
<p>
traffic over the loopback interface
</p>
</li>
<li>
<p>
already established connections
</p>
</li>
<li>
<p>
traffic using the IGMP protocol
</p>
</li>
<li>
<p>
TCP traffic from management hosts to port 8006 in order to allow access to
  the web interface
</p>
</li>
<li>
<p>
TCP traffic from management hosts to the port range 5900 to 5999 allowing
  traffic for the VNC web console
</p>
</li>
<li>
<p>
TCP traffic from management hosts to port 3128 for connections to the SPICE
  proxy
</p>
</li>
<li>
<p>
TCP traffic from management hosts to port 22 to allow ssh access
</p>
</li>
<li>
<p>
UDP traffic in the cluster network to ports 5405-5412 for corosync
</p>
</li>
<li>
<p>
UDP multicast traffic in the cluster network
</p>
</li>
<li>
<p>
ICMP traffic type 3 (Destination Unreachable), 4 (congestion control) or 11
  (Time Exceeded)
</p>
</li>
</ul></div>
<div class="paragraph">
<p>The following traffic is dropped, but not logged even with logging enabled:</p></div>
<div class="ulist"><ul>
<li>
<p>
TCP connections with invalid connection state
</p>
</li>
<li>
<p>
Broadcast, multicast and anycast traffic not related to corosync, i.e., not
  coming through ports 5405-5412
</p>
</li>
<li>
<p>
TCP traffic to port 43
</p>
</li>
<li>
<p>
UDP traffic to ports 135 and 445
</p>
</li>
<li>
<p>
UDP traffic to the port range 137 to 139
</p>
</li>
<li>
<p>
UDP traffic form source port 137 to port range 1024 to 65535
</p>
</li>
<li>
<p>
UDP traffic to port 1900
</p>
</li>
<li>
<p>
TCP traffic to port 135, 139 and 445
</p>
</li>
<li>
<p>
UDP traffic originating from source port 53
</p>
</li>
</ul></div>
<div class="paragraph">
<p>The rest of the traffic is dropped or rejected, respectively, and also logged.
This may vary depending on the additional options enabled in
<strong>Firewall</strong> → <strong>Options</strong>, such as NDP, SMURFS and TCP flag filtering.</p></div>
<div class="paragraph" id="pve_firewall_iptables_inspect">
<p>Please inspect the output of the</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre> # iptables-save</pre>
</div></div>
<div class="paragraph">
<p>system command to see the firewall chains and rules active on your system.
This output is also included in a <span class="monospaced">System Report</span>, accessible over a node’s
subscription tab in the web GUI, or through the <span class="monospaced">pvereport</span> command-line tool.</p></div>
</div>
<div class="sect3">
<h4 id="_vm_ct_incoming_outgoing_drop_reject">13.8.2. VM/CT incoming/outgoing DROP/REJECT
 <a class="headerlink" href="#_vm_ct_incoming_outgoing_drop_reject" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>This drops or rejects all the traffic to the VMs, with some exceptions for
DHCP, NDP, Router Advertisement, MAC and IP filtering depending on the set
configuration.  The same rules for dropping/rejecting packets are inherited
from the datacenter, while the exceptions for accepted incoming/outgoing
traffic of the host do not apply.</p></div>
<div class="paragraph">
<p>Again, you can use <a href="#pve_firewall_iptables_inspect">iptables-save (see above)</a>
to inspect all rules and chains applied.</p></div>
</div>
</div>
<div class="sect2">
<h3 id="_logging_of_firewall_rules">
<span>13.9. Logging of firewall rules</span>
 <a class="headerlink" href="#_logging_of_firewall_rules" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>By default, all logging of traffic filtered by the firewall rules is disabled.
To enable logging, the <span class="monospaced">loglevel</span> for incoming and/or outgoing traffic has to be
set in <strong>Firewall</strong> → <strong>Options</strong>. This can be done for the host as well as for the
VM/CT firewall individually. By this, logging of Proxmox VE’s standard firewall rules
is enabled and the output can be observed in <strong>Firewall</strong> → <strong>Log</strong>.
Further, only some dropped or rejected packets are logged for the standard rules
(see <a href="#pve_firewall_default_rules">default firewall rules</a>).</p></div>
<div class="paragraph">
<p><span class="monospaced">loglevel</span> does not affect how much of the filtered traffic is logged. It
changes a <span class="monospaced">LOGID</span> appended as prefix to the log output for easier filtering and
post-processing.</p></div>
<div class="paragraph">
<p><span class="monospaced">loglevel</span> is one of the following flags:</p></div>
<table class="tableblock frame-all grid-all" id="pve_firewall_log_levels" style="
width:25%;
">
<colgroup><col style="width:50%;">
<col style="width:50%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top"> loglevel </th>
<th class="tableblock halign-left valign-top"> LOGID</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">nolog</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"> — </p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">emerg</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">alert</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">crit</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">err</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">warning</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">4</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">notice</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">5</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">info</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">6</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">debug</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">7</p></td>
</tr>
</tbody>
</table>
<div class="paragraph">
<p>A typical firewall log output looks like this:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>VMID LOGID CHAIN TIMESTAMP POLICY: PACKET_DETAILS</pre>
</div></div>
<div class="paragraph">
<p>In case of the host firewall, <span class="monospaced">VMID</span> is equal to 0.</p></div>
<div class="sect3">
<h4 id="_logging_of_user_defined_firewall_rules">13.9.1. Logging of user defined firewall rules
 <a class="headerlink" href="#_logging_of_user_defined_firewall_rules" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>In order to log packets filtered by user-defined firewall rules, it is possible
to set a log-level parameter for each rule individually.
This allows to log in a fine grained manner and independent of the log-level
defined for the standard rules in <strong>Firewall</strong> → <strong>Options</strong>.</p></div>
<div class="paragraph">
<p>While the <span class="monospaced">loglevel</span> for each individual rule can be defined or changed easily
in the web UI during creation or modification of the rule, it is possible to set
this also via the corresponding <span class="monospaced">pvesh</span> API calls.</p></div>
<div class="paragraph">
<p>Further, the log-level can also be set via the firewall configuration file by
appending a <span class="monospaced">-log &lt;loglevel&gt;</span> to the selected rule (see
<a href="#pve_firewall_log_levels">possible log-levels</a>).</p></div>
<div class="paragraph">
<p>For example, the following two are identical:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>IN REJECT -p icmp -log nolog
IN REJECT -p icmp</pre>
</div></div>
<div class="paragraph">
<p>whereas</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>IN REJECT -p icmp -log debug</pre>
</div></div>
<div class="paragraph">
<p>produces a log output flagged with the <span class="monospaced">debug</span> level.</p></div>
</div>
</div>
<div class="sect2">
<h3 id="_tips_and_tricks">
<span>13.10. Tips and Tricks</span>
 <a class="headerlink" href="#_tips_and_tricks" title="Permalink to this heading"></a>
</h3>
<div class="sect3">
<h4 id="_how_to_allow_ftp">13.10.1. How to allow FTP
 <a class="headerlink" href="#_how_to_allow_ftp" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>FTP is an old style protocol which uses port 21 and several other dynamic ports. So you
need a rule to accept port 21. In addition, you need to load the <span class="monospaced">ip_conntrack_ftp</span> module.
So please run:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>modprobe ip_conntrack_ftp</pre>
</div></div>
<div class="paragraph">
<p>and add <span class="monospaced">ip_conntrack_ftp</span> to <span class="monospaced">/etc/modules</span> (so that it works after a reboot).</p></div>
</div>
<div class="sect3">
<h4 id="_suricata_ips_integration">13.10.2. Suricata IPS integration
 <a class="headerlink" href="#_suricata_ips_integration" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>If you want to use the <a href="https://suricata.io/">Suricata IPS</a>
(Intrusion Prevention System), it’s possible.</p></div>
<div class="paragraph">
<p>Packets will be forwarded to the IPS only after the firewall ACCEPTed
them.</p></div>
<div class="paragraph">
<p>Rejected/Dropped firewall packets don’t go to the IPS.</p></div>
<div class="paragraph">
<p>Install suricata on proxmox host:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># apt-get install suricata
# modprobe nfnetlink_queue</pre>
</div></div>
<div class="paragraph">
<p>Don’t forget to add <span class="monospaced">nfnetlink_queue</span> to <span class="monospaced">/etc/modules</span> for next reboot.</p></div>
<div class="paragraph">
<p>Then, enable IPS for a specific VM with:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># /etc/pve/firewall/&lt;VMID&gt;.fw

[OPTIONS]
ips: 1
ips_queues: 0</pre>
</div></div>
<div class="paragraph">
<p><span class="monospaced">ips_queues</span> will bind a specific cpu queue for this VM.</p></div>
<div class="paragraph">
<p>Available queues are defined in</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># /etc/default/suricata
NFQUEUE=0</pre>
</div></div>
</div>
</div>
<div class="sect2">
<h3 id="_notes_on_ipv6">
<span>13.11. Notes on IPv6</span>
 <a class="headerlink" href="#_notes_on_ipv6" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>The firewall contains a few IPv6 specific options. One thing to note is that
IPv6 does not use the ARP protocol anymore, and instead uses NDP (Neighbor
Discovery Protocol) which works on IP level and thus needs IP addresses to
succeed. For this purpose link-local addresses derived from the interface’s MAC
address are used. By default the <span class="monospaced">NDP</span> option is enabled on both host and VM
level to allow neighbor discovery (NDP) packets to be sent and received.</p></div>
<div class="paragraph">
<p>Beside neighbor discovery NDP is also used for a couple of other things, like
auto-configuration and advertising routers.</p></div>
<div class="paragraph">
<p>By default VMs are allowed to send out router solicitation messages (to query
for a router), and to receive router advertisement packets. This allows them to
use stateless auto configuration. On the other hand VMs cannot advertise
themselves as routers unless the “Allow Router Advertisement” (<span class="monospaced">radv: 1</span>) option
is set.</p></div>
<div class="paragraph">
<p>As for the link local addresses required for NDP, there’s also an “IP Filter”
(<span class="monospaced">ipfilter: 1</span>) option which can be enabled which has the same effect as adding
an <span class="monospaced">ipfilter-net*</span> ipset for each of the VM’s network interfaces containing the
corresponding link local addresses.  (See the
<a href="#pve_firewall_ipfilter_section">Standard IP set <span class="monospaced">ipfilter-net*</span></a> section for details.)</p></div>
</div>
<div class="sect2">
<h3 id="_ports_used_by_proxmox_ve">
<span>13.12. Ports used by Proxmox VE</span>
 <a class="headerlink" href="#_ports_used_by_proxmox_ve" title="Permalink to this heading"></a>
</h3>
<div class="ulist"><ul>
<li>
<p>
Web interface: 8006 (TCP, HTTP/1.1 over TLS)
</p>
</li>
<li>
<p>
VNC Web console: 5900-5999 (TCP, WebSocket)
</p>
</li>
<li>
<p>
SPICE proxy: 3128 (TCP)
</p>
</li>
<li>
<p>
sshd (used for cluster actions): 22 (TCP)
</p>
</li>
<li>
<p>
rpcbind: 111 (UDP)
</p>
</li>
<li>
<p>
sendmail: 25 (TCP, outgoing)
</p>
</li>
<li>
<p>
corosync cluster traffic: 5405-5412 UDP
</p>
</li>
<li>
<p>
live migration (VM memory and local-disk data): 60000-60050 (TCP)
</p>
</li>
</ul></div>
</div>
<div class="sect2">
<h3 id="_nftables">
<span>13.13. nftables</span>
 <a class="headerlink" href="#_nftables" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>As an alternative to <span class="monospaced">pve-firewall</span> we offer <span class="monospaced">proxmox-firewall</span>, which is an
implementation of the Proxmox VE firewall based on the newer
<a href="https://wiki.nftables.org/wiki-nftables/index.php/What_is_nftables%3F">nftables</a>
rather than iptables.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content"><span class="monospaced">proxmox-firewall</span> is currently in tech preview. There might be bugs or
incompatibilies with the original firewall. It is currently not suited for
production use.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>This implementation uses the same configuration files and configuration format,
so you can use your old configuration when switching. It provides the exact same
functionality with a few exceptions:</p></div>
<div class="ulist"><ul>
<li>
<p>
REJECT is currently not possible for guest traffic (traffic will instead be
  dropped).
</p>
</li>
<li>
<p>
Using the <span class="monospaced">NDP</span>, <span class="monospaced">Router Advertisement</span> or <span class="monospaced">DHCP</span> options will <strong>always</strong> create
  firewall rules, irregardless of your default policy.
</p>
</li>
<li>
<p>
firewall rules for guests are evaluated even for connections that have
  conntrack table entries.
</p>
</li>
</ul></div>
<div class="sect3">
<h4 id="_installation_and_usage">13.13.1. Installation and Usage
 <a class="headerlink" href="#_installation_and_usage" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Install the <span class="monospaced">proxmox-firewall</span> package:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>apt install proxmox-firewall</pre>
</div></div>
<div class="paragraph">
<p>Enable the nftables backend via the Web UI on your hosts (Host &gt; Firewall &gt;
Options &gt; nftables), or by enabling it in the configuration file for your hosts
(<span class="monospaced">/etc/pve/nodes/&lt;node_name&gt;/host.fw</span>):</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>[OPTIONS]

nftables: 1</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">After enabling/disabling <span class="monospaced">proxmox-firewall</span>, all running VMs and
containers need to be restarted for the old/new firewall to work properly.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>After setting the <span class="monospaced">nftables</span> configuration key, the new <span class="monospaced">proxmox-firewall</span>
service will take over. You can check if the new service is working by
checking the systemctl status of <span class="monospaced">proxmox-firewall</span>:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>systemctl status proxmox-firewall</pre>
</div></div>
<div class="paragraph">
<p>You can also examine the generated ruleset. You can find more information about
this in the section <a href="#pve_firewall_nft_helpful_commands">Helpful Commands</a>.
You should also check whether <span class="monospaced">pve-firewall</span> is no longer generating iptables
rules, you can find the respective commands in the
<a href="#pve_firewall_services_commands">Services and Commands</a> section.</p></div>
<div class="paragraph">
<p>Switching back to the old firewall can be done by simply setting the
configuration value back to 0 / No.</p></div>
</div>
<div class="sect3">
<h4 id="_usage">13.13.2. Usage
 <a class="headerlink" href="#_usage" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p><span class="monospaced">proxmox-firewall</span> will create two tables that are managed by the
<span class="monospaced">proxmox-firewall</span> service: <span class="monospaced">proxmox-firewall</span> and <span class="monospaced">proxmox-firewall-guests</span>. If
you want to create custom rules that live outside the Proxmox VE firewall
configuration you can create your own tables to manage your custom firewall
rules. <span class="monospaced">proxmox-firewall</span> will only touch the tables it generates, so you can
easily extend and modify the behavior of the <span class="monospaced">proxmox-firewall</span> by adding your
own tables.</p></div>
<div class="paragraph">
<p>Instead of using the <span class="monospaced">pve-firewall</span> command, the nftables-based firewall uses
<span class="monospaced">proxmox-firewall</span>. It is a systemd service, so you can start and stop it via
<span class="monospaced">systemctl</span>:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>systemctl start proxmox-firewall
systemctl stop proxmox-firewall</pre>
</div></div>
<div class="paragraph">
<p>Stopping the firewall service will remove all generated rules.</p></div>
<div class="paragraph">
<p>To query the status of the firewall, you can query the status of the systemctl
service:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>systemctl status proxmox-firewall</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="pve_firewall_nft_helpful_commands">13.13.3. Helpful Commands
 <a class="headerlink" href="#pve_firewall_nft_helpful_commands" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>You can check the generated ruleset via the following command:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>nft list ruleset</pre>
</div></div>
<div class="paragraph">
<p>If you want to debug <span class="monospaced">proxmox-firewall</span> you can simply run the daemon in
foreground with the <span class="monospaced">RUST_LOG</span> environment variable set to <span class="monospaced">trace</span>. This should
provide you with detailed debugging output:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>RUST_LOG=trace /usr/libexec/proxmox/proxmox-firewall</pre>
</div></div>
<div class="paragraph">
<p>You can also edit the systemctl service if you want to have detailed output for
your firewall daemon:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>systemctl edit proxmox-firewall</pre>
</div></div>
<div class="paragraph">
<p>Then you need to add the override for the <span class="monospaced">RUST_LOG</span> environment variable:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>[Service]
Environment="RUST_LOG=trace"</pre>
</div></div>
<div class="paragraph">
<p>This will generate a large amount of logs very quickly, so only use this for
debugging purposes. Other, less verbose, log levels are <span class="monospaced">info</span> and <span class="monospaced">debug</span>.</p></div>
<div class="paragraph">
<p>Running in foreground writes the log output to STDERR, so you can redirect it
with the following command (e.g. for submitting logs to the community forum):</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>RUST_LOG=trace /usr/libexec/proxmox/proxmox-firewall 2&gt; firewall_log_$(hostname).txt</pre>
</div></div>
<div class="paragraph">
<p>It can be helpful to trace packet flow through the different chains in order to
debug firewall rules. This can be achieved by setting <span class="monospaced">nftrace</span> to 1 for packets
that you want to track. It is advisable that you do not set this flag for <strong>all</strong>
packets, in the example below we only examine ICMP packets.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>#!/usr/sbin/nft -f
table bridge tracebridge
delete table bridge tracebridge

table bridge tracebridge {
    chain trace {
        meta l4proto icmp meta nftrace set 1
    }

    chain prerouting {
        type filter hook prerouting priority -350; policy accept;
        jump trace
    }

    chain postrouting {
        type filter hook postrouting priority -350; policy accept;
        jump trace
    }
}</pre>
</div></div>
<div class="paragraph">
<p>Saving this file, making it executable, and then running it once will create the
respective tracing chains. You can then inspect the tracing output via the
Proxmox VE Web UI (Firewall &gt; Log) or via <span class="monospaced">nft monitor trace</span>.</p></div>
<div class="paragraph">
<p>The above example traces traffic on all bridges, which is usually where guest
traffic flows through. If you want to examine host traffic, create those chains
in the <span class="monospaced">inet</span> table instead of the <span class="monospaced">bridge</span> table.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Be aware that this can generate a <strong>lot</strong> of log spam and slow down the
performance of your networking stack significantly.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>You can remove the tracing rules via running the following command:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>nft delete table bridge tracebridge</pre>
</div></div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="user_mgmt">
14. User Management
 <a class="headerlink" href="#user_mgmt" title="Permalink to this heading"></a>
</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Proxmox VE supports multiple authentication sources, for example Linux PAM,
an integrated Proxmox VE authentication server, LDAP, Microsoft Active
Directory and OpenID Connect.</p></div>
<div class="paragraph">
<p>By using role-based user and permission management for all objects (VMs,
Storage, nodes, etc.), granular access can be defined.</p></div>
<div class="sect2">
<h3 id="pveum_users">
<span>14.1. Users</span>
 <a class="headerlink" href="#pveum_users" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Proxmox VE stores user attributes in <span class="monospaced">/etc/pve/user.cfg</span>.
Passwords are not stored here; users are instead associated with the
<a href="#pveum_authentication_realms">authentication realms</a> described below.
Therefore, a user is often internally identified by their username and
realm in the form <span class="monospaced">&lt;userid&gt;@&lt;realm&gt;</span>.</p></div>
<div class="paragraph">
<p>Each user entry in this file contains the following information:</p></div>
<div class="ulist"><ul>
<li>
<p>
First name
</p>
</li>
<li>
<p>
Last name
</p>
</li>
<li>
<p>
E-mail address
</p>
</li>
<li>
<p>
Group memberships
</p>
</li>
<li>
<p>
An optional expiration date
</p>
</li>
<li>
<p>
A comment or note about this user
</p>
</li>
<li>
<p>
Whether this user is enabled or disabled
</p>
</li>
<li>
<p>
Optional two-factor authentication keys
</p>
</li>
</ul></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Caution" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKdUlEQVRoge1Ze1AV1x3+zt279wEi
DWCYGzRVktqa1MEmmtbWR22ncXxUrTDWV/5IqG2wUAUfmUwSkk6V+EAJCEQC6figwZBqZtRxqukf
tebRZBpFG1S0hiRErwiaKK977+6eX//YPXt37+UiIJlMZnpmdnb33PP4vt97z2VEhG9yc3zdAO60
/Z/A192+8QScX8Wifr+fWltbzffU1FT4fD72Vew15ASampqovr4eBw8eNPvGjRuHzMxMmj9//tCT
IKIhu958801yuVwEoNdr48aNNJT7EdHQEdi/fz/JshwTvLiKioqGlMSQLFJfX0+MMRvQsWNSqXLz
H2jez78fRWLTpk1DRuKOF6irqyOn02kDOP8XGdTz+VFSP91Hatu79NRvZ0SR2LJly5CQuKPJtbW1
JEmSDdivHp1AwctHSblYSsqFElLOF5PavIvWZU+LIrF169Y7JjHoibt3744CnzXrIQpeOUbKhVJS
mkpIPV9MyrnNpJwtIuVSNa15/MdRJLZt23ZHJBjRwIu5N954gxYvXgzOudm3aO5E1L5SBNZ1Hoxr
INLAyH6HJxVP/WkXtu9+37ZeSUkJVq9ePagQO+BMXFNTEwV+ybxHsPeVIjg6zwOkgSWNhuOuUUBC
KhxJY0AggDSgpwWbn1mO1csftq1ZUFCA0tLSQZXFA0pkVVVVlJOTA6vWsmY9hF07N0AS4OU48LgR
WLO+FPFxboxMS8G5hlPYmPMI4jwM1P0Jtj67DJyrKHvtNADdjPPz8+FwOCgvL29gmuivrVVUVESF
yqxZD1PoyjFSL+7QHbapmJTml+n9EzVRtl676ZekNKwl5VQ+KSfzKHR2E+UutodYxhiVl5cPyCf6
ZUJlZWWUm5trk/wTi6bgtVdf1M2Gq2DEwUgDNBWhUDBqjfSRwwGoIOEP3c3Y/uxirFw0zibMvLw8
VFZW9tucbkugoqKCVq1aZQO/YslUVJU+D9bVBIDroLgK4hygEDweybYGY8AD6d8yftfASL+j+2O8
9NwiPJk51kYiNzcXO3fu7BeJPgkUFxdTbm6ure/JZdNRub0QrOMcGFcBrgLgAHEAGqD2ID7OZZsz
Ji0RcR6HAV4DkaqPJxXouoSywkz8buF9NhIrV65EVVXVbUnEJHDixAlat26drS9n+U+xo/hZMMNh
yZA+uAbAIKOGMCJlmG3eA/clAYaJEamGBlRdC6QBXZewo3AhVswfbSORk5ODDz/8sE8SvRLw+/1U
XV0dBb5MgOe6LUNIEqpOwgCUlOCESw4HuAfG3KX7B2kAGaTJ0Jro77yI8ucX4DfzRtlIHD58GH6/
PyaJXgk0NjZi79695vuUiffrkr91Vt8cHAwc4FZJGiZEKqDcwsi0ZHP+d749POy8MS7iGtBxARWF
8/CDsQnm3BdeeAHWj6N+EYhspSVbgI5zYYkZGmAwzIEbwA0tUM9NfO+7aeb89HuGRQBWLcSNZxj3
zvOYMzk5NpjBEHi99mXD5lXdjqGDJh42h7CEOXhnK3448X5z/shUb1jSpIEM8yFuzOUcjAsNcnR0
KXdG4MEHH8TEiRPN9z0H3sfRE41m+GNcs9i0LjkGTTcrUoGem5g7/V5z/ohElw4O+jxGYQ3qJqmB
jN/+cbINtcfCJvPYY4+hr3qtVwI+n49lZmaa71fbbuH3hX/FycbPQKSCoIIMQEQcehjVo5Gw8/F3
3cSvH03H2HuHY1icQzcvYWKGIMiITMwQwjtnruPpqk9w/ZZq7p2RkYEJEyaAYrCIWY36/X7KyMhA
W1ub2XfP3Qmo3ZKFaQ+P0gmYIdSoOkU4lWSwxGRAdqHL/xm86k27w0blAw3/PncD6yo/xrsfdZj7
paSkoLGxESNGjAgDZsxWK8X0AZ/Px44fP460tLAzXrnWgUX5r+PYuxf1mG/Gf80sofWy2YuWbg/+
uONtPLHhPRw+0WIHL8hbwG95rcUGPiEhAYcOHUJSUhI456YZRWqiz+8BIqLm5mZMmjQJN27cMPsT
E9zYtWE25k4bYzh3mABIBRwOHPgPx+LH/6yDiXPi6pHZcEpk+I5qmtGpCzdQecCPPUevmevLsowj
R45gypQpcDgc5mVowKaJmBoQTEePHo2GhgaMGxcuum52BLHsqUOoP3oWgLB9Sz7QgpgxJoThw/SS
IutnaXBKZEpcgD998Uvs+3ubDbzT6URNTQ0mTZoEVVWhaRo459A0TeCy4etVA+JH8RvnHK2trZgz
Zw5Onz5tjnPJDrxSOAPLZt1nSVIqwDmIVJw6dx1nLt3A3J/cjZThkhE29cvf3o2KA59ja91lcz3G
GDZv3oylS5fC7XbD6XTC6XRCkiRIktS7Jm5HQLDXNA3Xr19HVlYWPvjgA4vEGLav+RHmTxsJX4rb
8AsO4kbOIG6GzTD4HvzlmB/PVH9q2zc/Px/Z2dnwer1wu92QZRkulwuSJEGWZZMEY8wkEGVCVvCC
AOcciqIgPj4e+/btw9SpU83xqkpYteU9HHm7Bf62rnC2RjikisTFSMPV9m4ceucannvVDn7p0qVY
uHAhgsEggsEgQqFQlAlZnVm0PjOxIKFpGogIqqrC5XKhuroa06dPt4wDcl78F+qPNeNKe6cRoTjs
ZYOKK+3dON7wBQrKm2H5pMbMmTOxZMkSBAIB9PT0oKenB4FAAIqiQFEUU4hRRyq3I2AlIZ4553A6
nSgvL8fs2bNtY9eVNaDub58YmhDZWpe+v70bJ5tu4clt/4WihqU4efJkrFixAkRkAg4Gg1AUBaqq
muDF/pGtX7WQNXeIZ0mSUFRUhAULFtjGPl3ZiJdev4QrbZ16zUQqrrZ342JLN5ZvaEJ3ICz68ePH
IycnB5qmmZIW4EOhUJTZWEKo+Rx1KsEYY9Zk4XA4IEmSKXnOuRkVJElCYWEhvF4v6urqzDVK9n2M
zu4Qnl5+L9q/7MHltgBWbr9kA5+eno7c3FzIsoyI5GrTeCzgosU8VmGMmSGLiEwSsixH2WNBQQEA
2EhUH/wc7330BaaOH479/2zHtS/CFabP58OaNWvg9Xp1EE4nZFmGLMtwu93wer1wuVy2ECpJko2I
iTNWJrZGI+HEwi6FisUVCAQQCASwZ88eVFZW9lk9JicnY/369fD5fPB4PCbo+Ph4k4TH44HL5YLX
64XX6zXH9EKE9aUBRkTEGLNJP1Kd4nI4HMjOzkZ8fDyKi4ttJ3eiJSYmYu3atRg1apQJzuv1wuPx
wOPxwOFwmEBFHnA6nXC5XGYSiywlbns2SnoDANN0hMNZnS0UCpkaOnPmDN566y20tLSY2ktLS8OM
GTOQkpICt9uNuLg4M2EJ8CJZSZJkZmFBwkrAWpH2+3BXEBEkRIhTVdVMOCL5qKqKy5cvQ1EUaJoG
VVWRnKx/Jlpt3eVy2QCKu9C68AFr9o0spwd0Oh1ZYggCIlNaiQktiXdhZgKQABcp3ci7LMtM7B0J
fsAErJoAYItG1mdr+hfvVlACtHDIXkploYXbHvQOikDEe6/ZmjFmAhf9ZvJxOk2Qol88W4kYz32S
GNQfHLG+TyMTkAVEr88xQfVnkBg7GAKixSIykDYQsL21/wFkW/B5QqT9lwAAAABJRU5ErkJggg==">
</td>
<td class="content">When you disable or delete a user, or if the expiry date set is
in the past, this user will not be able to log in to new sessions or start new
tasks. All tasks which have already been started by this user (for example,
terminal sessions) will <strong>not</strong> be terminated automatically by any such event.</td>
</tr></tbody></table>
</div>
<div class="sect3">
<h4 id="_system_administrator">14.1.1. System administrator
 <a class="headerlink" href="#_system_administrator" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The system’s root user can always log in via the Linux PAM realm and is an
unconfined administrator. This user cannot be deleted, but attributes can
still be changed. System mails will be sent to the email address
assigned to this user.</p></div>
</div>
</div>
<div class="sect2">
<h3 id="pveum_groups">
<span>14.2. Groups</span>
 <a class="headerlink" href="#pveum_groups" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Each user can be a member of several groups. Groups are the preferred
way to organize access permissions. You should always grant permissions
to groups instead of individual users. That way you will get a
much more maintainable access control list.</p></div>
</div>
<div class="sect2">
<h3 id="pveum_tokens">
<span>14.3. API Tokens</span>
 <a class="headerlink" href="#pveum_tokens" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>API tokens allow stateless access to most parts of the REST API from another
system, software or API client. Tokens can be generated for individual users
and can be given separate permissions and expiration dates to limit the scope
and duration of the access. Should the API token get compromised, it can be
revoked without disabling the user itself.</p></div>
<div class="paragraph">
<p>API tokens come in two basic types:</p></div>
<div class="ulist"><ul>
<li>
<p>
Separated privileges: The token needs to be given explicit access with ACLs.
  Its effective permissions are calculated by intersecting user and token
  permissions.
</p>
</li>
<li>
<p>
Full privileges: The token’s permissions are identical to that of the
  associated user.
</p>
</li>
</ul></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Caution" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKdUlEQVRoge1Ze1AV1x3+zt279wEi
DWCYGzRVktqa1MEmmtbWR22ncXxUrTDWV/5IqG2wUAUfmUwSkk6V+EAJCEQC6figwZBqZtRxqukf
tebRZBpFG1S0hiRErwiaKK977+6eX//YPXt37+UiIJlMZnpmdnb33PP4vt97z2VEhG9yc3zdAO60
/Z/A192+8QScX8Wifr+fWltbzffU1FT4fD72Vew15ASampqovr4eBw8eNPvGjRuHzMxMmj9//tCT
IKIhu958801yuVwEoNdr48aNNJT7EdHQEdi/fz/JshwTvLiKioqGlMSQLFJfX0+MMRvQsWNSqXLz
H2jez78fRWLTpk1DRuKOF6irqyOn02kDOP8XGdTz+VFSP91Hatu79NRvZ0SR2LJly5CQuKPJtbW1
JEmSDdivHp1AwctHSblYSsqFElLOF5PavIvWZU+LIrF169Y7JjHoibt3744CnzXrIQpeOUbKhVJS
mkpIPV9MyrnNpJwtIuVSNa15/MdRJLZt23ZHJBjRwIu5N954gxYvXgzOudm3aO5E1L5SBNZ1Hoxr
INLAyH6HJxVP/WkXtu9+37ZeSUkJVq9ePagQO+BMXFNTEwV+ybxHsPeVIjg6zwOkgSWNhuOuUUBC
KhxJY0AggDSgpwWbn1mO1csftq1ZUFCA0tLSQZXFA0pkVVVVlJOTA6vWsmY9hF07N0AS4OU48LgR
WLO+FPFxboxMS8G5hlPYmPMI4jwM1P0Jtj67DJyrKHvtNADdjPPz8+FwOCgvL29gmuivrVVUVESF
yqxZD1PoyjFSL+7QHbapmJTml+n9EzVRtl676ZekNKwl5VQ+KSfzKHR2E+UutodYxhiVl5cPyCf6
ZUJlZWWUm5trk/wTi6bgtVdf1M2Gq2DEwUgDNBWhUDBqjfSRwwGoIOEP3c3Y/uxirFw0zibMvLw8
VFZW9tucbkugoqKCVq1aZQO/YslUVJU+D9bVBIDroLgK4hygEDweybYGY8AD6d8yftfASL+j+2O8
9NwiPJk51kYiNzcXO3fu7BeJPgkUFxdTbm6ure/JZdNRub0QrOMcGFcBrgLgAHEAGqD2ID7OZZsz
Ji0RcR6HAV4DkaqPJxXouoSywkz8buF9NhIrV65EVVXVbUnEJHDixAlat26drS9n+U+xo/hZMMNh
yZA+uAbAIKOGMCJlmG3eA/clAYaJEamGBlRdC6QBXZewo3AhVswfbSORk5ODDz/8sE8SvRLw+/1U
XV0dBb5MgOe6LUNIEqpOwgCUlOCESw4HuAfG3KX7B2kAGaTJ0Jro77yI8ucX4DfzRtlIHD58GH6/
PyaJXgk0NjZi79695vuUiffrkr91Vt8cHAwc4FZJGiZEKqDcwsi0ZHP+d749POy8MS7iGtBxARWF
8/CDsQnm3BdeeAHWj6N+EYhspSVbgI5zYYkZGmAwzIEbwA0tUM9NfO+7aeb89HuGRQBWLcSNZxj3
zvOYMzk5NpjBEHi99mXD5lXdjqGDJh42h7CEOXhnK3448X5z/shUb1jSpIEM8yFuzOUcjAsNcnR0
KXdG4MEHH8TEiRPN9z0H3sfRE41m+GNcs9i0LjkGTTcrUoGem5g7/V5z/ohElw4O+jxGYQ3qJqmB
jN/+cbINtcfCJvPYY4+hr3qtVwI+n49lZmaa71fbbuH3hX/FycbPQKSCoIIMQEQcehjVo5Gw8/F3
3cSvH03H2HuHY1icQzcvYWKGIMiITMwQwjtnruPpqk9w/ZZq7p2RkYEJEyaAYrCIWY36/X7KyMhA
W1ub2XfP3Qmo3ZKFaQ+P0gmYIdSoOkU4lWSwxGRAdqHL/xm86k27w0blAw3/PncD6yo/xrsfdZj7
paSkoLGxESNGjAgDZsxWK8X0AZ/Px44fP460tLAzXrnWgUX5r+PYuxf1mG/Gf80sofWy2YuWbg/+
uONtPLHhPRw+0WIHL8hbwG95rcUGPiEhAYcOHUJSUhI456YZRWqiz+8BIqLm5mZMmjQJN27cMPsT
E9zYtWE25k4bYzh3mABIBRwOHPgPx+LH/6yDiXPi6pHZcEpk+I5qmtGpCzdQecCPPUevmevLsowj
R45gypQpcDgc5mVowKaJmBoQTEePHo2GhgaMGxcuum52BLHsqUOoP3oWgLB9Sz7QgpgxJoThw/SS
IutnaXBKZEpcgD998Uvs+3ubDbzT6URNTQ0mTZoEVVWhaRo459A0TeCy4etVA+JH8RvnHK2trZgz
Zw5Onz5tjnPJDrxSOAPLZt1nSVIqwDmIVJw6dx1nLt3A3J/cjZThkhE29cvf3o2KA59ja91lcz3G
GDZv3oylS5fC7XbD6XTC6XRCkiRIktS7Jm5HQLDXNA3Xr19HVlYWPvjgA4vEGLav+RHmTxsJX4rb
8AsO4kbOIG6GzTD4HvzlmB/PVH9q2zc/Px/Z2dnwer1wu92QZRkulwuSJEGWZZMEY8wkEGVCVvCC
AOcciqIgPj4e+/btw9SpU83xqkpYteU9HHm7Bf62rnC2RjikisTFSMPV9m4ceucannvVDn7p0qVY
uHAhgsEggsEgQqFQlAlZnVm0PjOxIKFpGogIqqrC5XKhuroa06dPt4wDcl78F+qPNeNKe6cRoTjs
ZYOKK+3dON7wBQrKm2H5pMbMmTOxZMkSBAIB9PT0oKenB4FAAIqiQFEUU4hRRyq3I2AlIZ4553A6
nSgvL8fs2bNtY9eVNaDub58YmhDZWpe+v70bJ5tu4clt/4WihqU4efJkrFixAkRkAg4Gg1AUBaqq
muDF/pGtX7WQNXeIZ0mSUFRUhAULFtjGPl3ZiJdev4QrbZ16zUQqrrZ342JLN5ZvaEJ3ICz68ePH
IycnB5qmmZIW4EOhUJTZWEKo+Rx1KsEYY9Zk4XA4IEmSKXnOuRkVJElCYWEhvF4v6urqzDVK9n2M
zu4Qnl5+L9q/7MHltgBWbr9kA5+eno7c3FzIsoyI5GrTeCzgosU8VmGMmSGLiEwSsixH2WNBQQEA
2EhUH/wc7330BaaOH479/2zHtS/CFabP58OaNWvg9Xp1EE4nZFmGLMtwu93wer1wuVy2ECpJko2I
iTNWJrZGI+HEwi6FisUVCAQQCASwZ88eVFZW9lk9JicnY/369fD5fPB4PCbo+Ph4k4TH44HL5YLX
64XX6zXH9EKE9aUBRkTEGLNJP1Kd4nI4HMjOzkZ8fDyKi4ttJ3eiJSYmYu3atRg1apQJzuv1wuPx
wOPxwOFwmEBFHnA6nXC5XGYSiywlbns2SnoDANN0hMNZnS0UCpkaOnPmDN566y20tLSY2ktLS8OM
GTOQkpICt9uNuLg4M2EJ8CJZSZJkZmFBwkrAWpH2+3BXEBEkRIhTVdVMOCL5qKqKy5cvQ1EUaJoG
VVWRnKx/Jlpt3eVy2QCKu9C68AFr9o0spwd0Oh1ZYggCIlNaiQktiXdhZgKQABcp3ci7LMtM7B0J
fsAErJoAYItG1mdr+hfvVlACtHDIXkploYXbHvQOikDEe6/ZmjFmAhf9ZvJxOk2Qol88W4kYz32S
GNQfHLG+TyMTkAVEr88xQfVnkBg7GAKixSIykDYQsL21/wFkW/B5QqT9lwAAAABJRU5ErkJggg==">
</td>
<td class="content">The token value is only displayed/returned once when the token is
generated. It cannot be retrieved again over the API at a later time!</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>To use an API token, set the HTTP header <em>Authorization</em> to the displayed value
of the form <span class="monospaced">PVEAPIToken=USER@REALM!TOKENID=UUID</span> when making API requests, or
refer to your API client’s documentation.</p></div>
</div>
<div class="sect2">
<h3 id="pveum_resource_pools">
<span>14.4. Resource Pools</span>
 <a class="headerlink" href="#pveum_resource_pools" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-datacenter-pool-window.png">
<img src="images/screenshot/gui-datacenter-pool-window.png" alt="screenshot/gui-datacenter-pool-window.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>A resource pool is a set of virtual machines, containers, and storage
devices. It is useful for permission handling in cases where certain users
should have controlled access to a specific set of resources, as it allows for a
single permission to be applied to a set of elements, rather than having to
manage this on a per-resource basis. Resource pools are often used in tandem
with groups, so that the members of a group have permissions on a set of
machines and storage.</p></div>
</div>
<div class="sect2">
<h3 id="pveum_authentication_realms">
<span>14.5. Authentication Realms</span>
 <a class="headerlink" href="#pveum_authentication_realms" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>As Proxmox VE users are just counterparts for users existing on some external
realm, the realms have to be configured in <span class="monospaced">/etc/pve/domains.cfg</span>.
The following realms (authentication methods) are available:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
Linux PAM Standard Authentication
</dt>
<dd>
<p>
Linux PAM is a framework for system-wide user authentication. These users are
created on the host system with commands such as <span class="monospaced">adduser</span>. If PAM users exist
on the Proxmox VE host system, corresponding entries can be added to Proxmox VE, to allow
these users to log in via their system username and password.
</p>
</dd>
<dt class="hdlist1">
Proxmox VE Authentication Server
</dt>
<dd>
<p>
This is a Unix-like password store, which stores hashed passwords in
<span class="monospaced">/etc/pve/priv/shadow.cfg</span>. Passwords are hashed using the SHA-256 hashing
algorithm. This is the most convenient realm for small-scale (or even
mid-scale) installations, where users do not need access to anything outside of
Proxmox VE. In this case, users are fully managed by Proxmox VE and are able to change
their own passwords via the GUI.
</p>
</dd>
<dt class="hdlist1">
LDAP
</dt>
<dd>
<p>
LDAP (Lightweight Directory Access Protocol) is an open, cross-platform protocol
for authentication using directory services. OpenLDAP is a popular open-source
implementations of the LDAP protocol.
</p>
</dd>
<dt class="hdlist1">
Microsoft Active Directory (AD)
</dt>
<dd>
<p>
Microsoft Active Directory (AD) is a directory service for Windows domain
networks and is supported as an authentication realm for Proxmox VE. It supports LDAP
as an authentication protocol.
</p>
</dd>
<dt class="hdlist1">
OpenID Connect
</dt>
<dd>
<p>
OpenID Connect is implemented as an identity layer on top of the OATH 2.0
protocol. It allows clients to verify the identity of the user, based on
authentication performed by an external authorization server.
</p>
</dd>
</dl></div>
<div class="sect3">
<h4 id="user-realms-pam">14.5.1. Linux PAM Standard Authentication
 <a class="headerlink" href="#user-realms-pam" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>As Linux PAM corresponds to host system users, a system user must exist on each
node which the user is allowed to log in on. The user authenticates with their
usual system password. This realm is added by default and can’t be removed. In
terms of configurability, an administrator can choose to require two-factor
authentication with logins from the realm and to set the realm as the default
authentication realm.</p></div>
</div>
<div class="sect3">
<h4 id="user-realms-pve">14.5.2. Proxmox VE Authentication Server
 <a class="headerlink" href="#user-realms-pve" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The Proxmox VE authentication server realm is a simple Unix-like password store.
The realm is created by default, and as with Linux PAM, the only configuration
items available are the ability to require two-factor authentication for users
of the realm, and to set it as the default realm for login.</p></div>
<div class="paragraph">
<p>Unlike the other Proxmox VE realm types, users are created and authenticated entirely
through Proxmox VE, rather than authenticating against another system. Hence, you are
required to set a password for this type of user upon creation.</p></div>
</div>
<div class="sect3">
<h4 id="user-realms-ldap">14.5.3. LDAP
 <a class="headerlink" href="#user-realms-ldap" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>You can also use an external LDAP server for user authentication (for examle,
OpenLDAP). In this realm type, users are searched under a <em>Base Domain Name</em>
(<span class="monospaced">base_dn</span>), using the username attribute specified in the <em>User Attribute Name</em>
(<span class="monospaced">user_attr</span>) field.</p></div>
<div class="paragraph">
<p>A server and optional fallback server can be configured, and the connection can
be encrypted via SSL. Furthermore, filters can be configured for directories and
groups. Filters allow you to further limit the scope of the realm.</p></div>
<div class="paragraph">
<p>For instance, if a user is represented via the following LDIF dataset:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># user1 of People at ldap-test.com
dn: uid=user1,ou=People,dc=ldap-test,dc=com
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
uid: user1
cn: Test User 1
sn: Testers
description: This is the first test user.</pre>
</div></div>
<div class="paragraph">
<p>The <em>Base Domain Name</em> would be <span class="monospaced">ou=People,dc=ldap-test,dc=com</span> and the user
attribute would be <span class="monospaced">uid</span>.</p></div>
<div class="paragraph">
<p>If Proxmox VE needs to authenticate (bind) to the LDAP server before being
able to query and authenticate users, a bind domain name can be
configured via the <span class="monospaced">bind_dn</span> property in <span class="monospaced">/etc/pve/domains.cfg</span>. Its
password then has to be stored in <span class="monospaced">/etc/pve/priv/ldap/&lt;realmname&gt;.pw</span>
(for example, <span class="monospaced">/etc/pve/priv/ldap/my-ldap.pw</span>). This file should contain a
single line with the raw password.</p></div>
<div class="paragraph">
<p>To verify certificates, you need to set <span class="monospaced">capath</span>. You can set it either
directly to the CA certificate of your LDAP server, or to the system path
containing all trusted CA certificates (<span class="monospaced">/etc/ssl/certs</span>).
Additionally, you need to set the <span class="monospaced">verify</span> option, which can also be done over
the web interface.</p></div>
<div class="paragraph">
<p>The main configuration options for an LDAP server realm are as follows:</p></div>
<div class="ulist"><ul>
<li>
<p>
<span class="monospaced">Realm</span> (<span class="monospaced">realm</span>): The realm identifier for Proxmox VE users
</p>
</li>
<li>
<p>
<span class="monospaced">Base Domain Name</span> (<span class="monospaced">base_dn</span>): The directory which users are searched under
</p>
</li>
<li>
<p>
<span class="monospaced">User Attribute Name</span> (<span class="monospaced">user_attr</span>): The LDAP attribute containing the
  username that users will log in with
</p>
</li>
<li>
<p>
<span class="monospaced">Server</span> (<span class="monospaced">server1</span>): The server hosting the LDAP directory
</p>
</li>
<li>
<p>
<span class="monospaced">Fallback Server</span> (<span class="monospaced">server2</span>): An optional fallback server address, in case
  the primary server is unreachable
</p>
</li>
<li>
<p>
<span class="monospaced">Port</span> (<span class="monospaced">port</span>): The port that the LDAP server listens on
</p>
</li>
</ul></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">In order to allow a particular user to authenticate using the LDAP server,
you must also add them as a user of that realm from the Proxmox VE server. This can
be carried out automatically with <a href="#pveum_ldap_sync">syncing</a>.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="user-realms-ad">14.5.4. Microsoft Active Directory (AD)
 <a class="headerlink" href="#user-realms-ad" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>To set up Microsoft AD as a realm, a server address and authentication domain
need to be specified. Active Directory supports most of the same properties as
LDAP, such as an optional fallback server, port, and SSL encryption.
Furthermore, users can be added to Proxmox VE automatically via
<a href="#pveum_ldap_sync">sync</a> operations, after configuration.</p></div>
<div class="paragraph">
<p>As with LDAP, if Proxmox VE needs to authenticate before it binds to the AD server,
you must configure the <em>Bind User</em> (<span class="monospaced">bind_dn</span>) property. This property is
typically required by default for Microsoft AD.</p></div>
<div class="paragraph">
<p>The main configuration settings for Microsoft Active Directory are:</p></div>
<div class="ulist"><ul>
<li>
<p>
<span class="monospaced">Realm</span> (<span class="monospaced">realm</span>): The realm identifier for Proxmox VE users
</p>
</li>
<li>
<p>
<span class="monospaced">Domain</span> (<span class="monospaced">domain</span>): The AD domain of the server
</p>
</li>
<li>
<p>
<span class="monospaced">Server</span> (<span class="monospaced">server1</span>): The FQDN or IP address of the server
</p>
</li>
<li>
<p>
<span class="monospaced">Fallback Server</span> (<span class="monospaced">server2</span>): An optional fallback server address, in case
  the primary server is unreachable
</p>
</li>
<li>
<p>
<span class="monospaced">Port</span> (<span class="monospaced">port</span>): The port that the Microsoft AD server listens on
</p>
</li>
</ul></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Microsoft AD normally checks values like usernames without case
sensitivity. To make Proxmox VE do the same, you can disable the default
<span class="monospaced">case-sensitive</span> option by editing the realm in the web UI, or using the CLI
(change the <span class="monospaced">ID</span> with the realm ID):
<span class="monospaced">pveum realm modify ID --case-sensitive 0</span></td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="pveum_ldap_sync">14.5.5. Syncing LDAP-Based Realms
 <a class="headerlink" href="#pveum_ldap_sync" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-datacenter-realm-add-ldap.png">
<img src="images/screenshot/gui-datacenter-realm-add-ldap.png" alt="screenshot/gui-datacenter-realm-add-ldap.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>It’s possible to automatically sync users and groups for LDAP-based realms (LDAP
&amp; Microsoft Active Directory), rather than having to add them to Proxmox VE manually.
You can access the sync options from the Add/Edit window of the web interface’s
<span class="monospaced">Authentication</span> panel or via the <span class="monospaced">pveum realm add/modify</span> commands. You can
then carry out the sync operation from the <span class="monospaced">Authentication</span> panel of the GUI or
using the following command:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>pveum realm sync &lt;realm&gt;</pre>
</div></div>
<div class="paragraph">
<p>Users and groups are synced to the cluster-wide configuration file,
<span class="monospaced">/etc/pve/user.cfg</span>.</p></div>
<div class="sect4">
<h5 id="_attributes_to_properties">Attributes to Properties
 <a class="headerlink" href="#_attributes_to_properties" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>If the sync response includes user attributes, they will be synced into the
matching user property in the <span class="monospaced">user.cfg</span>. For example: <span class="monospaced">firstname</span> or
<span class="monospaced">lastname</span>.</p></div>
<div class="paragraph">
<p>If the names of the attributes are not matching the Proxmox VE properties, you can
set a custom field-to-field map in the config by using the <span class="monospaced">sync_attributes</span>
option.</p></div>
<div class="paragraph">
<p>How such properties are handled if anything vanishes can be controlled via the
sync options, see below.</p></div>
</div>
<div class="sect4">
<h5 id="_sync_configuration">Sync Configuration
 <a class="headerlink" href="#_sync_configuration" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>The configuration options for syncing LDAP-based realms can be found in the
<span class="monospaced">Sync Options</span> tab of the Add/Edit window.</p></div>
<div class="paragraph">
<p>The configuration options are as follows:</p></div>
<div class="ulist"><ul>
<li>
<p>
<span class="monospaced">Bind User</span> (<span class="monospaced">bind_dn</span>): Refers to the LDAP account used to query users
  and groups. This account needs access to all desired entries. If it’s set, the
  search will be carried out via binding; otherwise, the search will be carried
  out anonymously. The user must be a complete LDAP formatted distinguished name
  (DN), for example, <span class="monospaced">cn=admin,dc=example,dc=com</span>.
</p>
</li>
<li>
<p>
Groupname attr. (group_name_attr): Represents the
  users' groups. Only entries which adhere to the usual character limitations of
  the <span class="monospaced">user.cfg</span> are synced. Groups are synced with <span class="monospaced">-$realm</span> attached to the
  name, in order to avoid naming conflicts. Please ensure that a sync does not
  overwrite manually created groups.
</p>
</li>
<li>
<p>
<span class="monospaced">User classes</span> (<span class="monospaced">user_classes</span>): Objects classes associated with users.
</p>
</li>
<li>
<p>
<span class="monospaced">Group classes</span> (<span class="monospaced">group_classes</span>): Objects classes associated with groups.
</p>
</li>
<li>
<p>
<span class="monospaced">E-Mail attribute</span>: If the LDAP-based server specifies user email addresses,
  these can also be included in the sync by setting the associated attribute
  here. From the command line, this is achievable through the
  <span class="monospaced">--sync_attributes</span> parameter.
</p>
</li>
<li>
<p>
<span class="monospaced">User Filter</span> (<span class="monospaced">filter</span>): For further filter options to target specific users.
</p>
</li>
<li>
<p>
<span class="monospaced">Group Filter</span> (<span class="monospaced">group_filter</span>): For further filter options to target specific
  groups.
</p>
</li>
</ul></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Filters allow you to create a set of additional match criteria, to narrow
down the scope of a sync. Information on available LDAP filter types and their
usage can be found at <a href="https://ldap.com/ldap-filters/">ldap.com</a>.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect4">
<h5 id="pveum_ldap_sync_options">Sync Options
 <a class="headerlink" href="#pveum_ldap_sync_options" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-datacenter-realm-add-ldap-sync-options.png">
<img src="images/screenshot/gui-datacenter-realm-add-ldap-sync-options.png" alt="screenshot/gui-datacenter-realm-add-ldap-sync-options.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>In addition to the options specified in the previous section, you can also
configure further options that describe the behavior of the sync operation.</p></div>
<div class="paragraph">
<p>These options are either set as parameters before the sync, or as defaults via
the realm option <span class="monospaced">sync-defaults-options</span>.</p></div>
<div class="paragraph">
<p>The main options for syncing are:</p></div>
<div class="ulist"><ul>
<li>
<p>
<span class="monospaced">Scope</span> (<span class="monospaced">scope</span>): The scope of what to sync. It can be either <span class="monospaced">users</span>,
  <span class="monospaced">groups</span> or <span class="monospaced">both</span>.
</p>
</li>
<li>
<p>
<span class="monospaced">Enable new</span> (<span class="monospaced">enable-new</span>): If set, the newly synced users are enabled and
  can log in. The default is <span class="monospaced">true</span>.
</p>
</li>
<li>
<p>
<span class="monospaced">Remove Vanished</span> (<span class="monospaced">remove-vanished</span>): This is a list of options which, when
  activated, determine if they are removed when they are not returned from
  the sync response. The options are:
</p>
<div class="ulist"><ul>
<li>
<p>
<span class="monospaced">ACL</span> (<span class="monospaced">acl)</span>: Remove ACLs of users and groups which were not returned
      returned in the sync response. This most often makes sense together with
      <span class="monospaced">Entry</span>.
</p>
</li>
<li>
<p>
<span class="monospaced">Entry</span> (<span class="monospaced">entry</span>): Removes entries (i.e. users and groups) when they are
      not returned in the sync response.
</p>
</li>
<li>
<p>
<span class="monospaced">Properties</span> (<span class="monospaced">properties</span>): Removes properties of entries where the user
      in the sync response did not contain those attributes. This includes
      all properties, even those never set by a sync. Exceptions are tokens
      and the enable flag, these will be retained even with this option enabled.
</p>
</li>
</ul></div>
</li>
<li>
<p>
<span class="monospaced">Preview</span> (<span class="monospaced">dry-run</span>): No data is written to the config. This is useful if you
  want to see which users and groups would get synced to the <span class="monospaced">user.cfg</span>.
</p>
</li>
</ul></div>
</div>
<div class="sect4">
<h5 id="pveum_ldap_reserved_characters">Reserved characters
 <a class="headerlink" href="#pveum_ldap_reserved_characters" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Certain characters are reserved (see <a href="https://www.ietf.org/rfc/rfc2253.txt">RFC2253</a>) and cannot be
easily used in attribute values in DNs without being escaped properly.</p></div>
<div class="paragraph">
<p>Following characters need escaping:</p></div>
<div class="ulist"><ul>
<li>
<p>
Space ( ) at the beginning or end
</p>
</li>
<li>
<p>
Number sign (<span class="monospaced">#</span>) at the beginning
</p>
</li>
<li>
<p>
Comma (<span class="monospaced">,</span>)
</p>
</li>
<li>
<p>
Plus sign (<span class="monospaced">+</span>)
</p>
</li>
<li>
<p>
Double quote (<span class="monospaced">"</span>)
</p>
</li>
<li>
<p>
Forward slashes (<span class="monospaced">/</span>)
</p>
</li>
<li>
<p>
Angle brackets (<span class="monospaced">&lt;&gt;</span>)
</p>
</li>
<li>
<p>
Semicolon (<span class="monospaced">;</span>)
</p>
</li>
<li>
<p>
Equals sign (<span class="monospaced">=</span>)
</p>
</li>
</ul></div>
<div class="paragraph">
<p>To use such characters in DNs, surround the attribute value in double quotes.
For example, to bind with a user with the CN (Common Name) <span class="monospaced">Example, User</span>, use
<span class="monospaced">CN="Example, User",OU=people,DC=example,DC=com</span> as value for <span class="monospaced">bind_dn</span>.</p></div>
<div class="paragraph">
<p>This applies to the <span class="monospaced">base_dn</span>, <span class="monospaced">bind_dn</span>, and <span class="monospaced">group_dn</span> attributes.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Users with colons and forward slashes cannot be synced since these are
reserved characters in usernames.</td>
</tr></tbody></table>
</div>
</div>
</div>
<div class="sect3">
<h4 id="pveum_openid">14.5.6. OpenID Connect
 <a class="headerlink" href="#pveum_openid" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The main OpenID Connect configuration options are:</p></div>
<div class="ulist"><ul>
<li>
<p>
<span class="monospaced">Issuer URL</span> (<span class="monospaced">issuer-url</span>): This is the URL of the authorization server.
Proxmox uses the OpenID Connect Discovery protocol to automatically configure
further details.
</p>
<div class="paragraph">
<p>While it is possible to use unencrypted <span class="monospaced">http://</span> URLs, we strongly recommend to
use encrypted <span class="monospaced">https://</span> connections.</p></div>
</li>
<li>
<p>
<span class="monospaced">Realm</span> (<span class="monospaced">realm</span>): The realm identifier for Proxmox VE users
</p>
</li>
<li>
<p>
<span class="monospaced">Client ID</span> (<span class="monospaced">client-id</span>):  OpenID Client ID.
</p>
</li>
<li>
<p>
<span class="monospaced">Client Key</span> (<span class="monospaced">client-key</span>): Optional OpenID Client Key.
</p>
</li>
<li>
<p>
<span class="monospaced">Autocreate Users</span> (<span class="monospaced">autocreate</span>): Automatically create users if they do not
exist. While authentication is done at the OpenID server, all users still need
an entry in the Proxmox VE user configuration. You can either add them manually, or
use the <span class="monospaced">autocreate</span> option to automatically add new users.
</p>
</li>
<li>
<p>
<span class="monospaced">Username Claim</span> (<span class="monospaced">username-claim</span>): OpenID claim used to generate the unique
username (<span class="monospaced">subject</span>, <span class="monospaced">username</span> or <span class="monospaced">email</span>).
</p>
</li>
</ul></div>
<div class="sect4">
<h5 id="_username_mapping">Username mapping
 <a class="headerlink" href="#_username_mapping" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>The OpenID Connect specification defines a single unique attribute
(<em>claim</em> in OpenID terms) named <span class="monospaced">subject</span>. By default, we use the
value of this attribute to generate Proxmox VE usernames, by simple adding
<span class="monospaced">@</span> and the realm name: <span class="monospaced">${subject}@${realm}</span>.</p></div>
<div class="paragraph">
<p>Unfortunately, most OpenID servers use random strings for <span class="monospaced">subject</span>, like
<span class="monospaced">DGH76OKH34BNG3245SB</span>, so a typical username would look like
<span class="monospaced">DGH76OKH34BNG3245SB@yourrealm</span>. While unique, it is difficult for
humans to remember such random strings, making it quite impossible to
associate real users with this.</p></div>
<div class="paragraph">
<p>The <span class="monospaced">username-claim</span> setting allows you to use other attributes for
the username mapping. Setting it to <span class="monospaced">username</span> is preferred if the
OpenID Connect server provides that attribute and guarantees its
uniqueness.</p></div>
<div class="paragraph">
<p>Another option is to use <span class="monospaced">email</span>, which also yields human readable
usernames. Again, only use this setting if the server guarantees the
uniqueness of this attribute.</p></div>
</div>
<div class="sect4">
<h5 id="_examples_9">Examples
 <a class="headerlink" href="#_examples_9" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Here is an example of creating an OpenID realm using Google. You need to
replace <span class="monospaced">--client-id</span> and <span class="monospaced">--client-key</span> with the values
from your Google OpenID settings.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>pveum realm add myrealm1 --type openid --issuer-url  https://accounts.google.com --client-id XXXX --client-key YYYY --username-claim email</pre>
</div></div>
<div class="paragraph">
<p>The above command uses <span class="monospaced">--username-claim email</span>, so that the usernames on the
Proxmox VE side look like <span class="monospaced">[email protected]@myrealm1</span>.</p></div>
<div class="paragraph">
<p>Keycloak (<a href="https://www.keycloak.org/">https://www.keycloak.org/</a>) is a popular open source Identity
and Access Management tool, which supports OpenID Connect. In the following
example, you need to replace the <span class="monospaced">--issuer-url</span> and <span class="monospaced">--client-id</span> with
your information:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>pveum realm add myrealm2 --type openid --issuer-url  https://your.server:8080/realms/your-realm --client-id XXX --username-claim username</pre>
</div></div>
<div class="paragraph">
<p>Using <span class="monospaced">--username-claim username</span> enables simple usernames on the
Proxmox VE side, like <span class="monospaced">example.user@myrealm2</span>.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">You need to ensure that the user is not allowed to edit
the username setting themselves (on the Keycloak server).</td>
</tr></tbody></table>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="pveum_tfa_auth">
<span>14.6. Two-Factor Authentication</span>
 <a class="headerlink" href="#pveum_tfa_auth" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>There are two ways to use two-factor authentication:</p></div>
<div class="paragraph">
<p>It can be required by the authentication realm, either via <em>TOTP</em>
(Time-based One-Time Password) or <em>YubiKey OTP</em>. In this case, a newly
created user needs to have their keys added immediately, as there is no way to
log in without the second factor. In the case of <em>TOTP</em>, users can
also change the <em>TOTP</em> later on, provided they can log in first.</p></div>
<div class="paragraph">
<p>Alternatively, users can choose to opt-in to two-factor authentication
later on, even if the realm does not enforce it.</p></div>
<div class="sect3">
<h4 id="_available_second_factors">14.6.1. Available Second Factors
 <a class="headerlink" href="#_available_second_factors" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>You can set up multiple second factors, in order to avoid a situation in
which losing your smartphone or security key locks you out of your
account permanently.</p></div>
<div class="paragraph">
<p>The following two-factor authentication methods are available in
addition to realm-enforced TOTP and YubiKey OTP:</p></div>
<div class="ulist"><ul>
<li>
<p>
User configured TOTP
  (<a href="https://en.wikipedia.org/wiki/Time-based_One-Time_Password">Time-based One-Time Password</a>).
  A short code derived from a shared secret and the current time, it changes
  every 30 seconds.
</p>
</li>
<li>
<p>
WebAuthn (<a href="https://en.wikipedia.org/wiki/WebAuthn">Web Authentication</a>).
  A general standard for authentication. It is implemented by various
  security devices, like hardware keys or trusted platform modules (TPM)
  from a computer or smart phone.
</p>
</li>
<li>
<p>
Single use Recovery Keys. A list of keys which should either be
  printed out and locked in a secure place or saved digitally in an
  electronic vault. Each key can be used only once. These are perfect for
  ensuring that you are not locked out, even if all of your other second
  factors are lost or corrupt.
</p>
</li>
</ul></div>
<div class="paragraph">
<p>Before WebAuthn was supported, U2F could be setup by the user. Existing
U2F factors can still be used, but it is recommended to switch to
WebAuthn, once it is configured on the server.</p></div>
</div>
<div class="sect3">
<h4 id="_realm_enforced_two_factor_authentication">14.6.2. Realm Enforced Two-Factor Authentication
 <a class="headerlink" href="#_realm_enforced_two_factor_authentication" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>This can be done by selecting one of the available methods via the
<em>TFA</em> dropdown box when adding or editing an Authentication Realm.
When a realm has TFA enabled, it becomes a requirement, and only users
with configured TFA will be able to log in.</p></div>
<div class="paragraph">
<p>Currently there are two methods available:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
Time-based OATH (TOTP)
</dt>
<dd>
<p>
This uses the standard HMAC-SHA1 algorithm,
where the current time is hashed with the user’s configured key. The
time step and password length parameters are configurable.
</p>
<div class="paragraph">
<p>A user can have multiple keys configured (separated by spaces), and the keys
can be specified in Base32 (RFC3548) or hexadecimal notation.</p></div>
<div class="paragraph">
<p>Proxmox VE provides a key generation tool (<span class="monospaced">oathkeygen</span>) which prints out a random
key in Base32 notation, that can be used directly with various OTP tools, such
as the <span class="monospaced">oathtool</span> command-line tool, or on Android Google Authenticator,
FreeOTP, andOTP or similar applications.</p></div>
</dd>
<dt class="hdlist1">
YubiKey OTP
</dt>
<dd>
<p>
For authenticating via a YubiKey a Yubico API ID, API KEY and validation
server URL must be configured, and users must have a YubiKey available. In
order to get the key ID from a YubiKey, you can trigger the YubiKey once
after connecting it via USB, and copy the first 12 characters of the typed
password into the user’s <em>Key IDs</em> field.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p>Please refer to the <a href="https://developers.yubico.com/OTP/">YubiKey OTP</a>
documentation for how to use the
<a href="https://www.yubico.com/products/services-software/yubicloud/">YubiCloud</a> or
<a href="https://developers.yubico.com/Software_Projects/Yubico_OTP/YubiCloud_Validation_Servers/">host your own verification server</a>.</p></div>
</div>
<div class="sect3">
<h4 id="pveum_tfa_lockout">14.6.3. Limits and Lockout of Two-Factor Authentication
 <a class="headerlink" href="#pveum_tfa_lockout" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>A second factor is meant to protect users if their password is somehow leaked
or guessed. However, some factors could still be broken by brute force. For
this reason, users will be locked out after too many failed 2nd factor login
attempts.</p></div>
<div class="paragraph">
<p>For TOTP, 8 failed attempts will disable the user’s TOTP factors. They are
unlocked when logging in with a recovery key. If TOTP was the only available
factor, admin intervention is required, and it is highly recommended to require
the user to change their password immediately.</p></div>
<div class="paragraph">
<p>Since FIDO2/Webauthn and recovery keys are less susceptible to brute force
attacks, the limit there is higher (100 tries), but all second factors are
blocked for an hour when exceeded.</p></div>
<div class="paragraph">
<p>An admin can unlock a user’s Two-Factor Authentication at any time via the user
list in the UI or the command line:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt> pveum user tfa unlock joe@pve</tt></pre></div></div>
</div>
<div class="sect3">
<h4 id="pveum_user_configured_totp">14.6.4. User Configured TOTP Authentication
 <a class="headerlink" href="#pveum_user_configured_totp" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Users can choose to enable <em>TOTP</em> or <em>WebAuthn</em> as a second factor on login, via
the <em>TFA</em> button in the user list (unless the realm enforces <em>YubiKey OTP</em>).</p></div>
<div class="paragraph">
<p>Users can always add and use one time <em>Recovery Keys</em>.</p></div>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-datacenter-two-factor.png">
<img src="images/screenshot/gui-datacenter-two-factor.png" alt="screenshot/gui-datacenter-two-factor.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>After opening the <em>TFA</em> window, the user is presented with a dialog to set up
<em>TOTP</em> authentication. The <em>Secret</em> field contains the key, which can be
randomly generated via the <em>Randomize</em> button. An optional <em>Issuer Name</em> can be
added to provide information to the <em>TOTP</em> app about what the key belongs to.
Most <em>TOTP</em> apps will show the issuer name together with the corresponding
<em>OTP</em> values. The username is also included in the QR code for the <em>TOTP</em> app.</p></div>
<div class="paragraph">
<p>After generating a key, a QR code will be displayed, which can be used with most
OTP apps such as FreeOTP. The user then needs to verify the current user
password (unless logged in as <em>root</em>), as well as the ability to correctly use
the <em>TOTP</em> key, by typing the current <em>OTP</em> value into the <em>Verification Code</em>
field and pressing the <em>Apply</em> button.</p></div>
</div>
<div class="sect3">
<h4 id="user_tfa_setup_totp">14.6.5. TOTP
 <a class="headerlink" href="#user_tfa_setup_totp" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<a class="image" href="images/screenshot/pve-gui-tfa-add-totp.png">
<img src="images/screenshot/pve-gui-tfa-add-totp.png" alt="screenshot/pve-gui-tfa-add-totp.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>There is no server setup required. Simply install a TOTP app on your
smartphone (for example, <a href="https://freeotp.github.io/">FreeOTP</a>) and use
the Proxmox VE web interface to add a TOTP factor.</p></div>
</div>
<div class="sect3">
<h4 id="user_tfa_setup_webauthn">14.6.6. WebAuthn
 <a class="headerlink" href="#user_tfa_setup_webauthn" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>For WebAuthn to work, you need to have two things:</p></div>
<div class="ulist"><ul>
<li>
<p>
A trusted HTTPS certificate (for example, by using
  <a href="https://pve.proxmox.com/wiki/Certificate_Management">Let’s Encrypt</a>).
  While it probably works with an untrusted certificate, some browsers may
  warn or refuse WebAuthn operations if it is not trusted.
</p>
</li>
<li>
<p>
Setup the WebAuthn configuration (see <strong>Datacenter → Options →
  WebAuthn Settings</strong> in the Proxmox VE web interface). This can be
  auto-filled in most setups.
</p>
</li>
</ul></div>
<div class="paragraph">
<p>Once you have fulfilled both of these requirements, you can add a WebAuthn
configuration in the <strong>Two Factor</strong> panel under <strong>Datacenter → Permissions → Two
Factor</strong>.</p></div>
</div>
<div class="sect3">
<h4 id="user_tfa_setup_recovery_keys">14.6.7. Recovery Keys
 <a class="headerlink" href="#user_tfa_setup_recovery_keys" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<a class="image" href="images/screenshot/pve-gui-tfa-add-recovery-keys.png">
<img src="images/screenshot/pve-gui-tfa-add-recovery-keys.png" alt="screenshot/pve-gui-tfa-add-recovery-keys.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>Recovery key codes do not need any preparation; you can simply create a
set of recovery keys in the <strong>Two Factor</strong> panel under <strong>Datacenter → Permissions
→ Two Factor</strong>.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">There can only be one set of single-use recovery keys per user at any
time.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="pveum_configure_webauthn">14.6.8. Server Side Webauthn Configuration
 <a class="headerlink" href="#pveum_configure_webauthn" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-datacenter-webauthn-edit.png">
<img src="images/screenshot/gui-datacenter-webauthn-edit.png" alt="screenshot/gui-datacenter-webauthn-edit.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>To allow users to use <em>WebAuthn</em> authentication, it is necessaary to use a valid
domain with a valid SSL certificate, otherwise some browsers may warn or refuse
to authenticate altogether.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Changing the <em>WebAuthn</em> configuration may render all existing <em>WebAuthn</em>
registrations unusable!</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>This is done via <span class="monospaced">/etc/pve/datacenter.cfg</span>. For instance:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>webauthn: rp=mypve.example.com,origin=https://mypve.example.com:8006,id=mypve.example.com</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="pveum_configure_u2f">14.6.9. Server Side U2F Configuration
 <a class="headerlink" href="#pveum_configure_u2f" title="Permalink to this heading"></a>
</h4>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">It is recommended to use WebAuthn instead.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>To allow users to use <em>U2F</em> authentication, it may be necessary to use a valid
domain with a valid SSL certificate, otherwise, some browsers may print
a warning or reject U2F usage altogether. Initially, an <em>AppId</em>
<span class="footnote" data-note="AppId <a href=&quot;https://developers.yubico.com/U2F/App_ID.html&quot;>https://developers.yubico.com/U2F/App_ID.html</a>">[<a id="_footnoteref_52" href="#_footnote_52" title="View footnote" class="footnote">52</a>]</span>
needs to be configured.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Changing the <em>AppId</em> will render all existing <em>U2F</em> registrations
unusable!</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>This is done via <span class="monospaced">/etc/pve/datacenter.cfg</span>. For instance:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>u2f: appid=https://mypve.example.com:8006</pre>
</div></div>
<div class="paragraph">
<p>For a single node, the <em>AppId</em> can simply be the address of the web interface,
exactly as it is used in the browser, including the <em>https://</em> and the port, as
shown above. Please note that some browsers may be more strict than others when
matching <em>AppIds</em>.</p></div>
<div class="paragraph">
<p>When using multiple nodes, it is best to have a separate <span class="monospaced">https</span> server
providing an <span class="monospaced">appid.json</span>
<span class="footnote" data-note="Multi-facet apps: <a href=&quot;https://developers.yubico.com/U2F/App_ID.html&quot;>https://developers.yubico.com/U2F/App_ID.html</a>">[<a id="_footnoteref_53" href="#_footnote_53" title="View footnote" class="footnote">53</a>]</span>
file, as it seems to be compatible with most
browsers. If all nodes use subdomains of the same top level domain, it may be
enough to use the TLD as <em>AppId</em>. It should however be noted that some browsers
may not accept this.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">A bad <em>AppId</em> will usually produce an error, but we have encountered
situations when this does not happen, particularly when using a top level domain
<em>AppId</em> for a node that is accessed via a subdomain in Chromium. For this reason
it is recommended to test the configuration with multiple browsers, as changing
the <em>AppId</em> later will render existing <em>U2F</em> registrations unusable.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="pveum_user_configured_u2f">14.6.10. Activating U2F as a User
 <a class="headerlink" href="#pveum_user_configured_u2f" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>To enable <em>U2F</em> authentication, open the <em>TFA</em> window’s <em>U2F</em> tab, type in the
current password (unless logged in as root), and press the <em>Register</em> button.
If the server is set up correctly and the browser accepts the server’s provided
<em>AppId</em>, a message will appear prompting the user to press the button on the
<em>U2F</em> device (if it is a <em>YubiKey</em>, the button light should be toggling on and
off steadily, roughly twice per second).</p></div>
<div class="paragraph">
<p>Firefox users may need to enable <em>security.webauth.u2f</em> via <em>about:config</em>
before they can use a <em>U2F</em> token.</p></div>
</div>
</div>
<div class="sect2">
<h3 id="pveum_permission_management">
<span>14.7. Permission Management</span>
 <a class="headerlink" href="#pveum_permission_management" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>In order for a user to perform an action (such as listing, modifying or
deleting parts of a VM’s configuration), the user needs to have the
appropriate permissions.</p></div>
<div class="paragraph">
<p>Proxmox VE uses a role and path based permission management system. An entry in
the permissions table allows a user, group or token to take on a specific role
when accessing an <em>object</em> or <em>path</em>. This means that such an access rule can
be represented as a triple of <em>(path, user, role)</em>, <em>(path, group,
role)</em> or <em>(path, token, role)</em>, with the role containing a set of allowed
actions, and the path representing the target of these actions.</p></div>
<div class="sect3">
<h4 id="pveum_roles">14.7.1. Roles
 <a class="headerlink" href="#pveum_roles" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>A role is simply a list of privileges. Proxmox VE comes with a number
of predefined roles, which satisfy most requirements.</p></div>
<div class="ulist"><ul>
<li>
<p>
<span class="monospaced">Administrator</span>: has full privileges
</p>
</li>
<li>
<p>
<span class="monospaced">NoAccess</span>: has no privileges (used to forbid access)
</p>
</li>
<li>
<p>
<span class="monospaced">PVEAdmin</span>: can do most tasks, but has no rights to modify system settings
  (<span class="monospaced">Sys.PowerMgmt</span>, <span class="monospaced">Sys.Modify</span>, <span class="monospaced">Realm.Allocate</span>) or permissions
  (<span class="monospaced">Permissions.Modify</span>)
</p>
</li>
<li>
<p>
<span class="monospaced">PVEAuditor</span>: has read only access
</p>
</li>
<li>
<p>
<span class="monospaced">PVEDatastoreAdmin</span>: create and allocate backup space and templates
</p>
</li>
<li>
<p>
<span class="monospaced">PVEDatastoreUser</span>: allocate backup space and view storage
</p>
</li>
<li>
<p>
<span class="monospaced">PVEMappingAdmin</span>: manage resource mappings
</p>
</li>
<li>
<p>
<span class="monospaced">PVEMappingUser</span>: view and use resource mappings
</p>
</li>
<li>
<p>
<span class="monospaced">PVEPoolAdmin</span>: allocate pools
</p>
</li>
<li>
<p>
<span class="monospaced">PVEPoolUser</span>: view pools
</p>
</li>
<li>
<p>
<span class="monospaced">PVESDNAdmin</span>: manage SDN configuration
</p>
</li>
<li>
<p>
<span class="monospaced">PVESDNUser</span>: access to bridges/vnets
</p>
</li>
<li>
<p>
<span class="monospaced">PVESysAdmin</span>: audit, system console and system logs
</p>
</li>
<li>
<p>
<span class="monospaced">PVETemplateUser</span>: view and clone templates
</p>
</li>
<li>
<p>
<span class="monospaced">PVEUserAdmin</span>: manage users
</p>
</li>
<li>
<p>
<span class="monospaced">PVEVMAdmin</span>: fully administer VMs
</p>
</li>
<li>
<p>
<span class="monospaced">PVEVMUser</span>: view, backup, configure CD-ROM, VM console, VM power management
</p>
</li>
</ul></div>
<div class="paragraph">
<p>You can see the whole set of predefined roles in the GUI.</p></div>
<div class="paragraph">
<p>You can add new roles via the GUI or the command line.</p></div>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-datacenter-role-add.png">
<img src="images/screenshot/gui-datacenter-role-add.png" alt="screenshot/gui-datacenter-role-add.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>From the GUI, navigate to the <em>Permissions → Roles</em> tab from <em>Datacenter</em> and
click on the <em>Create</em> button. There you can set a role name and select any
desired privileges from the <em>Privileges</em> drop-down menu.</p></div>
<div class="paragraph">
<p>To add a role through the command line, you can use the <em>pveum</em> CLI tool, for
example:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>pveum role add VM_Power-only --privs <span style="color: #FF0000">"VM.PowerMgmt VM.Console"</span>
pveum role add Sys_Power-only --privs <span style="color: #FF0000">"Sys.PowerMgmt Sys.Console"</span></tt></pre></div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Roles starting with <span class="monospaced">PVE</span> are always builtin, custom roles are not
allowed use this reserved prefix.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="_privileges">14.7.2. Privileges
 <a class="headerlink" href="#_privileges" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>A privilege is the right to perform a specific action. To simplify
management, lists of privileges are grouped into roles, which can then
be used in the permission table. Note that privileges cannot be directly
assigned to users and paths without being part of a role.</p></div>
<div class="paragraph">
<p>We currently support the following privileges:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
Node / System related privileges
</dt>
<dd>
<div class="ulist"><ul>
<li>
<p>
<span class="monospaced">Group.Allocate</span>: create/modify/remove groups
</p>
</li>
<li>
<p>
<span class="monospaced">Mapping.Audit</span>: view resource mappings
</p>
</li>
<li>
<p>
<span class="monospaced">Mapping.Modify</span>: manage resource mappings
</p>
</li>
<li>
<p>
<span class="monospaced">Mapping.Use</span>: use resource mappings
</p>
</li>
<li>
<p>
<span class="monospaced">Permissions.Modify</span>: modify access permissions
</p>
</li>
<li>
<p>
<span class="monospaced">Pool.Allocate</span>: create/modify/remove a pool
</p>
</li>
<li>
<p>
<span class="monospaced">Pool.Audit</span>: view a pool
</p>
</li>
<li>
<p>
<span class="monospaced">Realm.AllocateUser</span>: assign user to a realm
</p>
</li>
<li>
<p>
<span class="monospaced">Realm.Allocate</span>: create/modify/remove authentication realms
</p>
</li>
<li>
<p>
<span class="monospaced">SDN.Allocate</span>: manage SDN configuration
</p>
</li>
<li>
<p>
<span class="monospaced">SDN.Audit</span>: view SDN configuration
</p>
</li>
<li>
<p>
<span class="monospaced">Sys.Audit</span>: view node status/config, Corosync cluster config, and HA config
</p>
</li>
<li>
<p>
<span class="monospaced">Sys.Console</span>: console access to node
</p>
</li>
<li>
<p>
<span class="monospaced">Sys.Incoming</span>: allow incoming data streams from other clusters (experimental)
</p>
</li>
<li>
<p>
<span class="monospaced">Sys.Modify</span>: create/modify/remove node network parameters
</p>
</li>
<li>
<p>
<span class="monospaced">Sys.PowerMgmt</span>: node power management (start, stop, reset, shutdown, …)
</p>
</li>
<li>
<p>
<span class="monospaced">Sys.Syslog</span>: view syslog
</p>
</li>
<li>
<p>
<span class="monospaced">User.Modify</span>: create/modify/remove user access and details.
</p>
</li>
</ul></div>
</dd>
<dt class="hdlist1">
Virtual machine related privileges
</dt>
<dd>
<div class="ulist"><ul>
<li>
<p>
<span class="monospaced">SDN.Use</span>: access SDN vnets and local network bridges
</p>
</li>
<li>
<p>
<span class="monospaced">VM.Allocate</span>: create/remove VM on a server
</p>
</li>
<li>
<p>
<span class="monospaced">VM.Audit</span>: view VM config
</p>
</li>
<li>
<p>
<span class="monospaced">VM.Backup</span>: backup/restore VMs
</p>
</li>
<li>
<p>
<span class="monospaced">VM.Clone</span>: clone/copy a VM
</p>
</li>
<li>
<p>
<span class="monospaced">VM.Config.CDROM</span>: eject/change CD-ROM
</p>
</li>
<li>
<p>
<span class="monospaced">VM.Config.CPU</span>: modify CPU settings
</p>
</li>
<li>
<p>
<span class="monospaced">VM.Config.Cloudinit</span>: modify Cloud-init parameters
</p>
</li>
<li>
<p>
<span class="monospaced">VM.Config.Disk</span>: add/modify/remove disks
</p>
</li>
<li>
<p>
<span class="monospaced">VM.Config.HWType</span>: modify emulated hardware types
</p>
</li>
<li>
<p>
<span class="monospaced">VM.Config.Memory</span>: modify memory settings
</p>
</li>
<li>
<p>
<span class="monospaced">VM.Config.Network</span>: add/modify/remove network devices
</p>
</li>
<li>
<p>
<span class="monospaced">VM.Config.Options</span>: modify any other VM configuration
</p>
</li>
<li>
<p>
<span class="monospaced">VM.Console</span>: console access to VM
</p>
</li>
<li>
<p>
<span class="monospaced">VM.Migrate</span>: migrate VM to alternate server on cluster
</p>
</li>
<li>
<p>
<span class="monospaced">VM.Monitor</span>: access to VM monitor (kvm)
</p>
</li>
<li>
<p>
<span class="monospaced">VM.PowerMgmt</span>: power management (start, stop, reset, shutdown, …)
</p>
</li>
<li>
<p>
<span class="monospaced">VM.Snapshot.Rollback</span>: rollback VM to one of its snapshots
</p>
</li>
<li>
<p>
<span class="monospaced">VM.Snapshot</span>: create/delete VM snapshots
</p>
</li>
</ul></div>
</dd>
<dt class="hdlist1">
Storage related privileges
</dt>
<dd>
<div class="ulist"><ul>
<li>
<p>
<span class="monospaced">Datastore.Allocate</span>: create/modify/remove a datastore and delete volumes
</p>
</li>
<li>
<p>
<span class="monospaced">Datastore.AllocateSpace</span>: allocate space on a datastore
</p>
</li>
<li>
<p>
<span class="monospaced">Datastore.AllocateTemplate</span>: allocate/upload templates and ISO images
</p>
</li>
<li>
<p>
<span class="monospaced">Datastore.Audit</span>: view/browse a datastore
</p>
</li>
</ul></div>
</dd>
</dl></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">Both <span class="monospaced">Permissions.Modify</span> and <span class="monospaced">Sys.Modify</span> should be handled with
care, as they allow modifying aspects of the system and its configuration that
are dangerous or sensitive.</td>
</tr></tbody></table>
</div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">Carefully read the section about inheritance below to understand how
assigned roles (and their privileges) are propagated along the ACL tree.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="_objects_and_paths">14.7.3. Objects and Paths
 <a class="headerlink" href="#_objects_and_paths" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Access permissions are assigned to objects, such as virtual machines,
storages or resource pools.
We use file system like paths to address these objects. These paths form a
natural tree, and permissions of higher levels (shorter paths) can
optionally be propagated down within this hierarchy.</p></div>
<div class="paragraph" id="pveum_templated_paths">
<p>Paths can be templated. When an API call requires permissions on a
templated path, the path may contain references to parameters of the API
call. These references are specified in curly braces. Some parameters are
implicitly taken from the API call’s URI. For instance, the permission path
<span class="monospaced">/nodes/{node}</span> when calling <em>/nodes/mynode/status</em> requires permissions on
<span class="monospaced">/nodes/mynode</span>, while the path <span class="monospaced">{path}</span> in a PUT request to <span class="monospaced">/access/acl</span>
refers to the method’s <span class="monospaced">path</span> parameter.</p></div>
<div class="paragraph">
<p>Some examples are:</p></div>
<div class="ulist"><ul>
<li>
<p>
<span class="monospaced">/nodes/{node}</span>: Access to Proxmox VE server machines
</p>
</li>
<li>
<p>
<span class="monospaced">/vms</span>: Covers all VMs
</p>
</li>
<li>
<p>
<span class="monospaced">/vms/{vmid}</span>: Access to specific VMs
</p>
</li>
<li>
<p>
<span class="monospaced">/storage/{storeid}</span>: Access to a specific storage
</p>
</li>
<li>
<p>
<span class="monospaced">/pool/{poolname}</span>: Access to resources contained in a specific <a href="#pveum_pools">pool</a>
</p>
</li>
<li>
<p>
<span class="monospaced">/access/groups</span>: Group administration
</p>
</li>
<li>
<p>
<span class="monospaced">/access/realms/{realmid}</span>: Administrative access to realms
</p>
</li>
</ul></div>
<div class="sect4">
<h5 id="_inheritance">Inheritance
 <a class="headerlink" href="#_inheritance" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>As mentioned earlier, object paths form a file system like tree, and
permissions can be inherited by objects down that tree (the propagate flag is
set by default). We use the following inheritance rules:</p></div>
<div class="ulist"><ul>
<li>
<p>
Permissions for individual users always replace group permissions.
</p>
</li>
<li>
<p>
Permissions for groups apply when the user is member of that group.
</p>
</li>
<li>
<p>
Permissions on deeper levels replace those inherited from an upper level.
</p>
</li>
<li>
<p>
<span class="monospaced">NoAccess</span> cancels all other roles on a given path.
</p>
</li>
</ul></div>
<div class="paragraph">
<p>Additionally, privilege separated tokens can never have permissions on any
given path that their associated user does not have.</p></div>
</div>
</div>
<div class="sect3">
<h4 id="pveum_pools">14.7.4. Pools
 <a class="headerlink" href="#pveum_pools" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Pools can be used to group a set of virtual machines and datastores. You can
then simply set permissions on pools (<span class="monospaced">/pool/{poolid}</span>), which are inherited by
all pool members. This is a great way to simplify access control.</p></div>
</div>
<div class="sect3">
<h4 id="_which_permissions_do_i_need">14.7.5. Which Permissions Do I Need?
 <a class="headerlink" href="#_which_permissions_do_i_need" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The required API permissions are documented for each individual
method, and can be found at <a href="https://pve.proxmox.com/pve-docs/api-viewer/">https://pve.proxmox.com/pve-docs/api-viewer/</a>.</p></div>
<div class="paragraph">
<p>The permissions are specified as a list, which can be interpreted as a
tree of logic and access-check functions:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">["and", &lt;subtests&gt;...]</span> and <span class="monospaced">["or", &lt;subtests&gt;...]</span>
</dt>
<dd>
<p>
Each(<span class="monospaced">and</span>) or any(<span class="monospaced">or</span>) further element in the current list has to be true.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">["perm", &lt;path&gt;, [ &lt;privileges&gt;... ], &lt;options&gt;...]</span>
</dt>
<dd>
<p>
The <span class="monospaced">path</span> is a templated parameter (see
<a href="#pveum_templated_paths">Objects and Paths</a>). All (or, if the <span class="monospaced">any</span>
option is used, any) of the listed
privileges must be allowed on the specified path. If a <span class="monospaced">require-param</span>
option is specified, then its specified parameter is required even if the
API call’s schema otherwise lists it as being optional.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">["userid-group", [ &lt;privileges&gt;... ], &lt;options&gt;...]</span>
</dt>
<dd>
<p>
The caller must have any of the listed privileges on <span class="monospaced">/access/groups</span>. In
addition, there are two possible checks, depending on whether the
<span class="monospaced">groups_param</span> option is set:
</p>
<div class="ulist"><ul>
<li>
<p>
<span class="monospaced">groups_param</span> is set: The API call has a non-optional <span class="monospaced">groups</span> parameter
and the caller must have any of the listed privileges on all of the listed
groups.
</p>
</li>
<li>
<p>
<span class="monospaced">groups_param</span> is not set: The user passed via the <span class="monospaced">userid</span> parameter
must exist and be part of a group on which the caller has any of the listed
privileges (via the <span class="monospaced">/access/groups/&lt;group&gt;</span> path).
</p>
</li>
</ul></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">["userid-param", "self"]</span>
</dt>
<dd>
<p>
The value provided for the API call’s <span class="monospaced">userid</span> parameter must refer to the
user performing the action (usually in conjunction with <span class="monospaced">or</span>, to allow
users to perform an action on themselves, even if they don’t have elevated
privileges).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">["userid-param", "Realm.AllocateUser"]</span>
</dt>
<dd>
<p>
The user needs <span class="monospaced">Realm.AllocateUser</span> access to <span class="monospaced">/access/realm/&lt;realm&gt;</span>, with
<span class="monospaced">&lt;realm&gt;</span> referring to the realm of the user passed via the <span class="monospaced">userid</span>
parameter. Note that the user does not need to exist in order to be
associated with a realm, since user IDs are passed in the form of
<span class="monospaced">&lt;username&gt;@&lt;realm&gt;</span>.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">["perm-modify", &lt;path&gt;]</span>
</dt>
<dd>
<p>
The <span class="monospaced">path</span> is a templated parameter (see
<a href="#pveum_templated_paths">Objects and Paths</a>). The user needs either the
<span class="monospaced">Permissions.Modify</span> privilege or,
depending on the path, the following privileges as a possible substitute:
</p>
<div class="ulist"><ul>
<li>
<p>
<span class="monospaced">/storage/...</span>: requires 'Datastore.Allocate`
</p>
</li>
<li>
<p>
<span class="monospaced">/vms/...</span>: requires 'VM.Allocate`
</p>
</li>
<li>
<p>
<span class="monospaced">/pool/...</span>: requires 'Pool.Allocate`
</p>
<div class="paragraph">
<p>If the path is empty, <span class="monospaced">Permissions.Modify</span> on <span class="monospaced">/access</span> is required.</p></div>
<div class="paragraph">
<p>If the user does not have the <span class="monospaced">Permissions.Modify</span> privilege, they can only
delegate subsets of their own privileges on the given path (e.g., a user with
<span class="monospaced">PVEVMAdmin</span> could assign <span class="monospaced">PVEVMUser</span>, but not <span class="monospaced">PVEAdmin</span>).</p></div>
</li>
</ul></div>
</dd>
</dl></div>
</div>
</div>
<div class="sect2">
<h3 id="_command_line_tool">
<span>14.8. Command-line Tool</span>
 <a class="headerlink" href="#_command_line_tool" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Most users will simply use the GUI to manage users. But there is also
a fully featured command-line tool called <span class="monospaced">pveum</span> (short for “<strong>P</strong>roxmox
<strong>VE</strong> <strong>U</strong>ser <strong>M</strong>anager”). Please note that all Proxmox VE command-line
tools are wrappers around the API, so you can also access those
functions through the REST API.</p></div>
<div class="paragraph">
<p>Here are some simple usage examples. To show help, type:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt> pveum</tt></pre></div></div>
<div class="paragraph">
<p>or (to show detailed help about a specific command)</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt> pveum <span style="font-weight: bold"><span style="color: #0000FF">help</span></span> user add</tt></pre></div></div>
<div class="paragraph">
<p>Create a new user:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt> pveum user add testuser@pve -comment <span style="color: #FF0000">"Just a test"</span></tt></pre></div></div>
<div class="paragraph">
<p>Set or change the password (not all realms support this):</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt> pveum passwd testuser@pve</tt></pre></div></div>
<div class="paragraph">
<p>Disable a user:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt> pveum user modify testuser@pve -enable <span style="color: #993399">0</span></tt></pre></div></div>
<div class="paragraph">
<p>Create a new group:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt> pveum group add testgroup</tt></pre></div></div>
<div class="paragraph">
<p>Create a new role:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt> pveum role add PVE_Power-only -privs <span style="color: #FF0000">"VM.PowerMgmt VM.Console"</span></tt></pre></div></div>
</div>
<div class="sect2">
<h3 id="_real_world_examples">
<span>14.9. Real World Examples</span>
 <a class="headerlink" href="#_real_world_examples" title="Permalink to this heading"></a>
</h3>
<div class="sect3">
<h4 id="_administrator_group">14.9.1. Administrator Group
 <a class="headerlink" href="#_administrator_group" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>It is possible that an administrator would want to create a group of users with
full administrator rights (without using the root account).</p></div>
<div class="paragraph">
<p>To do this, first define the group:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt> pveum group add admin -comment <span style="color: #FF0000">"System Administrators"</span></tt></pre></div></div>
<div class="paragraph">
<p>Then assign the role:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt> pveum acl modify <span style="color: #990000">/</span> -group admin -role Administrator</tt></pre></div></div>
<div class="paragraph">
<p>Finally, you can add users to the new <em>admin</em> group:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt> pveum user modify testuser@pve -group admin</tt></pre></div></div>
</div>
<div class="sect3">
<h4 id="_auditors">14.9.2. Auditors
 <a class="headerlink" href="#_auditors" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>You can give read only access to users by assigning the <span class="monospaced">PVEAuditor</span>
role to users or groups.</p></div>
<div class="paragraph">
<p>Example 1: Allow user <span class="monospaced">joe@pve</span> to see everything</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt> pveum acl modify <span style="color: #990000">/</span> -user joe@pve -role PVEAuditor</tt></pre></div></div>
<div class="paragraph">
<p>Example 2: Allow user <span class="monospaced">joe@pve</span> to see all virtual machines</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt> pveum acl modify /vms -user joe@pve -role PVEAuditor</tt></pre></div></div>
</div>
<div class="sect3">
<h4 id="_delegate_user_management">14.9.3. Delegate User Management
 <a class="headerlink" href="#_delegate_user_management" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>If you want to delegate user management to user <span class="monospaced">joe@pve</span>, you can do
that with:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt> pveum acl modify /access -user joe@pve -role PVEUserAdmin</tt></pre></div></div>
<div class="paragraph">
<p>User <span class="monospaced">joe@pve</span> can now add and remove users, and change other user attributes,
such as passwords. This is a very powerful role, and you most
likely want to limit it to selected realms and groups. The following
example allows <span class="monospaced">joe@pve</span> to modify users within the realm <span class="monospaced">pve</span>, if they
are members of group <span class="monospaced">customers</span>:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt> pveum acl modify /access/realm/pve -user joe@pve -role PVEUserAdmin
 pveum acl modify /access/groups/customers -user joe@pve -role PVEUserAdmin</tt></pre></div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">The user is able to add other users, but only if they are
members of the group <span class="monospaced">customers</span> and within the realm <span class="monospaced">pve</span>.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="_limited_api_token_for_monitoring">14.9.4. Limited API Token for Monitoring
 <a class="headerlink" href="#_limited_api_token_for_monitoring" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Permissions on API tokens are always a subset of those of their corresponding
user, meaning that an API token can’t be used to carry out a task that the
backing user has no permission to do. This section will demonstrate how you can
use an API token with separate privileges, to limit the token owner’s
permissions further.</p></div>
<div class="paragraph">
<p>Give the user <span class="monospaced">joe@pve</span> the role PVEVMAdmin on all VMs:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt> pveum acl modify /vms -user joe@pve -role PVEVMAdmin</tt></pre></div></div>
<div class="paragraph">
<p>Add a new API token with separate privileges, which is only allowed to view VM
information (for example, for monitoring purposes):</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt> pveum user token add joe@pve monitoring -privsep <span style="color: #993399">1</span>
 pveum acl modify /vms -token <span style="color: #FF0000">'joe@pve!monitoring'</span> -role PVEAuditor</tt></pre></div></div>
<div class="paragraph">
<p>Verify the permissions of the user and token:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt> pveum user permissions joe@pve
 pveum user token permissions joe@pve monitoring</tt></pre></div></div>
</div>
<div class="sect3">
<h4 id="_resource_pools">14.9.5. Resource Pools
 <a class="headerlink" href="#_resource_pools" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>An enterprise is usually structured into several smaller departments, and it is
common that you want to assign resources and delegate management tasks to each
of these. Let’s assume that you want to set up a pool for a software development
department. First, create a group:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt> pveum group add developers -comment <span style="color: #FF0000">"Our software developers"</span></tt></pre></div></div>
<div class="paragraph">
<p>Now we create a new user which is a member of that group:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt> pveum user add developer1@pve -group developers -password</tt></pre></div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">The "-password" parameter will prompt you for a password</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>Then we create a resource pool for our development department to use:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt> pveum pool add dev-pool --comment <span style="color: #FF0000">"IT development pool"</span></tt></pre></div></div>
<div class="paragraph">
<p>Finally, we can assign permissions to that pool:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt> pveum acl modify /pool/dev-pool<span style="color: #990000">/</span> -group developers -role PVEAdmin</tt></pre></div></div>
<div class="paragraph">
<p>Our software developers can now administer the resources assigned to
that pool.</p></div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="chapter_ha_manager">
15. High Availability
 <a class="headerlink" href="#chapter_ha_manager" title="Permalink to this heading"></a>
</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Our modern society depends heavily on information provided by
computers over the network. Mobile devices amplified that dependency,
because people can access the network any time from anywhere. If you
provide such services, it is very important that they are available
most of the time.</p></div>
<div class="paragraph">
<p>We can mathematically define the availability as the ratio of (A), the
total time a service is capable of being used during a given interval
to (B), the length of the interval. It is normally expressed as a
percentage of uptime in a given year.</p></div>
<table class="tableblock frame-all grid-all" style="
width:60%;
">
<caption class="title">Table 18. Availability - Downtime per Year</caption>
<colgroup><col style="width:50%;">
<col style="width:50%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Availability % </th>
<th class="tableblock halign-left valign-top">Downtime per year</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">99</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3.65 days</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">99.9</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">8.76 hours</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">99.99</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">52.56 minutes</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">99.999</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">5.26 minutes</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">99.9999</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">31.5 seconds</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">99.99999</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3.15 seconds</p></td>
</tr>
</tbody>
</table>
<div class="paragraph">
<p>There are several ways to increase availability. The most elegant
solution is to rewrite your software, so that you can run it on
several hosts at the same time. The software itself needs to have a way
to detect errors and do failover. If you only want to serve read-only
web pages, then this is relatively simple. However, this is generally complex
and sometimes impossible, because you cannot modify the software yourself. The
following solutions works without modifying the software:</p></div>
<div class="ulist"><ul>
<li>
<p>
Use reliable “server” components
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Computer components with the same functionality can have varying
reliability numbers, depending on the component quality. Most vendors
sell components with higher reliability as “server” components -
usually at higher price.</td>
</tr></tbody></table>
</div>
</li>
<li>
<p>
Eliminate single point of failure (redundant components)
</p>
<div class="ulist"><ul>
<li>
<p>
use an uninterruptible power supply (UPS)
</p>
</li>
<li>
<p>
use redundant power supplies in your servers
</p>
</li>
<li>
<p>
use ECC-RAM
</p>
</li>
<li>
<p>
use redundant network hardware
</p>
</li>
<li>
<p>
use RAID for local storage
</p>
</li>
<li>
<p>
use distributed, redundant storage for VM data
</p>
</li>
</ul></div>
</li>
<li>
<p>
Reduce downtime
</p>
<div class="ulist"><ul>
<li>
<p>
rapidly accessible administrators (24/7)
</p>
</li>
<li>
<p>
availability of spare parts (other nodes in a Proxmox VE cluster)
</p>
</li>
<li>
<p>
automatic error detection (provided by <span class="monospaced">ha-manager</span>)
</p>
</li>
<li>
<p>
automatic failover (provided by <span class="monospaced">ha-manager</span>)
</p>
</li>
</ul></div>
</li>
</ul></div>
<div class="paragraph">
<p>Virtualization environments like Proxmox VE make it much easier to reach
high availability because they remove the “hardware” dependency. They
also support the setup and use of redundant storage and network
devices, so if one host fails, you can simply start those services on
another host within your cluster.</p></div>
<div class="paragraph">
<p>Better still, Proxmox VE provides a software stack called <span class="monospaced">ha-manager</span>,
which can do that automatically for you. It is able to automatically
detect errors and do automatic failover.</p></div>
<div class="paragraph">
<p>Proxmox VE <span class="monospaced">ha-manager</span> works like an “automated” administrator. First, you
configure what resources (VMs, containers, …) it should
manage. Then, <span class="monospaced">ha-manager</span> observes the correct functionality, and handles
service failover to another node in case of errors. <span class="monospaced">ha-manager</span> can
also handle normal user requests which may start, stop, relocate and
migrate a service.</p></div>
<div class="paragraph">
<p>But high availability comes at a price. High quality components are
more expensive, and making them redundant doubles the costs at
least. Additional spare parts increase costs further. So you should
carefully calculate the benefits, and compare with those additional
costs.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Tip" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKZUlEQVRoge2aa3BU5RmAn3Pbs7fs
JmwCRGITk0hVLFAtNWoq6pAiU0cKaYfa6ShT+YN4YbQw9F/8QX+UMv6gM3Q6oxMV6TgIbe10Gq2g
cSzDpRaFgmIk4SKB3LP3Pff+SM66m+xuFvEyzvSbeefsbva8+z7nvXzf934RHMfhmzzEr9uAqx3/
B/i6xzceQP6iFDmT1cBxHNzCkFsgBEHIXnNeC1f7u1cN4DiOY9s2rliWhWVZWRDHcbJGC4KAJElI
koQoioii6IiieFUgnxvAtm3HNdg0Tbq6uuju7ubYsWP09vYyMjKCpmmoqkokEqGhoYGFCxfS2tpK
W1sbiqJkRZIkZxLoikGEK50H3CdumiZ9fX3s3LmT3bt3U1V3A0033cKc2nkEQxV4PSqSJOI4Dpqu
k0gkGLx8kZ4T7zF87iSrV69m3bp1NDY2oqoqHo8HWZa5Uo9cEYBt245lWRiGQUdHB9u2beOe1Y8w
/6bFVAT9xJJpYvEUiVSGjG5gmBY4DqIoonoUfF4PoYAfRRE5/8kp3njlD6xfv54tW7YQCATw+Xyu
R8r2RtkAtm07pmly5MgRHn/8cZSaZpbcfjd+n5f+wVEGRqJkdCMv3vME8t77vB6qQn4+OX6YsXPH
2bp1Ky0tLQQCAVRVdb0xI0RZZdQ1ft++fSxbtozrlqzgrnvvI5nRee9UL+f6h9B0A1EQEIsBiOKE
TL7XdJOBkTg1jYtouu1+1qxZw549e4hGo6TTaUzTxLbtGZ/ujEmca/wvHnqYnz/2DLNn19B74TID
I9HPjCvwlLMls4RHdMNC8IRZ8dBmnnp6E7Zts2rVKgB8Ph+yLDulPFEyhBzHcUzT5PDhwyxbtow1
j3YQqanmozOfEk2kChuLQ3x0lGQihmM7qF4vVdWz8fr9hYFyoK30OG/ufpYXXniB1tZWwuEwXq8X
WZaLJnZJAMuyHE3TuPPOO2lcsoLGpmZO9ZzPM37q0x0ZuISla2xY2077j5ZSFargZM9Znt97gE8u
DBb3ziRIfPAcF4/v59VXX6W6uppQKISqqkiSVBCgaA64odPR0YFS00xjUzNnLlwmmkznxbKYI45j
k04mefaZJ3j04VXMqZ6Fx6Pw3QXXs/3Xv6Tp2rnTALL3T8wDBCLz8M2Zz/bt24nFYjPmQ0EAt9b3
9fWxbds2ltxxD0NjMQbdmC+QlIIgIIkSoWCAH971/Wk6PYrCg/f/oHiVmhSP6qWm/gY6Ozvp6ekh
mUyi6zq2bWeXK+UAYFkWO3fu5N72dQT8Pi5cGp6xuoiiiBoMktH0gl5trp87DbqQBEMRbl32U3bt
2kUikUDTtOzypGwAwzDYvXs3316wiEuDoxiGWVaZrAjP4qW/vFUQ4NAHPdlwKQWiqF4qa+ro6uoi
kUiQTqcxDKM8ADd8Xn/9dWZdewMVwSCDo7GicT8NSBTZt/8oT259jgOHThBNpIgmUjy3dz/P7z2Q
r2My7gs9FNUXoPpbN9Ld3Z0FKBRG0+YBN3y6u7tpWnAr8WR6+gxLfr03TYNMMolhGFimiWVbXDzb
x4G3/4XgOIiyTF3DdW45nHG2RhBQfX6q65o5evQoy5cvn9BtWUiSRG5FLQhg2zbHjh3j+tsfKFrv
3R8EGL7UT23NLNraWmi+ro5r5kSYHakiVOHH7/OiyDKxZIonf9NJIpWZMQcEwOPx4vNXcPr0B2Qy
mdxEzrO34ExsWRa9vb3csjzEaP9w1sUFZ1RBQJJk/vjbTdTXzS2kDoBQwI9HmcEDOSJ7PAiiSH9/
P7quY5omlmVN01soB3Ach5GREbyqiqabM8a+NxAglcmvPOf7h9jR+WdOfNQLwNtHTzIeT+XFfdGC
IAiIogSOQzQaxTRNdy4ozwO2baNpGpIkY1j2RAJTeJ0jCAKRmtmcPHORmkglxz48y/5DJ3jrnUPM
b7iGxx7+MZZls/efR0rG/VQPgwMC2eQtZHxRAABVVbM3lEpgV178azcvvfYOgiCgZTJomsbGR9oR
BIHzl4YYGo2VlcCuWOaE5xVFwbbtqVHiCJOZXBQgEomg6zqSKOIUMrqER+LRKItvaubW78wH4NLQ
WNmx7+q1DB1ZkgmFQohifqS7xhcFEEWRhoYGEokEqkeeWPLmurcEiGPbpJJJfvbAPVl95/qHJyYv
mH5/EdG1FA5QW1ubzZvc8pm1deoHroKFCxcycPkiPlWdnmC5iTxlVk2n0wT9Xu69Y3FW51g8OfH3
ye+WnAgnRcukyKQSNDU1Icty7n65NACAJEm0trbSc/zfVAT9JZ/U1NWklslwx/duxqMoWX0Zzcy/
bwr0VCDT0NDTSS6f/ZBFixZlN/ySJJXnAVEUaWtrY6DvOIoiFlx5FhPLsrjl5uvzdPq8nsLfL6I3
FR1FlhUG+v5LS0tLtmtRlgcEYaL5pCgKq1ev5lzPKfxeT8FwKSQA115Tk6eztjpcsubn6rUMnfj4
MLHxIZYuXYrX683rVpQDIIiiiKIorFu3jn+8vIPKCt+0cCkG4m4Bc0fd3OqCoVIIJDo2iCQrvPu3
F1m5cmVeu6VQz6hgDrj1t7GxkfXr1/Px+wdRPcr02C+wmgxVVnLm3KU8ffNmVxX03lSgRHSEVGyc
oYt9tLe3U19fTzAYzAKUVYVyw0hVVbZs2cJw7/uYyZGSIeCCeFWVd499jGGaWX1zq8OfrYOKeC+T
ijM+cBHHsRju/Q9r164lFAoRDAbdPfEVAQiiKOLxeAgEAmzdupW/v/A7RLPEyjTHuGjKYMfLb3B5
eBzdMNl/+CSmZReN+0wqztDFs4iSxIE9O9mwYQPhcJhwOEwgEMhN4GkEZXUlYrEYe/bs4elfbWLF
Q5tQKyJlVaRy+kSJ6AhjA58iihJdf9rBUxufYPny5cyZM6esrkTJxpabzIFAgFWrVmHbNps3b+bu
n6wnVF2H4lHLmlULgZiGTmxkgGR8DNu2efOV3/PUxo20tbURiUSorKwkEAhkk7fYmLE36rZX0uk0
0WiUgwcP0tHRQcW8G5ndsIBgaBYe1TvtyRYDMXWNZGyU+Ngwkiwz+GkfQ73vsWHDBhYvXkwkEmHW
rFmEw2G3M1eyR1pWczcXIh6PMz4+zvbt2+ns7OS2+x6kanYdqjeA1xdAUb3IioIoSjg42JaJaejo
mTRaOoGeTiHJEvGxYd55rZP29nbWrl1LOBymqqqKyspKKioqyjK+bIBcCE3TSCaTxGIxenp62LVr
F11dXdTUL2BO/Xx8/goEUcSxbYSJ2EGS5IlzgnSC/r4PuXzmOEuXLmXlypXU19cTCoUIh8OEQqEr
7k5/7vOBdDpNMpkkkUiQSCTo7u7m6NGjnD59mv7+fqLRKIZhoCgKoVCI2tpampqaWLRoES0tLfh8
Pvx+P8FgkGAw+OWfD7gj94RG13U0TSOdTpNOp8lMbmQ0TcvbArrrK1mW8Xg8eL3e7BLB5/N9dSc0
uSP3jMwwjKy4G3AXwB0ugAsx5YzMndW//DOy3OFMjGwrxrKs7NX9LBfAneFFUcxec6rU5zqpvCqA
qTCT16/0nPgLA/i6xjf+Xw3+B2ll/uiqTaJTAAAAAElFTkSuQmCC">
</td>
<td class="content">Increasing availability from 99% to 99.9% is relatively
simple. But increasing availability from 99.9999% to 99.99999% is very
hard and costly. <span class="monospaced">ha-manager</span> has typical error detection and failover
times of about 2 minutes, so you can get no more than 99.999%
availability.</td>
</tr></tbody></table>
</div>
<div class="sect2">
<h3 id="_requirements_3">
<span>15.1. Requirements</span>
 <a class="headerlink" href="#_requirements_3" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>You must meet the following requirements before you start with HA:</p></div>
<div class="ulist"><ul>
<li>
<p>
at least three cluster nodes (to get reliable quorum)
</p>
</li>
<li>
<p>
shared storage for VMs and containers
</p>
</li>
<li>
<p>
hardware redundancy (everywhere)
</p>
</li>
<li>
<p>
use reliable “server” components
</p>
</li>
<li>
<p>
hardware watchdog - if not available we fall back to the
  linux kernel software watchdog (<span class="monospaced">softdog</span>)
</p>
</li>
<li>
<p>
optional hardware fencing devices
</p>
</li>
</ul></div>
</div>
<div class="sect2">
<h3 id="ha_manager_resources">
<span>15.2. Resources</span>
 <a class="headerlink" href="#ha_manager_resources" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>We call the primary management unit handled by <span class="monospaced">ha-manager</span> a
resource. A resource (also called “service”) is uniquely
identified by a service ID (SID), which consists of the resource type
and a type specific ID, for example <span class="monospaced">vm:100</span>. That example would be a
resource of type <span class="monospaced">vm</span> (virtual machine) with the ID 100.</p></div>
<div class="paragraph">
<p>For now we have two important resources types - virtual machines and
containers. One basic idea here is that we can bundle related software
into such a VM or container, so there is no need to compose one big
service from other services, as was done with <span class="monospaced">rgmanager</span>. In
general, a HA managed resource should not depend on other resources.</p></div>
</div>
<div class="sect2">
<h3 id="_management_tasks">
<span>15.3. Management Tasks</span>
 <a class="headerlink" href="#_management_tasks" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>This section provides a short overview of common management tasks. The
first step is to enable HA for a resource. This is done by adding the
resource to the HA resource configuration. You can do this using the
GUI, or simply use the command-line tool, for example:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># ha-manager add vm:100</pre>
</div></div>
<div class="paragraph">
<p>The HA stack now tries to start the resources and keep them
running. Please note that you can configure the “requested”
resources state. For example you may want the HA stack to stop the
resource:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># ha-manager set vm:100 --state stopped</pre>
</div></div>
<div class="paragraph">
<p>and start it again later:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># ha-manager set vm:100 --state started</pre>
</div></div>
<div class="paragraph">
<p>You can also use the normal VM and container management commands. They
automatically forward the commands to the HA stack, so</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># qm start 100</pre>
</div></div>
<div class="paragraph">
<p>simply sets the requested state to <span class="monospaced">started</span>. The same applies to <span class="monospaced">qm
stop</span>, which sets the requested state to <span class="monospaced">stopped</span>.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">The HA stack works fully asynchronous and needs to communicate
with other cluster members. Therefore, it takes some seconds until you see
the result of such actions.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>To view the current HA resource configuration use:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># ha-manager config
vm:100
        state stopped</pre>
</div></div>
<div class="paragraph">
<p>And you can view the actual HA manager and resource state with:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># ha-manager status
quorum OK
master node1 (active, Wed Nov 23 11:07:23 2016)
lrm elsa (active, Wed Nov 23 11:07:19 2016)
service vm:100 (node1, started)</pre>
</div></div>
<div class="paragraph">
<p>You can also initiate resource migration to other nodes:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># ha-manager migrate vm:100 node2</pre>
</div></div>
<div class="paragraph">
<p>This uses online migration and tries to keep the VM running. Online
migration needs to transfer all used memory over the network, so it is
sometimes faster to stop the VM, then restart it on the new node. This can be
done using the <span class="monospaced">relocate</span> command:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># ha-manager relocate vm:100 node2</pre>
</div></div>
<div class="paragraph">
<p>Finally, you can remove the resource from the HA configuration using
the following command:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># ha-manager remove vm:100</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">This does not start or stop the resource.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>But all HA related tasks can be done in the GUI, so there is no need to
use the command line at all.</p></div>
</div>
<div class="sect2">
<h3 id="_how_it_works_2">
<span>15.4. How It Works</span>
 <a class="headerlink" href="#_how_it_works_2" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>This section provides a detailed description of the Proxmox VE HA manager
internals. It describes all involved daemons and how they work
together. To provide HA, two daemons run on each node:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">pve-ha-lrm</span>
</dt>
<dd>
<p>
The local resource manager (LRM), which controls the services running on
the local node. It reads the requested states for its services from
the current manager status file and executes the respective commands.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">pve-ha-crm</span>
</dt>
<dd>
<p>
The cluster resource manager (CRM), which makes the cluster-wide
decisions. It sends commands to the LRM, processes the results,
and moves resources to other nodes if something fails. The CRM also
handles node fencing.
</p>
</dd>
</dl></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">
<div class="title">Locks in the LRM &amp; CRM</div>Locks are provided by our distributed configuration file system (pmxcfs).
They are used to guarantee that each LRM is active once and working. As an
LRM only executes actions when it holds its lock, we can mark a failed node
as fenced if we can acquire its lock. This then lets us recover any failed
HA services securely without any interference from the now unknown failed node.
This all gets supervised by the CRM which currently holds the manager master
lock.</td>
</tr></tbody></table>
</div>
<div class="sect3">
<h4 id="ha_manager_service_states">15.4.1. Service States
 <a class="headerlink" href="#ha_manager_service_states" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The CRM uses a service state enumeration to record the current service
state. This state is displayed on the GUI and can be queried using
the <span class="monospaced">ha-manager</span> command-line tool:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># ha-manager status
quorum OK
master elsa (active, Mon Nov 21 07:23:29 2016)
lrm elsa (active, Mon Nov 21 07:23:22 2016)
service ct:100 (elsa, stopped)
service ct:102 (elsa, started)
service vm:501 (elsa, started)</pre>
</div></div>
<div class="paragraph">
<p>Here is the list of possible states:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
stopped
</dt>
<dd>
<p>
Service is stopped (confirmed by LRM). If the LRM detects a stopped
service is still running, it will stop it again.
</p>
</dd>
<dt class="hdlist1">
request_stop
</dt>
<dd>
<p>
Service should be stopped. The CRM waits for confirmation from the
LRM.
</p>
</dd>
<dt class="hdlist1">
stopping
</dt>
<dd>
<p>
Pending stop request. But the CRM did not get the request so far.
</p>
</dd>
<dt class="hdlist1">
started
</dt>
<dd>
<p>
Service is active an LRM should start it ASAP if not already running.
If the Service fails and is detected to be not running the LRM
restarts it
(see <a href="#ha_manager_start_failure_policy">Start Failure Policy</a>).
</p>
</dd>
<dt class="hdlist1">
starting
</dt>
<dd>
<p>
Pending start request. But the CRM has not got any confirmation from the
LRM that the service is running.
</p>
</dd>
<dt class="hdlist1">
fence
</dt>
<dd>
<p>
Wait for node fencing as the service node is not inside the quorate cluster
partition (see <a href="#ha_manager_fencing">Fencing</a>).
As soon as node gets fenced successfully the service will be placed into the
recovery state.
</p>
</dd>
<dt class="hdlist1">
recovery
</dt>
<dd>
<p>
Wait for recovery of the service. The HA manager tries to find a new node where
the service can run on. This search depends not only on the list of online and
quorate nodes, but also if the service is a group member and how such a group
is limited.
As soon as a new available node is found, the service will be moved there and
initially placed into stopped state. If it’s configured to run the new node
will do so.
</p>
</dd>
<dt class="hdlist1">
freeze
</dt>
<dd>
<p>
Do not touch the service state. We use this state while we reboot a
node, or when we restart the LRM daemon
(see <a href="#ha_manager_package_updates">Package Updates</a>).
</p>
</dd>
<dt class="hdlist1">
ignored
</dt>
<dd>
<p>
Act as if the service were not managed by HA at all.
Useful, when full control over the service is desired temporarily, without
removing it from the HA configuration.
</p>
</dd>
<dt class="hdlist1">
migrate
</dt>
<dd>
<p>
Migrate service (live) to other node.
</p>
</dd>
<dt class="hdlist1">
error
</dt>
<dd>
<p>
Service is disabled because of LRM errors. Needs manual intervention
(see <a href="#ha_manager_error_recovery">Error Recovery</a>).
</p>
</dd>
<dt class="hdlist1">
queued
</dt>
<dd>
<p>
Service is newly added, and the CRM has not seen it so far.
</p>
</dd>
<dt class="hdlist1">
disabled
</dt>
<dd>
<p>
Service is stopped and marked as <span class="monospaced">disabled</span>
</p>
</dd>
</dl></div>
</div>
<div class="sect3">
<h4 id="ha_manager_lrm">15.4.2. Local Resource Manager
 <a class="headerlink" href="#ha_manager_lrm" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The local resource manager (<span class="monospaced">pve-ha-lrm</span>) is started as a daemon on
boot and waits until the HA cluster is quorate and thus cluster-wide
locks are working.</p></div>
<div class="paragraph">
<p>It can be in three states:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
wait for agent lock
</dt>
<dd>
<p>
The LRM waits for our exclusive lock. This is also used as idle state if no
service is configured.
</p>
</dd>
<dt class="hdlist1">
active
</dt>
<dd>
<p>
The LRM holds its exclusive lock and has services configured.
</p>
</dd>
<dt class="hdlist1">
lost agent lock
</dt>
<dd>
<p>
The LRM lost its lock, this means a failure happened and quorum was lost.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p>After the LRM gets in the active state it reads the manager status
file in <span class="monospaced">/etc/pve/ha/manager_status</span> and determines the commands it
has to execute for the services it owns.
For each command a worker gets started, these workers are running in
parallel and are limited to at most 4 by default. This default setting
may be changed through the datacenter configuration key <span class="monospaced">max_worker</span>.
When finished the worker process gets collected and its result saved for
the CRM.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">
<div class="title">Maximum Concurrent Worker Adjustment Tips</div>The default value of at most 4 concurrent workers may be unsuited for
a specific setup. For example, 4 live migrations may occur at the same
time, which can lead to network congestions with slower networks and/or
big (memory wise) services. Also, ensure that in the worst case, congestion is
at a minimum, even if this means lowering the <span class="monospaced">max_worker</span> value. On the
contrary, if you have a particularly powerful, high-end setup you may also want
to increase it.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>Each command requested by the CRM is uniquely identifiable by a UID. When
the worker finishes, its result will be processed and written in the LRM
status file <span class="monospaced">/etc/pve/nodes/&lt;nodename&gt;/lrm_status</span>. There the CRM may collect
it and let its state machine - respective to the commands output - act on it.</p></div>
<div class="paragraph">
<p>The actions on each service between CRM and LRM are normally always synced.
This means that the CRM requests a state uniquely marked by a UID, the LRM
then executes this action <strong>one time</strong> and writes back the result, which is also
identifiable by the same UID. This is needed so that the LRM does not
execute an outdated command.
The only exceptions to this behaviour are the <span class="monospaced">stop</span> and <span class="monospaced">error</span> commands;
these two do not depend on the result produced and are executed
always in the case of the stopped state and once in the case of
the error state.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">
<div class="title">Read the Logs</div>The HA Stack logs every action it makes. This helps to understand what
and also why something happens in the cluster. Here its important to see
what both daemons, the LRM and the CRM, did. You may use
<span class="monospaced">journalctl -u pve-ha-lrm</span> on the node(s) where the service is and
the same command for the pve-ha-crm on the node which is the current master.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="ha_manager_crm">15.4.3. Cluster Resource Manager
 <a class="headerlink" href="#ha_manager_crm" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The cluster resource manager (<span class="monospaced">pve-ha-crm</span>) starts on each node and
waits there for the manager lock, which can only be held by one node
at a time.  The node which successfully acquires the manager lock gets
promoted to the CRM master.</p></div>
<div class="paragraph">
<p>It can be in three states:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
wait for agent lock
</dt>
<dd>
<p>
The CRM waits for our exclusive lock. This is also used as idle state if no
service is configured
</p>
</dd>
<dt class="hdlist1">
active
</dt>
<dd>
<p>
The CRM holds its exclusive lock and has services configured
</p>
</dd>
<dt class="hdlist1">
lost agent lock
</dt>
<dd>
<p>
The CRM lost its lock, this means a failure happened and quorum was lost.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p>Its main task is to manage the services which are configured to be highly
available and try to always enforce the requested state. For example, a
service with the requested state <em>started</em> will be started if its not
already running. If it crashes it will be automatically started again.
Thus the CRM dictates the actions the LRM needs to execute.</p></div>
<div class="paragraph">
<p>When a node leaves the cluster quorum, its state changes to unknown.
If the current CRM can then secure the failed node’s lock, the services
will be <em>stolen</em> and restarted on another node.</p></div>
<div class="paragraph">
<p>When a cluster member determines that it is no longer in the cluster
quorum, the LRM waits for a new quorum to form. As long as there is no
quorum the node cannot reset the watchdog. This will trigger a reboot
after the watchdog times out (this happens after 60 seconds).</p></div>
</div>
</div>
<div class="sect2">
<h3 id="_ha_simulator">
<span>15.5. HA Simulator</span>
 <a class="headerlink" href="#_ha_simulator" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-ha-manager-status.png">
<img src="images/screenshot/gui-ha-manager-status.png" alt="screenshot/gui-ha-manager-status.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>By using the HA simulator you can test and learn all functionalities of the
Proxmox VE HA solutions.</p></div>
<div class="paragraph">
<p>By default, the simulator allows you to watch and test the behaviour of a
real-world 3 node cluster with 6 VMs. You can also add or remove additional VMs
or Container.</p></div>
<div class="paragraph">
<p>You do not have to setup or configure a real cluster, the HA simulator runs out
of the box.</p></div>
<div class="paragraph">
<p>Install with apt:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>apt install pve-ha-simulator</pre>
</div></div>
<div class="paragraph">
<p>You can even install the package on any Debian-based system without any
other Proxmox VE packages.  For that you will need to download the package and
copy it to the system you want to run it on for installation.  When you install
the package with apt from the local file system it will also resolve the
required dependencies for you.</p></div>
<div class="paragraph">
<p>To start the simulator on a remote machine you must have an X11 redirection to
your current system.</p></div>
<div class="paragraph">
<p>If you are on a Linux machine you can use:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>ssh root@&lt;IPofPVE&gt; -Y</pre>
</div></div>
<div class="paragraph">
<p>On Windows it works with <a href="https://mobaxterm.mobatek.net/">mobaxterm</a>.</p></div>
<div class="paragraph">
<p>After connecting to an existing Proxmox VE with the simulator installed or
installing it on your local Debian-based system manually, you can try it out as
follows.</p></div>
<div class="paragraph">
<p>First you need to create a working directory where the simulator saves its
current state and writes its default config:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>mkdir working</pre>
</div></div>
<div class="paragraph">
<p>Then, simply pass the created directory as a parameter to <em>pve-ha-simulator</em>:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>pve-ha-simulator working/</pre>
</div></div>
<div class="paragraph">
<p>You can then start, stop, migrate the simulated HA services, or even check out
what happens on a node failure.</p></div>
</div>
<div class="sect2">
<h3 id="_configuration_16">
<span>15.6. Configuration</span>
 <a class="headerlink" href="#_configuration_16" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>The HA stack is well integrated into the Proxmox VE API. So, for example,
HA can be configured via the <span class="monospaced">ha-manager</span> command-line interface, or
the Proxmox VE web interface - both interfaces provide an easy way to
manage HA. Automation tools can use the API directly.</p></div>
<div class="paragraph">
<p>All HA configuration files are within <span class="monospaced">/etc/pve/ha/</span>, so they get
automatically distributed to the cluster nodes, and all nodes share
the same HA configuration.</p></div>
<div class="sect3">
<h4 id="ha_manager_resource_config">15.6.1. Resources
 <a class="headerlink" href="#ha_manager_resource_config" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-ha-manager-status.png">
<img src="images/screenshot/gui-ha-manager-status.png" alt="screenshot/gui-ha-manager-status.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>The resource configuration file <span class="monospaced">/etc/pve/ha/resources.cfg</span> stores
the list of resources managed by <span class="monospaced">ha-manager</span>. A resource configuration
inside that list looks like this:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>&lt;type&gt;: &lt;name&gt;
        &lt;property&gt; &lt;value&gt;
        ...</pre>
</div></div>
<div class="paragraph">
<p>It starts with a resource type followed by a resource specific name,
separated with colon. Together this forms the HA resource ID, which is
used by all <span class="monospaced">ha-manager</span> commands to uniquely identify a resource
(example: <span class="monospaced">vm:100</span> or <span class="monospaced">ct:101</span>). The next lines contain additional
properties:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">comment</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Description.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">group</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The HA group identifier.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">max_relocate</span>: <span class="monospaced">&lt;integer&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Maximal number of service relocate tries when a service failes to start.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">max_restart</span>: <span class="monospaced">&lt;integer&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Maximal number of tries to restart the service on a node after its start failed.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">state</span>: <span class="monospaced">&lt;disabled | enabled | ignored | started | stopped&gt;</span> (<em>default =</em> <span class="monospaced">started</span>)
</dt>
<dd>
<p>
Requested resource state. The CRM reads this state and acts accordingly.
Please note that <span class="monospaced">enabled</span> is just an alias for <span class="monospaced">started</span>.
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">started</span>
</dt>
<dd>
<div class="paragraph">
<p>The CRM tries to start the resource. Service state is
set to <span class="monospaced">started</span> after successful start. On node failures, or when start
fails, it tries to recover the resource.  If everything fails, service
state it set to <span class="monospaced">error</span>.</p></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">stopped</span>
</dt>
<dd>
<div class="paragraph">
<p>The CRM tries to keep the resource in <span class="monospaced">stopped</span> state, but it
still tries to relocate the resources on node failures.</p></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">disabled</span>
</dt>
<dd>
<div class="paragraph">
<p>The CRM tries to put the resource in <span class="monospaced">stopped</span> state, but does not try
to relocate the resources on node failures. The main purpose of this
state is error recovery, because it is the only way to move a resource out
of the <span class="monospaced">error</span> state.</p></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">ignored</span>
</dt>
<dd>
<div class="paragraph">
<p>The resource gets removed from the manager status and so the CRM and the LRM do
not touch the resource anymore. All {pve} API calls affecting this resource
will be executed, directly bypassing the HA stack. CRM commands will be thrown
away while there source is in this state. The resource will not get relocated
on node failures.</p></div>
</dd>
</dl></div>
</dd>
</dl></div>
<div class="paragraph">
<p>Here is a real world example with one VM and one container. As you see,
the syntax of those files is really simple, so it is even possible to
read or edit those files using your favorite editor:</p></div>
<div class="listingblock">
<div class="title">Configuration Example (<span class="monospaced">/etc/pve/ha/resources.cfg</span>)</div>
<div class="content monospaced">
<pre>vm: 501
    state started
    max_relocate 2

ct: 102
    # Note: use default settings for everything</pre>
</div></div>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-ha-manager-add-resource.png">
<img src="images/screenshot/gui-ha-manager-add-resource.png" alt="screenshot/gui-ha-manager-add-resource.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>The above config was generated using the <span class="monospaced">ha-manager</span> command-line tool:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># ha-manager add vm:501 --state started --max_relocate 2
# ha-manager add ct:102</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="ha_manager_groups">15.6.2. Groups
 <a class="headerlink" href="#ha_manager_groups" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-ha-manager-groups-view.png">
<img src="images/screenshot/gui-ha-manager-groups-view.png" alt="screenshot/gui-ha-manager-groups-view.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>The HA group configuration file <span class="monospaced">/etc/pve/ha/groups.cfg</span> is used to
define groups of cluster nodes. A resource can be restricted to run
only on the members of such group. A group configuration look like
this:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>group: &lt;group&gt;
       nodes &lt;node_list&gt;
       &lt;property&gt; &lt;value&gt;
       ...</pre>
</div></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">comment</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Description.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">nodes</span>: <span class="monospaced">&lt;node&gt;[:&lt;pri&gt;]{,&lt;node&gt;[:&lt;pri&gt;]}*</span> 
</dt>
<dd>
<p>
List of cluster node members, where a priority can be given to each node. A resource bound to a group will run on the available nodes with the highest priority. If there are more nodes in the highest priority class, the services will get distributed to those nodes. The priorities have a relative meaning only.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">nofailback</span>: <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
The CRM tries to run services on the node with the highest priority. If a node with higher priority comes online, the CRM migrates the service to that node. Enabling nofailback prevents that behavior.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">restricted</span>: <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Resources bound to restricted groups may only run on nodes defined by the group. The resource will be placed in the stopped state if no group node member is online. Resources on unrestricted groups may run on any cluster node if all group members are offline, but they will migrate back as soon as a group member comes online. One can implement a <em>preferred node</em> behavior using an unrestricted group with only one member.
</p>
</dd>
</dl></div>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-ha-manager-add-group.png">
<img src="images/screenshot/gui-ha-manager-add-group.png" alt="screenshot/gui-ha-manager-add-group.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>A common requirement is that a resource should run on a specific
node. Usually the resource is able to run on other nodes, so you can define
an unrestricted group with a single member:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># ha-manager groupadd prefer_node1 --nodes node1</pre>
</div></div>
<div class="paragraph">
<p>For bigger clusters, it makes sense to define a more detailed failover
behavior. For example, you may want to run a set of services on
<span class="monospaced">node1</span> if possible. If <span class="monospaced">node1</span> is not available, you want to run them
equally split on <span class="monospaced">node2</span> and <span class="monospaced">node3</span>. If those nodes also fail, the
services should run on <span class="monospaced">node4</span>. To achieve this you could set the node
list to:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># ha-manager groupadd mygroup1 -nodes "node1:2,node2:1,node3:1,node4"</pre>
</div></div>
<div class="paragraph">
<p>Another use case is if a resource uses other resources only available
on specific nodes, lets say <span class="monospaced">node1</span> and <span class="monospaced">node2</span>. We need to make sure
that HA manager does not use other nodes, so we need to create a
restricted group with said nodes:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># ha-manager groupadd mygroup2 -nodes "node1,node2" -restricted</pre>
</div></div>
<div class="paragraph">
<p>The above commands created the following group configuration file:</p></div>
<div class="listingblock">
<div class="title">Configuration Example (<span class="monospaced">/etc/pve/ha/groups.cfg</span>)</div>
<div class="content monospaced">
<pre>group: prefer_node1
       nodes node1

group: mygroup1
       nodes node2:1,node4,node1:2,node3:1

group: mygroup2
       nodes node2,node1
       restricted 1</pre>
</div></div>
<div class="paragraph">
<p>The <span class="monospaced">nofailback</span> options is mostly useful to avoid unwanted resource
movements during administration tasks. For example, if you need to
migrate a service to a node which doesn’t have the highest priority in the
group, you need to tell the HA manager not to instantly move this service
back by setting the <span class="monospaced">nofailback</span> option.</p></div>
<div class="paragraph">
<p>Another scenario is when a service was fenced and it got recovered to
another node. The admin tries to repair the fenced node and brings it
up online again to investigate the cause of failure and check if it runs
stably again. Setting the <span class="monospaced">nofailback</span> flag prevents the recovered services from
moving straight back to the fenced node.</p></div>
</div>
</div>
<div class="sect2">
<h3 id="ha_manager_fencing">
<span>15.7. Fencing</span>
 <a class="headerlink" href="#ha_manager_fencing" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>On node failures, fencing ensures that the erroneous node is
guaranteed to be offline. This is required to make sure that no
resource runs twice when it gets recovered on another node. This is a
really important task, because without this, it would not be possible to
recover a resource on another node.</p></div>
<div class="paragraph">
<p>If a node did not get fenced, it would be in an unknown state where
it may have still access to shared resources. This is really
dangerous! Imagine that every network but the storage one broke. Now,
while not reachable from the public network, the VM still runs and
writes to the shared storage.</p></div>
<div class="paragraph">
<p>If we then simply start up this VM on another node, we would get a
dangerous race condition, because we write from both nodes. Such
conditions can destroy all VM data and the whole VM could be rendered
unusable. The recovery could also fail if the storage protects against
multiple mounts.</p></div>
<div class="sect3">
<h4 id="_how_proxmox_ve_fences">15.7.1. How Proxmox VE Fences
 <a class="headerlink" href="#_how_proxmox_ve_fences" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>There are different methods to fence a node, for example, fence
devices which cut off the power from the node or disable their
communication completely. Those are often quite expensive and bring
additional critical components into a system, because if they fail you
cannot recover any service.</p></div>
<div class="paragraph">
<p>We thus wanted to integrate a simpler fencing method, which does not
require additional external hardware. This can be done using
watchdog timers.</p></div>
<div class="ulist"><div class="title">Possible Fencing Methods</div><ul>
<li>
<p>
external power switches
</p>
</li>
<li>
<p>
isolate nodes by disabling complete network traffic on the switch
</p>
</li>
<li>
<p>
self fencing using watchdog timers
</p>
</li>
</ul></div>
<div class="paragraph">
<p>Watchdog timers have been widely used in critical and dependable systems
since the beginning of microcontrollers. They are often simple, independent
integrated circuits which are used to detect and recover from computer malfunctions.</p></div>
<div class="paragraph">
<p>During normal operation, <span class="monospaced">ha-manager</span> regularly resets the watchdog
timer to prevent it from elapsing. If, due to a hardware fault or
program error, the computer fails to reset the watchdog, the timer
will elapse and trigger a reset of the whole server (reboot).</p></div>
<div class="paragraph">
<p>Recent server motherboards often include such hardware watchdogs, but
these need to be configured. If no watchdog is available or
configured, we fall back to the Linux Kernel <em>softdog</em>. While still
reliable, it is not independent of the servers hardware, and thus has
a lower reliability than a hardware watchdog.</p></div>
</div>
<div class="sect3">
<h4 id="_configure_hardware_watchdog">15.7.2. Configure Hardware Watchdog
 <a class="headerlink" href="#_configure_hardware_watchdog" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>By default, all hardware watchdog modules are blocked for security
reasons. They are like a loaded gun if not correctly initialized. To
enable a hardware watchdog, you need to specify the module to load in
<em>/etc/default/pve-ha-manager</em>, for example:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># select watchdog module (default is softdog)
WATCHDOG_MODULE=iTCO_wdt</pre>
</div></div>
<div class="paragraph">
<p>This configuration is read by the <em>watchdog-mux</em> service, which loads
the specified module at startup.</p></div>
</div>
<div class="sect3">
<h4 id="_recover_fenced_services">15.7.3. Recover Fenced Services
 <a class="headerlink" href="#_recover_fenced_services" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>After a node failed and its fencing was successful, the CRM tries to
move services from the failed node to nodes which are still online.</p></div>
<div class="paragraph">
<p>The selection of nodes, on which those services gets recovered, is
influenced by the resource <span class="monospaced">group</span> settings, the list of currently active
nodes, and their respective active service count.</p></div>
<div class="paragraph">
<p>The CRM first builds a set out of the intersection between user selected
nodes (from <span class="monospaced">group</span> setting) and available nodes. It then choose the
subset of nodes with the highest priority, and finally select the node
with the lowest active service count. This minimizes the possibility
of an overloaded node.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Caution" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAKdUlEQVRoge1Ze1AV1x3+zt279wEi
DWCYGzRVktqa1MEmmtbWR22ncXxUrTDWV/5IqG2wUAUfmUwSkk6V+EAJCEQC6figwZBqZtRxqukf
tebRZBpFG1S0hiRErwiaKK977+6eX//YPXt37+UiIJlMZnpmdnb33PP4vt97z2VEhG9yc3zdAO60
/Z/A192+8QScX8Wifr+fWltbzffU1FT4fD72Vew15ASampqovr4eBw8eNPvGjRuHzMxMmj9//tCT
IKIhu958801yuVwEoNdr48aNNJT7EdHQEdi/fz/JshwTvLiKioqGlMSQLFJfX0+MMRvQsWNSqXLz
H2jez78fRWLTpk1DRuKOF6irqyOn02kDOP8XGdTz+VFSP91Hatu79NRvZ0SR2LJly5CQuKPJtbW1
JEmSDdivHp1AwctHSblYSsqFElLOF5PavIvWZU+LIrF169Y7JjHoibt3744CnzXrIQpeOUbKhVJS
mkpIPV9MyrnNpJwtIuVSNa15/MdRJLZt23ZHJBjRwIu5N954gxYvXgzOudm3aO5E1L5SBNZ1Hoxr
INLAyH6HJxVP/WkXtu9+37ZeSUkJVq9ePagQO+BMXFNTEwV+ybxHsPeVIjg6zwOkgSWNhuOuUUBC
KhxJY0AggDSgpwWbn1mO1csftq1ZUFCA0tLSQZXFA0pkVVVVlJOTA6vWsmY9hF07N0AS4OU48LgR
WLO+FPFxboxMS8G5hlPYmPMI4jwM1P0Jtj67DJyrKHvtNADdjPPz8+FwOCgvL29gmuivrVVUVESF
yqxZD1PoyjFSL+7QHbapmJTml+n9EzVRtl676ZekNKwl5VQ+KSfzKHR2E+UutodYxhiVl5cPyCf6
ZUJlZWWUm5trk/wTi6bgtVdf1M2Gq2DEwUgDNBWhUDBqjfSRwwGoIOEP3c3Y/uxirFw0zibMvLw8
VFZW9tucbkugoqKCVq1aZQO/YslUVJU+D9bVBIDroLgK4hygEDweybYGY8AD6d8yftfASL+j+2O8
9NwiPJk51kYiNzcXO3fu7BeJPgkUFxdTbm6ure/JZdNRub0QrOMcGFcBrgLgAHEAGqD2ID7OZZsz
Ji0RcR6HAV4DkaqPJxXouoSywkz8buF9NhIrV65EVVXVbUnEJHDixAlat26drS9n+U+xo/hZMMNh
yZA+uAbAIKOGMCJlmG3eA/clAYaJEamGBlRdC6QBXZewo3AhVswfbSORk5ODDz/8sE8SvRLw+/1U
XV0dBb5MgOe6LUNIEqpOwgCUlOCESw4HuAfG3KX7B2kAGaTJ0Jro77yI8ucX4DfzRtlIHD58GH6/
PyaJXgk0NjZi79695vuUiffrkr91Vt8cHAwc4FZJGiZEKqDcwsi0ZHP+d749POy8MS7iGtBxARWF
8/CDsQnm3BdeeAHWj6N+EYhspSVbgI5zYYkZGmAwzIEbwA0tUM9NfO+7aeb89HuGRQBWLcSNZxj3
zvOYMzk5NpjBEHi99mXD5lXdjqGDJh42h7CEOXhnK3448X5z/shUb1jSpIEM8yFuzOUcjAsNcnR0
KXdG4MEHH8TEiRPN9z0H3sfRE41m+GNcs9i0LjkGTTcrUoGem5g7/V5z/ohElw4O+jxGYQ3qJqmB
jN/+cbINtcfCJvPYY4+hr3qtVwI+n49lZmaa71fbbuH3hX/FycbPQKSCoIIMQEQcehjVo5Gw8/F3
3cSvH03H2HuHY1icQzcvYWKGIMiITMwQwjtnruPpqk9w/ZZq7p2RkYEJEyaAYrCIWY36/X7KyMhA
W1ub2XfP3Qmo3ZKFaQ+P0gmYIdSoOkU4lWSwxGRAdqHL/xm86k27w0blAw3/PncD6yo/xrsfdZj7
paSkoLGxESNGjAgDZsxWK8X0AZ/Px44fP460tLAzXrnWgUX5r+PYuxf1mG/Gf80sofWy2YuWbg/+
uONtPLHhPRw+0WIHL8hbwG95rcUGPiEhAYcOHUJSUhI456YZRWqiz+8BIqLm5mZMmjQJN27cMPsT
E9zYtWE25k4bYzh3mABIBRwOHPgPx+LH/6yDiXPi6pHZcEpk+I5qmtGpCzdQecCPPUevmevLsowj
R45gypQpcDgc5mVowKaJmBoQTEePHo2GhgaMGxcuum52BLHsqUOoP3oWgLB9Sz7QgpgxJoThw/SS
IutnaXBKZEpcgD998Uvs+3ubDbzT6URNTQ0mTZoEVVWhaRo459A0TeCy4etVA+JH8RvnHK2trZgz
Zw5Onz5tjnPJDrxSOAPLZt1nSVIqwDmIVJw6dx1nLt3A3J/cjZThkhE29cvf3o2KA59ja91lcz3G
GDZv3oylS5fC7XbD6XTC6XRCkiRIktS7Jm5HQLDXNA3Xr19HVlYWPvjgA4vEGLav+RHmTxsJX4rb
8AsO4kbOIG6GzTD4HvzlmB/PVH9q2zc/Px/Z2dnwer1wu92QZRkulwuSJEGWZZMEY8wkEGVCVvCC
AOcciqIgPj4e+/btw9SpU83xqkpYteU9HHm7Bf62rnC2RjikisTFSMPV9m4ceucannvVDn7p0qVY
uHAhgsEggsEgQqFQlAlZnVm0PjOxIKFpGogIqqrC5XKhuroa06dPt4wDcl78F+qPNeNKe6cRoTjs
ZYOKK+3dON7wBQrKm2H5pMbMmTOxZMkSBAIB9PT0oKenB4FAAIqiQFEUU4hRRyq3I2AlIZ4553A6
nSgvL8fs2bNtY9eVNaDub58YmhDZWpe+v70bJ5tu4clt/4WihqU4efJkrFixAkRkAg4Gg1AUBaqq
muDF/pGtX7WQNXeIZ0mSUFRUhAULFtjGPl3ZiJdev4QrbZ16zUQqrrZ342JLN5ZvaEJ3ICz68ePH
IycnB5qmmZIW4EOhUJTZWEKo+Rx1KsEYY9Zk4XA4IEmSKXnOuRkVJElCYWEhvF4v6urqzDVK9n2M
zu4Qnl5+L9q/7MHltgBWbr9kA5+eno7c3FzIsoyI5GrTeCzgosU8VmGMmSGLiEwSsixH2WNBQQEA
2EhUH/wc7330BaaOH479/2zHtS/CFabP58OaNWvg9Xp1EE4nZFmGLMtwu93wer1wuVy2ECpJko2I
iTNWJrZGI+HEwi6FisUVCAQQCASwZ88eVFZW9lk9JicnY/369fD5fPB4PCbo+Ph4k4TH44HL5YLX
64XX6zXH9EKE9aUBRkTEGLNJP1Kd4nI4HMjOzkZ8fDyKi4ttJ3eiJSYmYu3atRg1apQJzuv1wuPx
wOPxwOFwmEBFHnA6nXC5XGYSiywlbns2SnoDANN0hMNZnS0UCpkaOnPmDN566y20tLSY2ktLS8OM
GTOQkpICt9uNuLg4M2EJ8CJZSZJkZmFBwkrAWpH2+3BXEBEkRIhTVdVMOCL5qKqKy5cvQ1EUaJoG
VVWRnKx/Jlpt3eVy2QCKu9C68AFr9o0spwd0Oh1ZYggCIlNaiQktiXdhZgKQABcp3ci7LMtM7B0J
fsAErJoAYItG1mdr+hfvVlACtHDIXkploYXbHvQOikDEe6/ZmjFmAhf9ZvJxOk2Qol88W4kYz32S
GNQfHLG+TyMTkAVEr88xQfVnkBg7GAKixSIykDYQsL21/wFkW/B5QqT9lwAAAABJRU5ErkJggg==">
</td>
<td class="content">On node failure, the CRM distributes services to the
remaining nodes. This increases the service count on those nodes, and
can lead to high load, especially on small clusters. Please design
your cluster so that it can handle such worst case scenarios.</td>
</tr></tbody></table>
</div>
</div>
</div>
<div class="sect2">
<h3 id="ha_manager_start_failure_policy">
<span>15.8. Start Failure Policy</span>
 <a class="headerlink" href="#ha_manager_start_failure_policy" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>The start failure policy comes into effect if a service failed to start on a
node one or more times. It can be used to configure how often a restart
should be triggered on the same node and how often a service should be
relocated, so that it has an attempt to be started on another node.
The aim of this policy is to circumvent temporary unavailability of shared
resources on a specific node. For example, if a shared storage isn’t available
on a quorate node anymore, for instance due to network problems, but is still
available on other nodes, the relocate policy allows the service to start
nonetheless.</p></div>
<div class="paragraph">
<p>There are two service start recover policy settings which can be configured
specific for each resource.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
max_restart
</dt>
<dd>
<p>
Maximum number of attempts to restart a failed service on the actual
node.  The default is set to one.
</p>
</dd>
<dt class="hdlist1">
max_relocate
</dt>
<dd>
<p>
Maximum number of attempts to relocate the service to a different node.
A relocate only happens after the max_restart value is exceeded on the
actual node. The default is set to one.
</p>
</dd>
</dl></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">The relocate count state will only reset to zero when the
service had at least one successful start. That means if a service is
re-started without fixing the error only the restart policy gets
repeated.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect2">
<h3 id="ha_manager_error_recovery">
<span>15.9. Error Recovery</span>
 <a class="headerlink" href="#ha_manager_error_recovery" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>If, after all attempts, the service state could not be recovered, it gets
placed in an error state. In this state, the service won’t get touched
by the HA stack anymore. The only way out is disabling a service:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># ha-manager set vm:100 --state disabled</pre>
</div></div>
<div class="paragraph">
<p>This can also be done in the web interface.</p></div>
<div class="paragraph">
<p>To recover from the error state you should do the following:</p></div>
<div class="ulist"><ul>
<li>
<p>
bring the resource back into a safe and consistent state (e.g.:
kill its process if the service could not be stopped)
</p>
</li>
<li>
<p>
disable the resource to remove the error flag
</p>
</li>
<li>
<p>
fix the error which led to this failures
</p>
</li>
<li>
<p>
<strong>after</strong> you fixed all errors you may request that the service starts again
</p>
</li>
</ul></div>
</div>
<div class="sect2">
<h3 id="ha_manager_package_updates">
<span>15.10. Package Updates</span>
 <a class="headerlink" href="#ha_manager_package_updates" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>When updating the ha-manager, you should do one node after the other, never
all at once for various reasons. First, while we test our software
thoroughly, a bug affecting your specific setup cannot totally be ruled out.
Updating one node after the other and checking the functionality of each node
after finishing the update helps to recover from eventual problems, while
updating all at once could result in a broken cluster and is generally not
good practice.</p></div>
<div class="paragraph">
<p>Also, the Proxmox VE HA stack uses a request acknowledge protocol to perform
actions between the cluster and the local resource manager. For restarting,
the LRM makes a request to the CRM to freeze all its services. This prevents
them from getting touched by the Cluster during the short time the LRM is restarting.
After that, the LRM may safely close the watchdog during a restart.
Such a restart happens normally during a package update and, as already stated,
an active master CRM is needed to acknowledge the requests from the LRM. If
this is not the case the update process can take too long which, in the worst
case, may result in a reset triggered by the watchdog.</p></div>
</div>
<div class="sect2">
<h3 id="ha_manager_node_maintenance">
<span>15.11. Node Maintenance</span>
 <a class="headerlink" href="#ha_manager_node_maintenance" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Sometimes it is necessary to perform maintenance on a node, such as replacing
hardware or simply installing a new kernel image. This also applies while the
HA stack is in use.</p></div>
<div class="paragraph">
<p>The HA stack can support you mainly in two types of maintenance:</p></div>
<div class="ulist"><ul>
<li>
<p>
for general shutdowns or reboots, the behavior can be configured, see
  <a href="#ha_manager_shutdown_policy">Shutdown Policy</a>.
</p>
</li>
<li>
<p>
for maintenance that does not require a shutdown or reboot, or that should
  not be switched off automatically after only one reboot, you can enable the
  manual maintenance mode.
</p>
</li>
</ul></div>
<div class="sect3">
<h4 id="_maintenance_mode">15.11.1. Maintenance Mode
 <a class="headerlink" href="#_maintenance_mode" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>You can use the manual maintenance mode to mark the node as unavailable for HA
operation, prompting all services managed by HA to migrate to other nodes.</p></div>
<div class="paragraph">
<p>The target nodes for these migrations are selected from the other currently
available nodes, and determined by the HA group configuration and the configured
cluster resource scheduler (CRS) mode.
During each migration, the original node will be recorded in the HA managers'
state, so that the service can be moved back again automatically once the
maintenance mode is disabled and the node is back online.</p></div>
<div class="paragraph">
<p>Currently you can enabled or disable the maintenance mode using the ha-manager
CLI tool.</p></div>
<div class="listingblock">
<div class="title">Enabling maintenance mode for a node</div>
<div class="content monospaced">
<pre># ha-manager crm-command node-maintenance enable NODENAME</pre>
</div></div>
<div class="paragraph">
<p>This will queue a CRM command, when the manager processes this command it will
record the request for maintenance-mode in the manager status. This allows you
to submit the command on any node, not just on the one you want to place in, or
out of the maintenance mode.</p></div>
<div class="paragraph">
<p>Once the LRM on the respective node picks the command up it will mark itself as
unavailable, but still process all migration commands. This means that the LRM
self-fencing watchdog will stay active until all active services got moved, and
all running workers finished.</p></div>
<div class="paragraph">
<p>Note that the LRM status will read <span class="monospaced">maintenance</span> mode as soon as the LRM
picked the requested state up, not only when all services got moved away, this
user experience is planned to be improved in the future.
For now, you can check for any active HA service left on the node, or watching
out for a log line like: <span class="monospaced">pve-ha-lrm[PID]: watchdog closed (disabled)</span> to know
when the node finished its transition into the maintenance mode.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">The manual maintenance mode is not automatically deleted on node reboot,
but only if it is either manually deactivated using the <span class="monospaced">ha-manager</span> CLI or if
the manager-status is manually cleared.</td>
</tr></tbody></table>
</div>
<div class="listingblock">
<div class="title">Disabling maintenance mode for a node</div>
<div class="content monospaced">
<pre># ha-manager crm-command node-maintenance disable NODENAME</pre>
</div></div>
<div class="paragraph">
<p>The process of disabling the manual maintenance mode is similar to enabling it.
Using the <span class="monospaced">ha-manager</span> CLI command shown above will queue a CRM command that,
once processed, marks the respective LRM node as available again.</p></div>
<div class="paragraph">
<p>If you deactivate the maintenance mode, all services that were on the node when
the maintenance mode was activated will be moved back.</p></div>
</div>
<div class="sect3">
<h4 id="ha_manager_shutdown_policy">15.11.2. Shutdown Policy
 <a class="headerlink" href="#ha_manager_shutdown_policy" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Below you will find a description of the different HA policies for a node
shutdown. Currently <em>Conditional</em> is the default due to backward compatibility.
Some users may find that <em>Migrate</em> behaves more as expected.</p></div>
<div class="paragraph">
<p>The shutdown policy can be configured in the Web UI (<span class="monospaced">Datacenter</span> → <span class="monospaced">Options</span>
→ <span class="monospaced">HA Settings</span>), or directly in <span class="monospaced">datacenter.cfg</span>:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>ha: shutdown_policy=&lt;value&gt;</pre>
</div></div>
<div class="sect4">
<h5 id="_migrate">Migrate
 <a class="headerlink" href="#_migrate" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Once the Local Resource manager (LRM) gets a shutdown request and this policy
is enabled, it will mark itself as unavailable for the current HA manager.
This triggers a migration of all HA Services currently located on this node.
The LRM will try to delay the shutdown process, until all running services get
moved away. But, this expects that the running services <strong>can</strong> be migrated to
another node. In other words, the service must not be locally bound, for example
by using hardware passthrough. As non-group member nodes are considered as
runnable target if no group member is available, this policy can still be used
when making use of HA groups with only some nodes selected. But, marking a group
as <em>restricted</em> tells the HA manager that the service cannot run outside of the
chosen set of nodes. If all of those nodes are unavailable, the shutdown will
hang until you manually intervene. Once the shut down node comes back online
again, the previously displaced services will be moved back, if they were not
already manually migrated in-between.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">The watchdog is still active during the migration process on shutdown.
If the node loses quorum it will be fenced and the services will be recovered.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>If you start a (previously stopped) service on a node which is currently being
maintained, the node needs to be fenced to ensure that the service can be moved
and started on another available node.</p></div>
</div>
<div class="sect4">
<h5 id="_failover">Failover
 <a class="headerlink" href="#_failover" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>This mode ensures that all services get stopped, but that they will also be
recovered, if the current node is not online soon. It can be useful when doing
maintenance on a cluster scale, where live-migrating VMs may not be possible if
too many nodes are powered off at a time, but you still want to ensure HA
services get recovered and started again as soon as possible.</p></div>
</div>
<div class="sect4">
<h5 id="_freeze">Freeze
 <a class="headerlink" href="#_freeze" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>This mode ensures that all services get stopped and frozen, so that they won’t
get recovered until the current node is online again.</p></div>
</div>
<div class="sect4">
<h5 id="_conditional">Conditional
 <a class="headerlink" href="#_conditional" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>The <em>Conditional</em> shutdown policy automatically detects if a shutdown or a
reboot is requested, and changes behaviour accordingly.</p></div>
<div class="paragraph">
<div class="title">Shutdown</div><p>A shutdown (<em>poweroff</em>) is usually done if it is planned for the node to stay
down for some time. The LRM stops all managed services in this case. This means
that other nodes will take over those services afterwards.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Recent hardware has large amounts of memory (RAM). So we stop all
resources, then restart them to avoid online migration of all that RAM. If you
want to use online migration, you need to invoke that manually before you
shutdown the node.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<div class="title">Reboot</div><p>Node reboots are initiated with the <em>reboot</em> command. This is usually done
after installing a new kernel. Please note that this is different from
“shutdown”, because the node immediately starts again.</p></div>
<div class="paragraph">
<p>The LRM tells the CRM that it wants to restart, and waits until the CRM puts
all resources into the <span class="monospaced">freeze</span> state (same mechanism is used for
<a href="#ha_manager_package_updates">Package Updates</a>). This prevents those resources
from being moved to other nodes. Instead, the CRM starts the resources after the
reboot on the same node.</p></div>
</div>
<div class="sect4">
<h5 id="_manual_resource_movement">Manual Resource Movement
 <a class="headerlink" href="#_manual_resource_movement" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Last but not least, you can also manually move resources to other nodes, before
you shutdown or restart a node. The advantage is that you have full control,
and you can decide if you want to use online migration or not.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Please do not <em>kill</em> services like <span class="monospaced">pve-ha-crm</span>, <span class="monospaced">pve-ha-lrm</span> or
<span class="monospaced">watchdog-mux</span>. They manage and use the watchdog, so this can result in an
immediate node reboot or even reset.</td>
</tr></tbody></table>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="ha_manager_crs">
<span>15.12. Cluster Resource Scheduling</span>
 <a class="headerlink" href="#ha_manager_crs" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>The cluster resource scheduler (CRS) mode controls how HA selects nodes for the
recovery of a service as well as for migrations that are triggered by a
shutdown policy. The default mode is <span class="monospaced">basic</span>, you can change it in the Web UI
(<span class="monospaced">Datacenter</span> → <span class="monospaced">Options</span>), or directly in <span class="monospaced">datacenter.cfg</span>:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>crs: ha=static</pre>
</div></div>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-datacenter-options-crs.png">
<img src="images/screenshot/gui-datacenter-options-crs.png" alt="screenshot/gui-datacenter-options-crs.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>The change will be in effect starting with the next manager round (after a few
seconds).</p></div>
<div class="paragraph">
<p>For each service that needs to be recovered or migrated, the scheduler
iteratively chooses the best node among the nodes with the highest priority in
the service’s group.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">There are plans to add modes for (static and dynamic) load-balancing in
the future.</td>
</tr></tbody></table>
</div>
<div class="sect3">
<h4 id="_basic_scheduler">15.12.1. Basic Scheduler
 <a class="headerlink" href="#_basic_scheduler" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The number of active HA services on each node is used to choose a recovery node.
Non-HA-managed services are currently not counted.</p></div>
</div>
<div class="sect3">
<h4 id="_static_load_scheduler">15.12.2. Static-Load Scheduler
 <a class="headerlink" href="#_static_load_scheduler" title="Permalink to this heading"></a>
</h4>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Important" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAALa0lEQVRogdWZa2wc1RXHfzM7O/te
P9e1vXHSmEdjx3YeDkkaF6REKRRCEDSEFNkRjdSWSsgC2iqoRWqLQKiqIBg1NOQDiMeHtkQIQkRR
S9S4aWwgCQoUgl232KkT28J21l6vd3d2dx79sDuTXdtre03Uqlc62tl53Pv/n3PuOffcKxiGwf9z
E//XAL5sk65WR0a6AaDrOjMtKwgCoigiCAKCIAhXa9wvTcAEHhoc5KMjR7jw3nucf+MN67k/GCS4
YQOr77yTpl27kGUZm81mXC0ywlLnQDbwEwcO0P2b3yz4jV5RwZq2Nm5/5BH8fj+SJGGz2b4UiSUR
MAzD0DSNEwcO8MdHHln8d0AcsAUCfOe551h7yy04HA7sdvuSrVEwAV3XjdDgIC/v2cOl06dznqWA
JCAAMvn9MwlEgF1PP8037rsPt9uN3W5fkjUKIqDrujF+4QLP3XQTU0NDOc8UIAa4y8sJtrQQqK/H
7XQSfv99Pn/nHQRRxNB16/0EEAUatm9n76uv4vP5cDgcBZNYNIF84I0MEK20lIZvf5sNe/dSWlqK
1+vFxBEZHubT3/+evqNHmR4dJRWP55BevW0bra+8gt/vx+l0IknSokksikA+8ElgClixYwc3799P
ZWUlTqczbz+x0VFOPvEEQ6dPE/niCwxdt0jUbd1K68svU1xcXBCJBQnMBz5eWkrL/v2s++Y3KSsr
QxQXlxc7f/5zBjo7CQ8NoadSKKQn943t7dyyf79FYjHuNC8BwzAMVVV56Z57+OzNN3PAJ8vK+FZH
B2s2b55X6/nahRMnOP7TnxLOKEXJyO2/+hVf37uXoqIinE6nmfzyksirMjNUvvGjH+WAVwG1vJxb
Dhxg1bp1uFwuMwQWJCu3bWPN3r24SkoAcGbA/OXXv2bws8+IRqOkUqlZGX1RBAzDMHRd51+nTvHe
wYNX7gPTQMO+faxav56SkpIlgTdlU3s7TXv2IGUs6AGioRDH2tuZmJggFouhqirGPCzyEUBVVX7X
1pZzPwI0fP/7bNmzh7KysrzADh8+zP3338++ffs4ePBg3vdEUWTDD3/IsuZmRLsdW4bEpd5eOp99
lnA4jKIoaJqWl8QsAqb2Txw4kDNp40BxXR0bWlspLy/PC+qZZ55hdHQUSZIYHByko6ODxx57bBZw
cy3kq6xk6y9/ib+qCgQBmbQ7nT50iIs9PUSjUZLJJIZhzEliLgKMDQzwp5/9zLqnkQ51W3/xCwKB
AE6nMy+B3t5exsfH6ezspLe3l3g8zuHDh2cBz5bKpibWtrXhLi0FwA3owNsPP8zExATxeBxVVecy
QC4Bc+L+taNjluvU33031dddN6/2BUGgpKSE/v5+JicncweaA3i2tDz8MIHrr0eU0gsQBzDW10ff
yZNEIhESicScrpRDQNd1xgYG+OC3v7XuJQBXIMDWn/yEioqKeUGMjY0hCAKKoszS1MjIyLzfAmx/
4gmKli0DQcBJOmicfOopJicnicfjJoG5LWBq/+Szz+a8oABNe/bgdrvndR1BEKiqqiIejxOLxWYR
CAaDC0alqjVrqGxowFNWhkDalUKff07f3/5mWSFTLFksLAK6rjN+4UKO9lXAUV7O2tbWeaNOthZl
WZ4FHlh0aN3y4IPIHk+6L8AGvNfRQTgctuZCthVEU/u6rnPutddyBk0A9bt3W4us+QY2fbyurg6v
15vTzw033LBoAsHmZoLNzdgcDgBcQKi/n391dTE9PU0ymcyxgkmAVCrF+4cO5WjfFgiwfoGwOTOy
lJWVUVtba1mkUAKCILClvR1veTkA9owVPnzllZy8YFnA1H5/dzfTw8PWgziwcutWfD4fsiwvCNyU
xsZG3G43lZWVVl933XVXQQSq161j+ebNlhXcwBdnzxIOh60lhmkFEUDTNP7++uvWgBrp6mrNHBk3
H/BsAtFoFEdm8GAwyMaNGwsiIAgCjbt346uosKwgAp8cPWolNj1THInmsqHn6FGLQApY1thIxcqV
+Hy+WX6+0ETesWMHLpcLWZZ54IEHCgYvCAKrbrsNyem0JrQEXMzkhKzlBaKu61z+97+ZHhnJmbwr
tm2zJu5CWp8pra2tNDU1sXPnTtrb25dEQBAEvrplC57MXJCB8XPniEQixONxa6Uq6brOQHd3jvvo
QP2ttxZUpMxsL7zwwpK+AyxLXrt9Oxe6uxFEEXvGZc4fO0bVD35AMplM1wuapjGYtbugAp5AgKLq
aquu/W/JTEs37dqFqihWzSADk8PD1jzQNC3tQtmrThUINDbm+H4h0tfXx6OPPsodd9xBW1sbPT09
BQPPluu2b7fWRxIQ6usjFotZBCRN06wZbbZAfT1FRUWWKQtpzz//PMPDw/T09DA1NcXo6CjHjx+f
11Xma95AwHJjEdB0HUVRrhCYuRGrAiXBoFUqFtq6u7sZHx9nJBMUurq65uxnsX0Hm5v56A9/ANIJ
LXrpEolEwgqlUqZQyPlo6OOPlzx5ly1bxnBWQmxpackBW6hSBEC02axrpqasNZGmaek8kN0kYPDj
j1FVdUlz4Mknn6Surs4Cb5aUhYZiUy59+GGOgm3V1WiadiUPCILAV9autV5wAaM9PXQeOWJprBBp
aGjgxIkTpFIpOjs7aWxsXBJwUwZOnUKZmgLS2zl4vUjSlV1XURRFqtavv3IDKAVe/O53efPxx69a
SCxUBs+c4dV77+Vyfz+xy5eBdG3iX7ECm81m9S2JokiwuZmiujrCPT1AuqgOAqcef5zzL77Imt27
uemhhyipqVnYZ5cw8bPbR0eOcOall7g8MEB4aIhEJAJk9qOAwKZNyLJsHpQgRKNRIxQK0XPmDH/+
3vdQQ6G8na/YtIk199zDypYWVmzceFWAT1y8yMWzZzn/9tv0vfsudqeTlKIQHRvL2QSOAqU7d/K1
W2/l2muvpba2Nl3iJhIJIxKJMDw8TO/Zs3zw4x+jTUzkHVCw2fCUleHw+QiuXYu/upqqhgb8VVUY
hkFNczPFweCs7z556y3r+vyxYwiiyD+PHwdBwNB1krEYhqYRC4UwdB2D9LJmmnRtXHz77VTfeCPL
ly+ntraWmpqa9MaaqqqGoiiEQiGGhoYYHBzk00OHiHR2LkqDNlnG7nJhd7nQswqN+ZogiuipFMlY
DDWRgBl5KEF6woqAUFFByc03U756NZWVlSxfvpyamhoCgQAejwdB13VDVVVisRihUIiRkRGGh4cZ
+sc/GO3qIt7VhZGJAle7mVpWuVKDCBng8jXX4N68Gc8111BUVER5eTnV1dVUV1cTCASuFFqGYaDr
upFKpYjH40xMTDA+Ps7Y2Bjj4+OEw2EmenqYPneO1OAgZCWpxTY9S9QMcLhyHGUDRI8HqbISZ309
zoYGnE4nHo8Hv99PaWkpgUCAiooK6/DE4XCkI1HW2a6hqiqKojA9Pc3k5KQl4XCY6elpotFoetuk
v59UKIR66RLToRBJTcMYHUWIRvMSMPO6rbY2/SsIOFatQpYk5NWrkSQJWZZxOBy43W68Xi9+v5+i
oiJKSkooLi7G5/PhdruRZdnads85HzD3hpLJJIqiEIvFiEajRCIRIpEI0WiUWCxGPB631iOKopBK
pazUbi4Oc7Y+snKCzWbDZrMhSRKSJGG3262w6HK5cLlceDwevF4vPp8Pr9eL2+3G5XLhcDjMkxuE
TNibdcBhnv9qmkYqlSKZTJJIJFAUhXg8jqIoKIpCIpGwSCSTSVRVtWSuk3qTiAnebrfngHc4HDid
TpxOJy6Xy7qWZdk6wZzrxCbvCY1JRNd1NE3LAWgSM69ngp+PgCiK1gF3NhFTzHvmO+Y3wCzw8xLI
JpL5tYCZpEyw2f/nWt1ag2UtM0x3Mq1iAp1j+bH0M7KFCGWDnfmbd8AMnuxfUzL/C0rp/wFnFd4n
EQn3XQAAAABJRU5ErkJggg==">
</td>
<td class="content">The static mode is still a technology preview.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>Static usage information from HA services on each node is used to choose a
recovery node. Usage of non-HA-managed services is currently not considered.</p></div>
<div class="paragraph">
<p>For this selection, each node in turn is considered as if the service was
already running on it, using CPU and memory usage from the associated guest
configuration. Then for each such alternative, CPU and memory usage of all nodes
are considered, with memory being weighted much more, because it’s a truly
limited resource. For both, CPU and memory, highest usage among nodes (weighted
more, as ideally no node should be overcommitted) and average usage of all nodes
(to still be able to distinguish in case there already is a more highly
committed node) are considered.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Important" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAALa0lEQVRogdWZa2wc1RXHfzM7O/te
P9e1vXHSmEdjx3YeDkkaF6REKRRCEDSEFNkRjdSWSsgC2iqoRWqLQKiqIBg1NOQDiMeHtkQIQkRR
S9S4aWwgCQoUgl232KkT28J21l6vd3d2dx79sDuTXdtre03Uqlc62tl53Pv/n3PuOffcKxiGwf9z
E//XAL5sk65WR0a6AaDrOjMtKwgCoigiCAKCIAhXa9wvTcAEHhoc5KMjR7jw3nucf+MN67k/GCS4
YQOr77yTpl27kGUZm81mXC0ywlLnQDbwEwcO0P2b3yz4jV5RwZq2Nm5/5BH8fj+SJGGz2b4UiSUR
MAzD0DSNEwcO8MdHHln8d0AcsAUCfOe551h7yy04HA7sdvuSrVEwAV3XjdDgIC/v2cOl06dznqWA
JCAAMvn9MwlEgF1PP8037rsPt9uN3W5fkjUKIqDrujF+4QLP3XQTU0NDOc8UIAa4y8sJtrQQqK/H
7XQSfv99Pn/nHQRRxNB16/0EEAUatm9n76uv4vP5cDgcBZNYNIF84I0MEK20lIZvf5sNe/dSWlqK
1+vFxBEZHubT3/+evqNHmR4dJRWP55BevW0bra+8gt/vx+l0IknSokksikA+8ElgClixYwc3799P
ZWUlTqczbz+x0VFOPvEEQ6dPE/niCwxdt0jUbd1K68svU1xcXBCJBQnMBz5eWkrL/v2s++Y3KSsr
QxQXlxc7f/5zBjo7CQ8NoadSKKQn943t7dyyf79FYjHuNC8BwzAMVVV56Z57+OzNN3PAJ8vK+FZH
B2s2b55X6/nahRMnOP7TnxLOKEXJyO2/+hVf37uXoqIinE6nmfzyksirMjNUvvGjH+WAVwG1vJxb
Dhxg1bp1uFwuMwQWJCu3bWPN3r24SkoAcGbA/OXXv2bws8+IRqOkUqlZGX1RBAzDMHRd51+nTvHe
wYNX7gPTQMO+faxav56SkpIlgTdlU3s7TXv2IGUs6AGioRDH2tuZmJggFouhqirGPCzyEUBVVX7X
1pZzPwI0fP/7bNmzh7KysrzADh8+zP3338++ffs4ePBg3vdEUWTDD3/IsuZmRLsdW4bEpd5eOp99
lnA4jKIoaJqWl8QsAqb2Txw4kDNp40BxXR0bWlspLy/PC+qZZ55hdHQUSZIYHByko6ODxx57bBZw
cy3kq6xk6y9/ib+qCgQBmbQ7nT50iIs9PUSjUZLJJIZhzEliLgKMDQzwp5/9zLqnkQ51W3/xCwKB
AE6nMy+B3t5exsfH6ezspLe3l3g8zuHDh2cBz5bKpibWtrXhLi0FwA3owNsPP8zExATxeBxVVecy
QC4Bc+L+taNjluvU33031dddN6/2BUGgpKSE/v5+JicncweaA3i2tDz8MIHrr0eU0gsQBzDW10ff
yZNEIhESicScrpRDQNd1xgYG+OC3v7XuJQBXIMDWn/yEioqKeUGMjY0hCAKKoszS1MjIyLzfAmx/
4gmKli0DQcBJOmicfOopJicnicfjJoG5LWBq/+Szz+a8oABNe/bgdrvndR1BEKiqqiIejxOLxWYR
CAaDC0alqjVrqGxowFNWhkDalUKff07f3/5mWSFTLFksLAK6rjN+4UKO9lXAUV7O2tbWeaNOthZl
WZ4FHlh0aN3y4IPIHk+6L8AGvNfRQTgctuZCthVEU/u6rnPutddyBk0A9bt3W4us+QY2fbyurg6v
15vTzw033LBoAsHmZoLNzdgcDgBcQKi/n391dTE9PU0ymcyxgkmAVCrF+4cO5WjfFgiwfoGwOTOy
lJWVUVtba1mkUAKCILClvR1veTkA9owVPnzllZy8YFnA1H5/dzfTw8PWgziwcutWfD4fsiwvCNyU
xsZG3G43lZWVVl933XVXQQSq161j+ebNlhXcwBdnzxIOh60lhmkFEUDTNP7++uvWgBrp6mrNHBk3
H/BsAtFoFEdm8GAwyMaNGwsiIAgCjbt346uosKwgAp8cPWolNj1THInmsqHn6FGLQApY1thIxcqV
+Hy+WX6+0ETesWMHLpcLWZZ54IEHCgYvCAKrbrsNyem0JrQEXMzkhKzlBaKu61z+97+ZHhnJmbwr
tm2zJu5CWp8pra2tNDU1sXPnTtrb25dEQBAEvrplC57MXJCB8XPniEQixONxa6Uq6brOQHd3jvvo
QP2ttxZUpMxsL7zwwpK+AyxLXrt9Oxe6uxFEEXvGZc4fO0bVD35AMplM1wuapjGYtbugAp5AgKLq
aquu/W/JTEs37dqFqihWzSADk8PD1jzQNC3tQtmrThUINDbm+H4h0tfXx6OPPsodd9xBW1sbPT09
BQPPluu2b7fWRxIQ6usjFotZBCRN06wZbbZAfT1FRUWWKQtpzz//PMPDw/T09DA1NcXo6CjHjx+f
11Xma95AwHJjEdB0HUVRrhCYuRGrAiXBoFUqFtq6u7sZHx9nJBMUurq65uxnsX0Hm5v56A9/ANIJ
LXrpEolEwgqlUqZQyPlo6OOPlzx5ly1bxnBWQmxpackBW6hSBEC02axrpqasNZGmaek8kN0kYPDj
j1FVdUlz4Mknn6Surs4Cb5aUhYZiUy59+GGOgm3V1WiadiUPCILAV9autV5wAaM9PXQeOWJprBBp
aGjgxIkTpFIpOjs7aWxsXBJwUwZOnUKZmgLS2zl4vUjSlV1XURRFqtavv3IDKAVe/O53efPxx69a
SCxUBs+c4dV77+Vyfz+xy5eBdG3iX7ECm81m9S2JokiwuZmiujrCPT1AuqgOAqcef5zzL77Imt27
uemhhyipqVnYZ5cw8bPbR0eOcOall7g8MEB4aIhEJAJk9qOAwKZNyLJsHpQgRKNRIxQK0XPmDH/+
3vdQQ6G8na/YtIk199zDypYWVmzceFWAT1y8yMWzZzn/9tv0vfsudqeTlKIQHRvL2QSOAqU7d/K1
W2/l2muvpba2Nl3iJhIJIxKJMDw8TO/Zs3zw4x+jTUzkHVCw2fCUleHw+QiuXYu/upqqhgb8VVUY
hkFNczPFweCs7z556y3r+vyxYwiiyD+PHwdBwNB1krEYhqYRC4UwdB2D9LJmmnRtXHz77VTfeCPL
ly+ntraWmpqa9MaaqqqGoiiEQiGGhoYYHBzk00OHiHR2LkqDNlnG7nJhd7nQswqN+ZogiuipFMlY
DDWRgBl5KEF6woqAUFFByc03U756NZWVlSxfvpyamhoCgQAejwdB13VDVVVisRihUIiRkRGGh4cZ
+sc/GO3qIt7VhZGJAle7mVpWuVKDCBng8jXX4N68Gc8111BUVER5eTnV1dVUV1cTCASuFFqGYaDr
upFKpYjH40xMTDA+Ps7Y2Bjj4+OEw2EmenqYPneO1OAgZCWpxTY9S9QMcLhyHGUDRI8HqbISZ309
zoYGnE4nHo8Hv99PaWkpgUCAiooK6/DE4XCkI1HW2a6hqiqKojA9Pc3k5KQl4XCY6elpotFoetuk
v59UKIR66RLToRBJTcMYHUWIRvMSMPO6rbY2/SsIOFatQpYk5NWrkSQJWZZxOBy43W68Xi9+v5+i
oiJKSkooLi7G5/PhdruRZdnads85HzD3hpLJJIqiEIvFiEajRCIRIpEI0WiUWCxGPB631iOKopBK
pazUbi4Oc7Y+snKCzWbDZrMhSRKSJGG3262w6HK5cLlceDwevF4vPp8Pr9eL2+3G5XLhcDjMkxuE
TNibdcBhnv9qmkYqlSKZTJJIJFAUhXg8jqIoKIpCIpGwSCSTSVRVtWSuk3qTiAnebrfngHc4HDid
TpxOJy6Xy7qWZdk6wZzrxCbvCY1JRNd1NE3LAWgSM69ngp+PgCiK1gF3NhFTzHvmO+Y3wCzw8xLI
JpL5tYCZpEyw2f/nWt1ag2UtM0x3Mq1iAp1j+bH0M7KFCGWDnfmbd8AMnuxfUzL/C0rp/wFnFd4n
EQn3XQAAAABJRU5ErkJggg==">
</td>
<td class="content">The more services the more possible combinations there are, so it’s
currently not recommended to use it if you have thousands of HA managed
services.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="_crs_scheduling_points">15.12.3. CRS Scheduling Points
 <a class="headerlink" href="#_crs_scheduling_points" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The CRS algorithm is not applied for every service in every round, since this
would mean a large number of constant migrations. Depending on the workload,
this could put more strain on the cluster than could be avoided by constant
balancing.
That’s why the Proxmox VE HA manager favors keeping services on their current node.</p></div>
<div class="paragraph">
<p>The CRS is currently used at the following scheduling points:</p></div>
<div class="ulist"><ul>
<li>
<p>
Service recovery (always active). When a node with active HA services fails,
  all its services need to be recovered to other nodes. The CRS algorithm will
  be used here to balance that recovery over the remaining nodes.
</p>
</li>
<li>
<p>
HA group config changes (always active). If a node is removed from a group,
  or its priority is reduced, the HA stack will use the CRS algorithm to find a
  new target node for the HA services in that group, matching the adapted
  priority constraints.
</p>
</li>
<li>
<p>
HA service stopped → start transtion (opt-in). Requesting that a stopped
  service should be started is an good opportunity to check for the best suited
  node as per the CRS algorithm, as moving stopped services is  cheaper to do
  than moving them started, especially if their disk volumes reside on shared
  storage. You can enable this by setting the <strong><span class="monospaced">ha-rebalance-on-start</span></strong>
  CRS option in the datacenter config. You can change that option also in the
  Web UI, under <span class="monospaced">Datacenter</span> → <span class="monospaced">Options</span> → <span class="monospaced">Cluster Resource Scheduling</span>.
</p>
</li>
</ul></div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="chapter_vzdump">
16. Backup and Restore
 <a class="headerlink" href="#chapter_vzdump" title="Permalink to this heading"></a>
</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Backups are a requirement for any sensible IT deployment, and Proxmox VE
provides a fully integrated solution, using the capabilities of each
storage and each guest system type. This allows the system
administrator to fine tune via the <span class="monospaced">mode</span> option between consistency
of the backups and downtime of the guest system.</p></div>
<div class="paragraph">
<p>Proxmox VE backups are always full backups - containing the VM/CT
configuration and all data.  Backups can be started via the GUI or via
the <span class="monospaced">vzdump</span> command-line tool.</p></div>
<div class="paragraph">
<div class="title">Backup Storage</div><p>Before a backup can run, a backup storage must be defined. Refer to the
<a href="#chapter_storage">storage documentation</a> on how to add a storage. It can
either be a Proxmox Backup Server storage, where backups are stored as
de-duplicated chunks and metadata, or a file-level storage, where backups are
stored as regular files. Using Proxmox Backup Server on a dedicated host is
recommended, because of its advanced features. Using an NFS server is a good
alternative. In both cases, you might want to save those backups later to a tape
drive, for off-site archiving.</p></div>
<div class="paragraph">
<div class="title">Scheduled Backup</div><p>Backup jobs can be scheduled so that they are executed automatically on specific
days and times, for selectable nodes and guest systems. See the
<a href="#vzdump_jobs">Backup Jobs</a> section for more.</p></div>
<div class="sect2">
<h3 id="_backup_modes">
<span>16.1. Backup Modes</span>
 <a class="headerlink" href="#_backup_modes" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>There are several ways to provide consistency (option <span class="monospaced">mode</span>),
depending on the guest type.</p></div>
<div class="dlist"><div class="title">Backup modes for VMs:</div><dl>
<dt class="hdlist1">
<span class="monospaced">stop</span> mode
</dt>
<dd>
<p>
This mode provides the highest consistency of the backup, at the cost
of a short downtime in the VM operation. It works by executing an
orderly shutdown of the VM, and then runs a background QEMU process to
backup the VM data. After the backup is started, the VM goes to full
operation mode if it was previously running. Consistency is guaranteed
by using the live backup feature.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">suspend</span> mode
</dt>
<dd>
<p>
This mode is provided for compatibility reason, and suspends the VM
before calling the <span class="monospaced">snapshot</span> mode. Since suspending the VM results in
a longer downtime and does not necessarily improve the data
consistency, the use of the <span class="monospaced">snapshot</span> mode is recommended instead.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">snapshot</span> mode
</dt>
<dd>
<p>
This mode provides the lowest operation downtime, at the cost of a
small inconsistency risk. It works by performing a Proxmox VE live
backup, in which data blocks are copied while the VM is running. If the
guest agent is enabled (<span class="monospaced">agent: 1</span>) and running, it calls
<span class="monospaced">guest-fsfreeze-freeze</span> and <span class="monospaced">guest-fsfreeze-thaw</span> to improve
consistency.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p>A technical overview of the Proxmox VE live backup for QemuServer can
be found online
<a href="https://git.proxmox.com/?p=pve-qemu.git;a=blob_plain;f=backup.txt">here</a>.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Proxmox VE live backup provides snapshot-like semantics on any
storage type. It does not require that the underlying storage supports
snapshots. Also please note that since the backups are done via
a background QEMU process, a stopped VM will appear as running for a
short amount of time while the VM disks are being read by QEMU.
However the VM itself is not booted, only its disk(s) are read.</td>
</tr></tbody></table>
</div>
<div class="dlist"><div class="title">Backup modes for Containers:</div><dl>
<dt class="hdlist1">
<span class="monospaced">stop</span> mode
</dt>
<dd>
<p>
Stop the container for the duration of the backup. This potentially
results in a very long downtime.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">suspend</span> mode
</dt>
<dd>
<p>
This mode uses rsync to copy the container data to a temporary
location (see option <span class="monospaced">--tmpdir</span>). Then the container is suspended and
a second rsync copies changed files. After that, the container is
started (resumed) again. This results in minimal downtime, but needs
additional space to hold the container copy.
</p>
<div class="paragraph">
<p>When the container is on a local file system and the target storage of
the backup is an NFS/CIFS server, you should set <span class="monospaced">--tmpdir</span> to reside on a
local file system too, as this will result in a many fold performance
improvement.  Use of a local <span class="monospaced">tmpdir</span> is also required if you want to
backup a local container using ACLs in suspend mode if the backup
storage is an NFS server.</p></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">snapshot</span> mode
</dt>
<dd>
<p>
This mode uses the snapshotting facilities of the underlying
storage. First, the container will be suspended to ensure data consistency.
A temporary snapshot of the container’s volumes will be made and the
snapshot content will be archived in a tar file. Finally, the temporary
snapshot is deleted again.
</p>
</dd>
</dl></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content"><span class="monospaced">snapshot</span> mode requires that all backed up volumes are on a storage that
supports snapshots. Using the <span class="monospaced">backup=no</span> mount point option individual volumes
can be excluded from the backup (and thus this requirement).</td>
</tr></tbody></table>
</div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">By default additional mount points besides the Root Disk mount point are
not included in backups. For volume mount points you can set the <strong>Backup</strong> option
to include the mount point in the backup. Device and bind mounts are never
backed up as their content is managed outside the Proxmox VE storage library.</td>
</tr></tbody></table>
</div>
<div class="sect3">
<h4 id="_vm_backup_fleecing">16.1.1. VM Backup Fleecing
 <a class="headerlink" href="#_vm_backup_fleecing" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>When a backup for a VM is started, QEMU will install a "copy-before-write"
filter in its block layer. This filter ensures that upon new guest writes, old
data still needed for the backup is sent to the backup target first. The guest
write blocks until this operation is finished so guest IO to not-yet-backed-up
sectors will be limited by the speed of the backup target.</p></div>
<div class="paragraph">
<p>With backup fleecing, such old data is cached in a fleecing image rather than
sent directly to the backup target. This can help guest IO performance and even
prevent hangs in certain scenarios, at the cost of requiring more storage space.</p></div>
<div class="paragraph">
<p>To manually start a backup of VM <span class="monospaced">123</span> with fleecing images created on the
storage <span class="monospaced">local-lvm</span>, run</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>vzdump 123 --fleecing enabled=1,storage=local-lvm</pre>
</div></div>
<div class="paragraph">
<p>As always, you can set the option for specific backup jobs, or as a node-wide
fallback via the <a href="#vzdump_configuration">configuration options</a>. In the UI,
fleecing can be configured in the <em>Advanced</em> tab when editing a backup job.</p></div>
<div class="paragraph">
<p>The fleecing storage should be a fast local storage, with thin provisioning and
discard support. Examples are LVM-thin, RBD, ZFS with <span class="monospaced">sparse 1</span> in the storage
configuration, many file-based storages. Ideally, the fleecing storage is a
dedicated storage, so it running full will not affect other guests and just fail
the backup. Parts of the fleecing image that have been backed up will be
discarded to try and keep the space usage low.</p></div>
<div class="paragraph">
<p>For file-based storages that do not support discard (for example, NFS before
version 4.2), you should set <span class="monospaced">preallocation off</span> in the storage configuration.
In combination with <span class="monospaced">qcow2</span> (used automatically as the format for the fleecing
image when the storage supports it), this has the advantage that already
allocated parts of the image can be re-used later, which can still help save
quite a bit of space.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">On a storage that’s not thinly provisioned, for exampple, LVM or ZFS
without the <span class="monospaced">sparse</span> option, the full size of the original disk needs to be
reserved for the fleecing image up-front. On a thinly provisioned storage, the
fleecing image can grow to the same size as the original image only if the guest
re-writes a whole disk while the backup is busy with another disk.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="_ct_change_detection_mode">16.1.2. CT Change Detection Mode
 <a class="headerlink" href="#_ct_change_detection_mode" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Setting the change detection mode defines the encoding format for the pxar
archives and how changed, and unchanged files are handled for container backups
with Proxmox Backup Server as the target.</p></div>
<div class="paragraph">
<p>The change detection mode option can be configured for individual backup jobs in
the <em>Advanced</em> tab while editing a job. Further, this option can be set as
node-wide fallback via the <a href="#vzdump_configuration">configuration options</a>.</p></div>
<div class="paragraph">
<p>There are 3 change detection modes available:</p></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:20%;">
<col style="width:80%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top"> Mode     </th>
<th class="tableblock halign-left valign-top"> Description</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">Default</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Read and encode all files into a single archive, using the pxar
format version 1.</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">Data</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">(EXPERIMENTAL): Read and encode all files, but split data and
metadata into separate streams, using the pxar format version 2.</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock monospaced">Metadata</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">(EXPERIMENTAL): Split streams and use archive format version 2 like
<span class="monospaced">Data</span>, but use the metadata archive of the previous snapshot (if one exists) to
detect unchanged files, and reuse their data chunks without reading file
contents from disk, whenever possible.</p></td>
</tr>
</tbody>
</table>
<div class="paragraph">
<p>To perform a backup using the change detecation mode <span class="monospaced">metadata</span> you can run</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>vzdump 123 --storage pbs-storage --pbs-change-detection-mode metadata</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Backups of VMs or to storage backends other than Proxmox Backup Server are
not affected by this setting.</td>
</tr></tbody></table>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_backup_file_names">
<span>16.2. Backup File Names</span>
 <a class="headerlink" href="#_backup_file_names" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Newer versions of vzdump encode the guest type and the
backup time into the filename, for example</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>vzdump-lxc-105-2009_10_09-11_04_43.tar</pre>
</div></div>
<div class="paragraph">
<p>That way it is possible to store several backup in the same directory. You can
limit the number of backups that are kept with various retention options, see
the <a href="#vzdump_retention">Backup Retention</a> section below.</p></div>
</div>
<div class="sect2">
<h3 id="_backup_file_compression">
<span>16.3. Backup File Compression</span>
 <a class="headerlink" href="#_backup_file_compression" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>The backup file can be compressed with one of the following algorithms: <span class="monospaced">lzo</span>
<span class="footnote" data-note="Lempel–Ziv–Oberhumer a lossless data compression algorithm
<a href=&quot;https://en.wikipedia.org/wiki/Lempel-Ziv-Oberhumer&quot;>https://en.wikipedia.org/wiki/Lempel-Ziv-Oberhumer</a>">[<a id="_footnoteref_54" href="#_footnote_54" title="View footnote" class="footnote">54</a>]</span>, <span class="monospaced">gzip</span> <span class="footnote" data-note="gzip -
based on the DEFLATE algorithm <a href=&quot;https://en.wikipedia.org/wiki/Gzip&quot;>https://en.wikipedia.org/wiki/Gzip</a>">[<a id="_footnoteref_55" href="#_footnote_55" title="View footnote" class="footnote">55</a>]</span> or <span class="monospaced">zstd</span>
<span class="footnote" data-note="Zstandard a lossless data compression algorithm
<a href=&quot;https://en.wikipedia.org/wiki/Zstandard&quot;>https://en.wikipedia.org/wiki/Zstandard</a>">[<a id="_footnoteref_56" href="#_footnote_56" title="View footnote" class="footnote">56</a>]</span>.</p></div>
<div class="paragraph">
<p>Currently, Zstandard (zstd) is the fastest of these three algorithms.
Multi-threading is another advantage of zstd over lzo and gzip. Lzo and gzip
are more widely used and often installed by default.</p></div>
<div class="paragraph">
<p>You can install pigz <span class="footnote" data-note="pigz - parallel implementation of gzip
<a href=&quot;https://zlib.net/pigz/&quot;>https://zlib.net/pigz/</a>">[<a id="_footnoteref_57" href="#_footnote_57" title="View footnote" class="footnote">57</a>]</span> as a drop-in replacement for gzip to provide better
performance due to multi-threading. For pigz &amp; zstd, the amount of
threads/cores can be adjusted. See the
<a href="#vzdump_configuration">configuration options</a> below.</p></div>
<div class="paragraph">
<p>The extension of the backup file name can usually be used to determine which
compression algorithm has been used to create the backup.</p></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:50%;">
<col style="width:50%;">
</colgroup><tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">.zst</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Zstandard (zstd) compression</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">.gz or .tgz</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">gzip compression</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">.lzo</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">lzo compression</p></td>
</tr>
</tbody>
</table>
<div class="paragraph">
<p>If the backup file name doesn’t end with one of the above file extensions, then
it was not compressed by vzdump.</p></div>
</div>
<div class="sect2">
<h3 id="_backup_encryption">
<span>16.4. Backup Encryption</span>
 <a class="headerlink" href="#_backup_encryption" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>For Proxmox Backup Server storages, you can optionally set up client-side
encryption of backups, see <a href="#storage_pbs_encryption">the corresponding section.</a></p></div>
</div>
<div class="sect2">
<h3 id="vzdump_jobs">
<span>16.5. Backup Jobs</span>
 <a class="headerlink" href="#vzdump_jobs" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-cluster-backup-overview.png">
<img src="images/screenshot/gui-cluster-backup-overview.png" alt="screenshot/gui-cluster-backup-overview.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>Besides triggering a backup manually, you can also setup periodic jobs that
backup all, or a selection of virtual guest to a storage. You can manage the
jobs in the UI under <em>Datacenter</em> → <em>Backup</em> or via the <span class="monospaced">/cluster/backup</span> API
endpoint. Both will generate job entries in <span class="monospaced">/etc/pve/jobs.cfg</span>, which are
parsed and executed by the <span class="monospaced">pvescheduler</span> daemon.</p></div>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-cluster-backup-edit-01-general.png">
<img src="images/screenshot/gui-cluster-backup-edit-01-general.png" alt="screenshot/gui-cluster-backup-edit-01-general.png" width="250" style="padding: 0 10px 0 0;float:left;"></a>
<p>A job is either configured for all cluster nodes or a specific node, and is
executed according to a given schedule. The format for the schedule is very
similar to <span class="monospaced">systemd</span> calendar events, see the
<a href="#chapter_calendar_events">calendar events</a> section for details. The
<em>Schedule</em> field in the UI can be freely edited, and it contains several
examples that can be used as a starting point in its drop-down list.</p></div>
<div class="paragraph">
<p>You can configure job-specific <a href="#vzdump_retention">retention options</a>
overriding those from the storage or node configuration, as well as a
<a href="#vzdump_notes">template for notes</a> for additional information to be saved
together with the backup.</p></div>
<div class="paragraph">
<p>Since scheduled backups miss their execution when the host was offline or the
pvescheduler was disabled during the scheduled time, it is possible to configure
the behaviour for catching up. By enabling the <span class="monospaced">Repeat missed</span> option (in the
<em>Advanced</em> tab in the UI, <span class="monospaced">repeat-missed</span> in the config), you can tell the
scheduler that it should run missed jobs as soon as possible.</p></div>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-cluster-backup-edit-04-advanced.png">
<img src="images/screenshot/gui-cluster-backup-edit-04-advanced.png" alt="screenshot/gui-cluster-backup-edit-04-advanced.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>There are a few settings for tuning backup performance (some of which are
exposed in the <em>Advanced</em> tab in the UI). The most notable is <span class="monospaced">bwlimit</span> for
limiting IO bandwidth. The amount of threads used for the compressor can be
controlled with the <span class="monospaced">pigz</span> (replacing <span class="monospaced">gzip</span>), respectively, <span class="monospaced">zstd</span> setting.
Furthermore, there are <span class="monospaced">ionice</span> (when the BFQ scheduler is used) and, as part of
the <span class="monospaced">performance</span> setting, <span class="monospaced">max-workers</span> (affects VM backups only) and
<span class="monospaced">pbs-entries-max</span> (affects container backups only). See the
<a href="#vzdump_configuration">configuration options</a> for details.</p></div>
</div>
<div class="sect2">
<h3 id="vzdump_retention">
<span>16.6. Backup Retention</span>
 <a class="headerlink" href="#vzdump_retention" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>With the <span class="monospaced">prune-backups</span> option you can specify which backups you want to keep
in a flexible manner.</p></div>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-cluster-backup-edit-02-retention.png">
<img src="images/screenshot/gui-cluster-backup-edit-02-retention.png" alt="screenshot/gui-cluster-backup-edit-02-retention.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>The following retention options are available:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">keep-all &lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Keep all backups. If this is <span class="monospaced">true</span>, no other options can be set.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">keep-last &lt;N&gt;</span> 
</dt>
<dd>
<p>
Keep the last <span class="monospaced">&lt;N&gt;</span> backups.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">keep-hourly &lt;N&gt;</span> 
</dt>
<dd>
<p>
Keep backups for the last <span class="monospaced">&lt;N&gt;</span> hours. If there is more than one
backup for a single hour, only the latest is kept.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">keep-daily &lt;N&gt;</span> 
</dt>
<dd>
<p>
Keep backups for the last <span class="monospaced">&lt;N&gt;</span> days. If there is more than one
backup for a single day, only the latest is kept.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">keep-weekly &lt;N&gt;</span> 
</dt>
<dd>
<p>
Keep backups for the last <span class="monospaced">&lt;N&gt;</span> weeks. If there is more than one
backup for a single week, only the latest is kept.
</p>
</dd>
</dl></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Weeks start on Monday and end on Sunday. The software uses the
<span class="monospaced">ISO week date</span>-system and handles weeks at the end of the year correctly.</td>
</tr></tbody></table>
</div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">keep-monthly &lt;N&gt;</span> 
</dt>
<dd>
<p>
Keep backups for the last <span class="monospaced">&lt;N&gt;</span> months. If there is more than one
backup for a single month, only the latest is kept.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">keep-yearly &lt;N&gt;</span> 
</dt>
<dd>
<p>
Keep backups for the last <span class="monospaced">&lt;N&gt;</span> years. If there is more than one
backup for a single year, only the latest is kept.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p>The retention options are processed in the order given above. Each option
only covers backups within its time period. The next option does not take care
of already covered backups. It will only consider older backups.</p></div>
<div class="paragraph">
<p>Specify the retention options you want to use as a
comma-separated list, for example:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># vzdump 777 --prune-backups keep-last=3,keep-daily=13,keep-yearly=9</pre>
</div></div>
<div class="paragraph">
<p>While you can pass <span class="monospaced">prune-backups</span> directly to <span class="monospaced">vzdump</span>, it is often more
sensible to configure the setting on the storage level, which can be done via
the web interface.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">The old <span class="monospaced">maxfiles</span> option is deprecated and should be replaced either by
<span class="monospaced">keep-last</span> or, in case <span class="monospaced">maxfiles</span> was <span class="monospaced">0</span> for unlimited retention, by
<span class="monospaced">keep-all</span>.</td>
</tr></tbody></table>
</div>
<div class="sect3">
<h4 id="_prune_simulator">16.6.1. Prune Simulator
 <a class="headerlink" href="#_prune_simulator" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>You can use the  <a href="https://pbs.proxmox.com/docs/prune-simulator">prune simulator
of the Proxmox Backup Server documentation</a> to explore the effect of different
retention options with various backup schedules.</p></div>
</div>
<div class="sect3">
<h4 id="_retention_settings_example">16.6.2. Retention Settings Example
 <a class="headerlink" href="#_retention_settings_example" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The backup frequency and retention of old backups may depend on how often data
changes, and how important an older state may be, in a specific work load.
When backups act as a company’s document archive, there may also be legal
requirements for how long backups must be kept.</p></div>
<div class="paragraph">
<p>For this example, we assume that you are doing daily backups, have a retention
period of 10 years, and the period between backups stored gradually grows.</p></div>
<div class="paragraph">
<p><span class="monospaced">keep-last=3</span> - even if only daily backups are taken, an admin may want to
  create an extra one just before or after a big upgrade. Setting keep-last
  ensures this.</p></div>
<div class="paragraph">
<p><span class="monospaced">keep-hourly</span> is not set - for daily backups this is not relevant. You cover
  extra manual backups already, with keep-last.</p></div>
<div class="paragraph">
<p><span class="monospaced">keep-daily=13</span> - together with keep-last, which covers at least one
  day, this ensures that you have at least two weeks of backups.</p></div>
<div class="paragraph">
<p><span class="monospaced">keep-weekly=8</span> - ensures that you have at least two full months of
  weekly backups.</p></div>
<div class="paragraph">
<p><span class="monospaced">keep-monthly=11</span> - together with the previous keep settings, this
  ensures that you have at least a year of monthly backups.</p></div>
<div class="paragraph">
<p><span class="monospaced">keep-yearly=9</span> - this is for the long term archive. As you covered the
  current year with the previous options, you would set this to nine for the
  remaining ones, giving you a total of at least 10 years of coverage.</p></div>
<div class="paragraph">
<p>We recommend that you use a higher retention period than is minimally required
by your environment; you can always reduce it if you find it is unnecessarily
high, but you cannot recreate backups once they have been removed.</p></div>
</div>
</div>
<div class="sect2">
<h3 id="vzdump_protection">
<span>16.7. Backup Protection</span>
 <a class="headerlink" href="#vzdump_protection" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>You can mark a backup as <span class="monospaced">protected</span> to prevent its removal. Attempting to
remove a protected backup via Proxmox VE’s UI, CLI or API will fail. However, this
is enforced by Proxmox VE and not the file-system, that means that a manual removal
of a backup file itself is still possible for anyone with write access to the
underlying backup storage.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Protected backups are ignored by pruning and do not count towards the
retention settings.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>For filesystem-based storages, the protection is implemented via a sentinel file
<span class="monospaced">&lt;backup-name&gt;.protected</span>. For Proxmox Backup Server, it is handled on the
server side (available since Proxmox Backup Server version 2.1).</p></div>
<div class="paragraph">
<p>Use the storage option <span class="monospaced">max-protected-backups</span> to control how many protected
backups per guest are allowed on the storage. Use <span class="monospaced">-1</span> for unlimited. The
default is unlimited for users with <span class="monospaced">Datastore.Allocate</span> privilege and <span class="monospaced">5</span> for
other users.</p></div>
</div>
<div class="sect2">
<h3 id="vzdump_notes">
<span>16.8. Backup Notes</span>
 <a class="headerlink" href="#vzdump_notes" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>You can add notes to backups using the <em>Edit Notes</em> button in the UI or via the
storage content API.</p></div>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-cluster-backup-edit-03-template.png">
<img src="images/screenshot/gui-cluster-backup-edit-03-template.png" alt="screenshot/gui-cluster-backup-edit-03-template.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>It is also possible to specify a template for generating notes dynamically for
a backup job and for manual backup. The template string can contain variables,
surrounded by two curly braces, which will be replaced by the corresponding
value when the backup is executed.</p></div>
<div class="paragraph">
<p>Currently supported are:</p></div>
<div class="ulist"><ul>
<li>
<p>
<span class="monospaced">{{cluster}}</span> the cluster name, if any
</p>
</li>
<li>
<p>
<span class="monospaced">{{guestname}}</span> the virtual guest’s assigned name
</p>
</li>
<li>
<p>
<span class="monospaced">{{node}}</span> the host name of the node the backup is being created
</p>
</li>
<li>
<p>
<span class="monospaced">{{vmid}}</span> the numerical VMID of the guest
</p>
</li>
</ul></div>
<div class="paragraph">
<p>When specified via API or CLI, it needs to be a single line, where newline and
backslash need to be escaped as literal <span class="monospaced">\n</span> and <span class="monospaced">\\</span> respectively.</p></div>
</div>
<div class="sect2">
<h3 id="vzdump_restore">
<span>16.9. Restore</span>
 <a class="headerlink" href="#vzdump_restore" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>A backup archive can be restored through the Proxmox VE web GUI or through the
following CLI tools:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">pct restore</span>
</dt>
<dd>
<p>
Container restore utility
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">qmrestore</span>
</dt>
<dd>
<p>
Virtual Machine restore utility
</p>
</dd>
</dl></div>
<div class="paragraph">
<p>For details see the corresponding manual pages.</p></div>
<div class="sect3">
<h4 id="_bandwidth_limit">16.9.1. Bandwidth Limit
 <a class="headerlink" href="#_bandwidth_limit" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Restoring one or more big backups may need a lot of resources, especially
storage bandwidth for both reading from the backup storage and writing to
the target storage. This can negatively affect other virtual guests as access
to storage can get congested.</p></div>
<div class="paragraph">
<p>To avoid this you can set bandwidth limits for a backup job. Proxmox VE
implements two kinds of limits for restoring and archive:</p></div>
<div class="ulist"><ul>
<li>
<p>
per-restore limit: denotes the maximal amount of bandwidth for
  reading from a backup archive
</p>
</li>
<li>
<p>
per-storage write limit: denotes the maximal amount of bandwidth used for
  writing to a specific storage
</p>
</li>
</ul></div>
<div class="paragraph">
<p>The read limit indirectly affects the write limit, as we cannot write more
than we read. A smaller per-job limit will overwrite a bigger per-storage
limit. A bigger per-job limit will only overwrite the per-storage limit if
you have ‘Data.Allocate’ permissions on the affected storage.</p></div>
<div class="paragraph">
<p>You can use the ‘--bwlimit &lt;integer&gt;` option from the restore CLI commands
to set up a restore job specific bandwidth limit. KiB/s is used as unit
for the limit, this means passing `10240’ will limit the read speed of the
backup to 10 MiB/s, ensuring that the rest of the possible storage bandwidth
is available for the already running virtual guests, and thus the backup
does not impact their operations.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">You can use ‘0` for the <span class="monospaced">bwlimit</span> parameter to disable all limits for
a specific restore job. This can be helpful if you need to restore a very
important virtual guest as fast as possible. (Needs `Data.Allocate’
permissions on storage)</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>Most times your storage’s generally available bandwidth stays the same over
time, thus we implemented the possibility to set a default bandwidth limit
per configured storage, this can be done with:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre># pvesm set STORAGEID --bwlimit restore=KIBs</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_live_restore">16.9.2. Live-Restore
 <a class="headerlink" href="#_live_restore" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Restoring a large backup can take a long time, in which a guest is still
unavailable. For VM backups stored on a Proxmox Backup Server, this wait
time can be mitigated using the live-restore option.</p></div>
<div class="paragraph">
<p>Enabling live-restore via either the checkbox in the GUI or the <span class="monospaced">--live-restore</span>
argument of <span class="monospaced">qmrestore</span> causes the VM to start as soon as the restore
begins. Data is copied in the background, prioritizing chunks that the VM is
actively accessing.</p></div>
<div class="paragraph">
<p>Note that this comes with two caveats:</p></div>
<div class="ulist"><ul>
<li>
<p>
During live-restore, the VM will operate with limited disk read speeds, as
  data has to be loaded from the backup server (once loaded, it is immediately
  available on the destination storage however, so accessing data twice only
  incurs the penalty the first time). Write speeds are largely unaffected.
</p>
</li>
<li>
<p>
If the live-restore fails for any reason, the VM will be left in an
  undefined state - that is, not all data might have been copied from the
  backup, and it is <em>most likely</em> not possible to keep any data that was written
  during the failed restore operation.
</p>
</li>
</ul></div>
<div class="paragraph">
<p>This mode of operation is especially useful for large VMs, where only a small
amount of data is required for initial operation, e.g. web servers - once the OS
and necessary services have been started, the VM is operational, while the
background task continues copying seldom used data.</p></div>
</div>
<div class="sect3">
<h4 id="_single_file_restore">16.9.3. Single File Restore
 <a class="headerlink" href="#_single_file_restore" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The <em>File Restore</em> button in the <em>Backups</em> tab of the storage GUI can be used to
open a file browser directly on the data contained in a backup. This feature
is only available for backups on a Proxmox Backup Server.</p></div>
<div class="paragraph">
<p>For containers, the first layer of the file tree shows all included <em>pxar</em>
archives, which can be opened and browsed freely. For VMs, the first layer shows
contained drive images, which can be opened to reveal a list of supported
storage technologies found on the drive. In the most basic case, this will be an
entry called <em>part</em>, representing a partition table, which contains entries for
each partition found on the drive. Note that for VMs, not all data might be
accessible (unsupported guest file systems, storage technologies, etc…).</p></div>
<div class="paragraph">
<p>Files and directories can be downloaded using the <em>Download</em> button, the latter
being compressed into a zip archive on the fly.</p></div>
<div class="paragraph">
<p>To enable secure access to VM images, which might contain untrusted data, a
temporary VM (not visible as a guest) is started. This does not mean that data
downloaded from such an archive is inherently safe, but it avoids exposing the
hypervisor system to danger. The VM will stop itself after a timeout. This
entire process happens transparently from a user’s point of view.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">For troubleshooting purposes, each temporary VM instance generates a log
file in <span class="monospaced">/var/log/proxmox-backup/file-restore/</span>. The log file might contain
additional information in case an attempt to restore individual files or
accessing file systems contained in a backup archive fails.</td>
</tr></tbody></table>
</div>
</div>
</div>
<div class="sect2">
<h3 id="vzdump_configuration">
<span>16.10. Configuration</span>
 <a class="headerlink" href="#vzdump_configuration" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Global configuration is stored in <span class="monospaced">/etc/vzdump.conf</span>. The file uses a
simple colon separated key/value format. Each line has the following
format:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>OPTION: value</pre>
</div></div>
<div class="paragraph">
<p>Blank lines in the file are ignored, and lines starting with a <span class="monospaced">#</span>
character are treated as comments and are also ignored. Values from
this file are used as default, and can be overwritten on the command
line.</p></div>
<div class="paragraph">
<p>We currently support the following options:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">bwlimit</span>: <span class="monospaced">&lt;integer&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Limit I/O bandwidth (in KiB/s).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">compress</span>: <span class="monospaced">&lt;0 | 1 | gzip | lzo | zstd&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Compress dump file.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">dumpdir</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Store resulting files to specified directory.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">exclude-path</span>: <span class="monospaced">&lt;array&gt;</span> 
</dt>
<dd>
<p>
Exclude certain files/directories (shell globs). Paths starting with <em>/</em> are anchored to the container’s root, other paths match relative to each subdirectory.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">fleecing</span>: <span class="monospaced">[[enabled=]&lt;1|0&gt;] [,storage=&lt;storage ID&gt;]</span> 
</dt>
<dd>
<p>
Options for backup fleecing (VM only).
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">enabled</span>=<span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Enable backup fleecing. Cache backup data from blocks where new guest writes happen on specified storage instead of copying them directly to the backup target. This can help guest IO performance and even prevent hangs, at the cost of requiring more storage space.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">storage</span>=<span class="monospaced">&lt;storage ID&gt;</span> 
</dt>
<dd>
<p>
Use this storage to storage fleecing images. For efficient space usage, it’s best to use a local storage that supports discard and either thin provisioning or sparse files.
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">ionice</span>: <span class="monospaced">&lt;integer&gt; (0 - 8)</span> (<em>default =</em> <span class="monospaced">7</span>)
</dt>
<dd>
<p>
Set IO priority when using the BFQ scheduler. For snapshot and suspend mode backups of VMs, this only affects the compressor. A value of 8 means the idle priority is used, otherwise the best-effort priority is used with the specified value.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">lockwait</span>: <span class="monospaced">&lt;integer&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">180</span>)
</dt>
<dd>
<p>
Maximal time to wait for the global lock (minutes).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mailnotification</span>: <span class="monospaced">&lt;always | failure&gt;</span> (<em>default =</em> <span class="monospaced">always</span>)
</dt>
<dd>
<p>
Deprecated: use notification targets/matchers instead. Specify when to send a notification mail
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mailto</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Deprecated: Use notification targets/matchers instead. Comma-separated list of email addresses or users that should receive email notifications.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">maxfiles</span>: <span class="monospaced">&lt;integer&gt; (1 - N)</span> 
</dt>
<dd>
<p>
Deprecated: use <em>prune-backups</em> instead. Maximal number of backup files per guest system.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mode</span>: <span class="monospaced">&lt;snapshot | stop | suspend&gt;</span> (<em>default =</em> <span class="monospaced">snapshot</span>)
</dt>
<dd>
<p>
Backup mode.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">notes-template</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Template string for generating notes for the backup(s). It can contain variables which will be replaced by their values. Currently supported are {\{\cluster}}, {\{\guestname}}, {\{\node}}, and {\{\vmid}}, but more might be added in the future. Needs to be a single line, newline and backslash need to be escaped as <em>\n</em> and <em>\\</em> respectively.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Requires option(s): <span class="monospaced">storage</span></td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">notification-mode</span>: <span class="monospaced">&lt;auto | legacy-sendmail | notification-system&gt;</span> (<em>default =</em> <span class="monospaced">auto</span>)
</dt>
<dd>
<p>
Determine which notification system to use. If set to <em>legacy-sendmail</em>, vzdump will consider the mailto/mailnotification parameters and send emails to the specified address(es) via the <em>sendmail</em> command. If set to <em>notification-system</em>, a notification will be sent via PVE’s notification system, and the mailto and mailnotification will be ignored. If set to <em>auto</em> (default setting), an email will be sent if mailto is set, and the notification system will be used if not.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">notification-policy</span>: <span class="monospaced">&lt;always | failure | never&gt;</span> (<em>default =</em> <span class="monospaced">always</span>)
</dt>
<dd>
<p>
Deprecated: Do not use
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">notification-target</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Deprecated: Do not use
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">pbs-change-detection-mode</span>: <span class="monospaced">&lt;data | legacy | metadata&gt;</span> 
</dt>
<dd>
<p>
PBS mode used to detect file changes and switch encoding. NOTE: <span class="monospaced">data</span> and <span class="monospaced">metadata</span> modes are experimental. format for container backups.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">performance</span>: <span class="monospaced">[max-workers=&lt;integer&gt;] [,pbs-entries-max=&lt;integer&gt;]</span> 
</dt>
<dd>
<p>
Other performance-related settings.
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">max-workers</span>=<span class="monospaced">&lt;integer&gt; (1 - 256)</span> (<em>default =</em> <span class="monospaced">16</span>)
</dt>
<dd>
<p>
Applies to VMs. Allow up to this many IO workers at the same time.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">pbs-entries-max</span>=<span class="monospaced">&lt;integer&gt; (1 - N)</span> (<em>default =</em> <span class="monospaced">1048576</span>)
</dt>
<dd>
<p>
Applies to container backups sent to PBS. Limits the number of entries allowed in memory at a given time to avoid unintended OOM situations. Increase it to enable backups of containers with a large amount of files.
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">pigz</span>: <span class="monospaced">&lt;integer&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Use pigz instead of gzip when N&gt;0. N=1 uses half of cores, N&gt;1 uses N as thread count.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">pool</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Backup all known guest systems included in the specified pool.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">protected</span>: <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
If true, mark backup(s) as protected.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Requires option(s): <span class="monospaced">storage</span></td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">prune-backups</span>: <span class="monospaced">[keep-all=&lt;1|0&gt;] [,keep-daily=&lt;N&gt;] [,keep-hourly=&lt;N&gt;] [,keep-last=&lt;N&gt;] [,keep-monthly=&lt;N&gt;] [,keep-weekly=&lt;N&gt;] [,keep-yearly=&lt;N&gt;]</span> (<em>default =</em> <span class="monospaced">keep-all=1</span>)
</dt>
<dd>
<p>
Use these retention options instead of those from the storage configuration.
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">keep-all</span>=<span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Keep all backups. Conflicts with the other options when true.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">keep-daily</span>=<span class="monospaced">&lt;N&gt;</span> 
</dt>
<dd>
<p>
Keep backups for the last &lt;N&gt; different days. If there is morethan one backup for a single day, only the latest one is kept.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">keep-hourly</span>=<span class="monospaced">&lt;N&gt;</span> 
</dt>
<dd>
<p>
Keep backups for the last &lt;N&gt; different hours. If there is morethan one backup for a single hour, only the latest one is kept.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">keep-last</span>=<span class="monospaced">&lt;N&gt;</span> 
</dt>
<dd>
<p>
Keep the last &lt;N&gt; backups.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">keep-monthly</span>=<span class="monospaced">&lt;N&gt;</span> 
</dt>
<dd>
<p>
Keep backups for the last &lt;N&gt; different months. If there is morethan one backup for a single month, only the latest one is kept.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">keep-weekly</span>=<span class="monospaced">&lt;N&gt;</span> 
</dt>
<dd>
<p>
Keep backups for the last &lt;N&gt; different weeks. If there is morethan one backup for a single week, only the latest one is kept.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">keep-yearly</span>=<span class="monospaced">&lt;N&gt;</span> 
</dt>
<dd>
<p>
Keep backups for the last &lt;N&gt; different years. If there is morethan one backup for a single year, only the latest one is kept.
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">remove</span>: <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Prune older backups according to <em>prune-backups</em>.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">script</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Use specified hook script.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">stdexcludes</span>: <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Exclude temporary files and logs.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">stopwait</span>: <span class="monospaced">&lt;integer&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">10</span>)
</dt>
<dd>
<p>
Maximal time to wait until a guest system is stopped (minutes).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">storage</span>: <span class="monospaced">&lt;storage ID&gt;</span> 
</dt>
<dd>
<p>
Store resulting file to this storage.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">tmpdir</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Store temporary files to specified directory.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">zstd</span>: <span class="monospaced">&lt;integer&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Zstd threads. N=0 uses half of the available cores, if N is set to a value bigger than 0, N is used as thread count.
</p>
</dd>
</dl></div>
<div class="listingblock">
<div class="title">Example <span class="monospaced">vzdump.conf</span> Configuration</div>
<div class="content monospaced">
<pre>tmpdir: /mnt/fast_local_disk
storage: my_backup_storage
mode: snapshot
bwlimit: 10000</pre>
</div></div>
</div>
<div class="sect2">
<h3 id="_hook_scripts">
<span>16.11. Hook Scripts</span>
 <a class="headerlink" href="#_hook_scripts" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>You can specify a hook script with option <span class="monospaced">--script</span>. This script is
called at various phases of the backup process, with parameters
accordingly set. You can find an example in the documentation
directory (<span class="monospaced">vzdump-hook-script.pl</span>).</p></div>
</div>
<div class="sect2">
<h3 id="_file_exclusions">
<span>16.12. File Exclusions</span>
 <a class="headerlink" href="#_file_exclusions" title="Permalink to this heading"></a>
</h3>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">this option is only available for container backups.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p><span class="monospaced">vzdump</span> skips the following files by default (disable with the option
<span class="monospaced">--stdexcludes 0</span>)</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>/tmp/?*
/var/tmp/?*
/var/run/?*pid</pre>
</div></div>
<div class="paragraph">
<p>You can also manually specify (additional) exclude paths, for example:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># vzdump 777 --exclude-path /tmp/ --exclude-path '/var/foo*'</pre>
</div></div>
<div class="paragraph">
<p>excludes the directory <span class="monospaced">/tmp/</span> and any file or directory named <span class="monospaced">/var/foo</span>,
<span class="monospaced">/var/foobar</span>, and so on.</p></div>
<div class="paragraph">
<p>Paths that do not start with a <span class="monospaced">/</span> are not anchored to the container’s root,
but will match relative to any subdirectory. For example:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># vzdump 777 --exclude-path bar</pre>
</div></div>
<div class="paragraph">
<p>excludes any file or directory named <span class="monospaced">/bar</span>, <span class="monospaced">/var/bar</span>, <span class="monospaced">/var/foo/bar</span>, and
so on, but not <span class="monospaced">/bar2</span>.</p></div>
<div class="paragraph">
<p>Configuration files are also stored inside the backup archive
(in <span class="monospaced">./etc/vzdump/</span>) and will be correctly restored.</p></div>
</div>
<div class="sect2">
<h3 id="_examples_10">
<span>16.13. Examples</span>
 <a class="headerlink" href="#_examples_10" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Simply dump guest 777 - no snapshot, just archive the guest private area and
configuration files to the default dump directory (usually
<span class="monospaced">/var/lib/vz/dump/</span>).</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># vzdump 777</pre>
</div></div>
<div class="paragraph">
<p>Use rsync and suspend/resume to create a snapshot (minimal downtime).</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># vzdump 777 --mode suspend</pre>
</div></div>
<div class="paragraph">
<p>Backup all guest systems and send notification mails to root and admin.
Due to <span class="monospaced">mailto</span> being set and <span class="monospaced">notification-mode</span> being set to <span class="monospaced">auto</span> by
default, the notification mails are sent via the system’s <span class="monospaced">sendmail</span>
command instead of the notification system.</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># vzdump --all --mode suspend --mailto root --mailto admin</pre>
</div></div>
<div class="paragraph">
<p>Use snapshot mode (no downtime) and non-default dump directory.</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># vzdump 777 --dumpdir /mnt/backup --mode snapshot</pre>
</div></div>
<div class="paragraph">
<p>Backup more than one guest (selectively)</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># vzdump 101 102 103 --mailto root</pre>
</div></div>
<div class="paragraph">
<p>Backup all guests excluding 101 and 102</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># vzdump --mode suspend --exclude 101,102</pre>
</div></div>
<div class="paragraph">
<p>Restore a container to a new CT 600</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># pct restore 600 /mnt/backup/vzdump-lxc-777.tar</pre>
</div></div>
<div class="paragraph">
<p>Restore a QemuServer VM to VM 601</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># qmrestore /mnt/backup/vzdump-qemu-888.vma 601</pre>
</div></div>
<div class="paragraph">
<p>Clone an existing container 101 to a new container 300 with a 4GB root
file system, using pipes</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># vzdump 101 --stdout | pct restore --rootfs 4 300 -</pre>
</div></div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="chapter_notifications">
17. Notifications
 <a class="headerlink" href="#chapter_notifications" title="Permalink to this heading"></a>
</h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="_overview">
<span>17.1. Overview</span>
 <a class="headerlink" href="#_overview" title="Permalink to this heading"></a>
</h3>
<div class="ulist"><ul>
<li>
<p>
Proxmox VE emits <a href="#notification_events">Notification Events</a> in case of
  storage replication failures, node fencing, finished/failed backups
  and other events.
  These events are handled by the notification system. A notification
  event has metadata, for example a timestamp, a severity level,
  a type, and other optional metadata fields.
</p>
</li>
<li>
<p>
<a href="#notification_matchers">Notification Matchers</a> route a notification
  event to one or more notification targets. A matcher can have match
  rules to selectively route based on the metadata of a notification event.
</p>
</li>
<li>
<p>
<a href="#notification_targets">Notification Targets</a> are a destination to
  which a notification event is routed to by a matcher.
  There are multiple types of target, mail-based (Sendmail and SMTP)
  and Gotify.
</p>
</li>
</ul></div>
<div class="paragraph">
<p>Backup jobs have a configurable <a href="#notification_mode">Notification Mode</a>.
It allows you to choose between the notification system and a legacy mode
for sending notification emails. The legacy mode is equivalent to the
way notifications were handled before Proxmox VE 8.1.</p></div>
<div class="paragraph">
<p>The notification system can be configured in the GUI under
Datacenter → Notifications. The configuration is stored in
<span class="monospaced">/etc/pve/notifications.cfg</span> and <span class="monospaced">/etc/pve/priv/notifications.cfg</span> -
the latter contains sensitive configuration options such as
passwords or authentication tokens for notification targets and can
only be read by <span class="monospaced">root</span>.</p></div>
</div>
<div class="sect2">
<h3 id="notification_targets">
<span>17.2. Notification Targets</span>
 <a class="headerlink" href="#notification_targets" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Proxmox VE offers multiple types of notification targets.</p></div>
<div class="sect3">
<h4 id="notification_targets_sendmail">17.2.1. Sendmail
 <a class="headerlink" href="#notification_targets_sendmail" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-datacenter-notification-sendmail.png">
<img src="images/screenshot/gui-datacenter-notification-sendmail.png" alt="screenshot/gui-datacenter-notification-sendmail.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>The sendmail binary is a program commonly found on Unix-like operating systems
that handles the sending of email messages.
It is a command-line utility that allows users and applications to send emails
directly from the command line or from within scripts.</p></div>
<div class="paragraph">
<p>The sendmail notification target uses the <span class="monospaced">sendmail</span> binary to send emails to a
list of configured users or email addresses. If a user is selected as a recipient,
the email address configured in user’s settings will be used.
For the <span class="monospaced">root@pam</span> user, this is the email address entered during installation.
A user’s email address can be configured in
<span class="monospaced">Datacenter → Permissions → Users</span>.
If a user has no associated email address, no email will be sent.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">In standard Proxmox VE installations, the <span class="monospaced">sendmail</span> binary is provided by
Postfix. It may be necessary to configure Postfix so that it can deliver
mails correctly - for example by setting an external mail relay (smart host).
In case of failed delivery, check the system logs for messages logged by
the Postfix daemon.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>The configuration for Sendmail target plugins has the following options:</p></div>
<div class="ulist"><ul>
<li>
<p>
<span class="monospaced">mailto</span>: E-Mail address to which the notification shall be sent to. Can be
set multiple times to accomodate multiple recipients.
</p>
</li>
<li>
<p>
<span class="monospaced">mailto-user</span>: Users to which emails shall be sent to. The user’s email
address will be looked up in <span class="monospaced">users.cfg</span>. Can be set multiple times to
accomodate multiple recipients.
</p>
</li>
<li>
<p>
<span class="monospaced">author</span>: Sets the author of the E-Mail. Defaults to <span class="monospaced">Proxmox VE</span>.
</p>
</li>
<li>
<p>
<span class="monospaced">from-address</span>: Sets the from address of the E-Mail. If the parameter is not
set, the plugin will fall back to the <span class="monospaced">email_from</span> setting from
<span class="monospaced">datacenter.cfg</span>. If that is also not set, the plugin will default to
<span class="monospaced">root@$hostname</span>, where <span class="monospaced">$hostname</span> is the hostname of the node.
</p>
</li>
<li>
<p>
<span class="monospaced">comment</span>: Comment for this target
The <span class="monospaced">From</span> header in the email will be set to <span class="monospaced">$author &lt;$from-address&gt;</span>.
</p>
</li>
</ul></div>
<div class="paragraph">
<p>Example configuration (<span class="monospaced">/etc/pve/notifications.cfg</span>):</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>sendmail: example
        mailto-user root@pam
        mailto-user admin@pve
        mailto [email protected]
        from-address [email protected]
        comment Send to multiple users/addresses</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="notification_targets_smtp">17.2.2. SMTP
 <a class="headerlink" href="#notification_targets_smtp" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-datacenter-notification-smtp.png">
<img src="images/screenshot/gui-datacenter-notification-smtp.png" alt="screenshot/gui-datacenter-notification-smtp.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>SMTP notification targets can send emails directly to an SMTP mail relay.
This target does not use the system’s MTA to deliver emails.
Similar to sendmail targets, if a user is selected as a recipient, the user’s configured
email address will be used.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Unlike sendmail targets, SMTP targets do not have any queuing/retry mechanism
in case of a failed mail delivery.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>The configuration for SMTP target plugins has the following options:</p></div>
<div class="ulist"><ul>
<li>
<p>
<span class="monospaced">mailto</span>: E-Mail address to which the notification shall be sent to. Can be
set multiple times to accomodate multiple recipients.
</p>
</li>
<li>
<p>
<span class="monospaced">mailto-user</span>: Users to which emails shall be sent to. The user’s email
address will be looked up in <span class="monospaced">users.cfg</span>. Can be set multiple times to
accomodate multiple recipients.
</p>
</li>
<li>
<p>
<span class="monospaced">author</span>: Sets the author of the E-Mail. Defaults to <span class="monospaced">Proxmox VE</span>.
</p>
</li>
<li>
<p>
<span class="monospaced">from-address</span>: Sets the From-addresss of the email. SMTP relays might require
that this address is owned by the user in order to avoid spoofing.
The <span class="monospaced">From</span> header in the email will be set to <span class="monospaced">$author &lt;$from-address&gt;</span>.
</p>
</li>
<li>
<p>
<span class="monospaced">username</span>: Username to use during authentication. If no username is set,
no authentication will be performed. The PLAIN and LOGIN authentication methods
are supported.
</p>
</li>
<li>
<p>
<span class="monospaced">password</span>: Password to use when authenticating.
</p>
</li>
<li>
<p>
<span class="monospaced">mode</span>: Sets the encryption mode (<span class="monospaced">insecure</span>, <span class="monospaced">starttls</span> or <span class="monospaced">tls</span>). Defaults
to <span class="monospaced">tls</span>.
</p>
</li>
<li>
<p>
<span class="monospaced">server</span>: Address/IP of the SMTP relay
</p>
</li>
<li>
<p>
<span class="monospaced">port</span>: The port to connect to. If not set, the used port
defaults to 25 (<span class="monospaced">insecure</span>), 465 (<span class="monospaced">tls</span>) or 587 (<span class="monospaced">starttls</span>), depending on the
value of <span class="monospaced">mode</span>.
</p>
</li>
<li>
<p>
<span class="monospaced">comment</span>: Comment for this target
</p>
</li>
</ul></div>
<div class="paragraph">
<p>Example configuration (<span class="monospaced">/etc/pve/notifications.cfg</span>):</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>smtp: example
        mailto-user root@pam
        mailto-user admin@pve
        mailto [email protected]
        from-address [email protected]
        username pve1
        server mail.example.com
        mode starttls</pre>
</div></div>
<div class="paragraph">
<p>The matching entry in <span class="monospaced">/etc/pve/priv/notifications.cfg</span>, containing the
secret token:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>smtp: example
        password somepassword</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="notification_targets_gotify">17.2.3. Gotify
 <a class="headerlink" href="#notification_targets_gotify" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-datacenter-notification-gotify.png">
<img src="images/screenshot/gui-datacenter-notification-gotify.png" alt="screenshot/gui-datacenter-notification-gotify.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p><a href="http://gotify.net">Gotify</a> is an open-source self-hosted notification server that
allows you to send and receive push notifications to various devices and
applications. It provides a simple API and web interface, making it easy to
integrate with different platforms and services.</p></div>
<div class="paragraph">
<p>The configuration for Gotify target plugins has the following options:</p></div>
<div class="ulist"><ul>
<li>
<p>
<span class="monospaced">server</span>: The base URL of the Gotify server, e.g. <span class="monospaced">http://&lt;ip&gt;:8888</span>
</p>
</li>
<li>
<p>
<span class="monospaced">token</span>: The authentication token. Tokens can be generated within the Gotify
web interface.
</p>
</li>
<li>
<p>
<span class="monospaced">comment</span>: Comment for this target
</p>
</li>
</ul></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">The Gotify target plugin will respect the HTTP proxy settings from the
 <a href="#datacenter_configuration_file">datacenter configuration</a></td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>Example configuration (<span class="monospaced">/etc/pve/notifications.cfg</span>):</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>gotify: example
        server http://gotify.example.com:8888
        comment Send to multiple users/addresses</pre>
</div></div>
<div class="paragraph">
<p>The matching entry in <span class="monospaced">/etc/pve/priv/notifications.cfg</span>, containing the
secret token:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>gotify: example
        token somesecrettoken</pre>
</div></div>
</div>
</div>
<div class="sect2">
<h3 id="notification_matchers">
<span>17.3. Notification Matchers</span>
 <a class="headerlink" href="#notification_matchers" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<a class="image" href="images/screenshot/gui-datacenter-notification-matcher.png">
<img src="images/screenshot/gui-datacenter-notification-matcher.png" alt="screenshot/gui-datacenter-notification-matcher.png" width="250" style="padding: 0 0 0 10px;float:right;"></a>
<p>Notification matchers route notifications to notification targets based
on their matching rules. These rules can match certain properties of a
notification, such as the timestamp (<span class="monospaced">match-calendar</span>), the severity of
the notification (<span class="monospaced">match-severity</span>) or metadata fields (<span class="monospaced">match-field</span>).
If a notification is matched by a matcher, all targets configured for the
matcher will receive the notification.</p></div>
<div class="paragraph">
<p>An arbitrary number of matchers can be created, each with with their own
matching rules and targets to notify.
Every target is notified at most once for every notification, even if
the target is used in multiple matchers.</p></div>
<div class="paragraph">
<p>A matcher without any matching rules is always true; the configured targets
will always be notified.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>matcher: always-matches
        target admin
        comment This matcher always matches</pre>
</div></div>
<div class="sect3">
<h4 id="_matcher_options">17.3.1. Matcher Options
 <a class="headerlink" href="#_matcher_options" title="Permalink to this heading"></a>
</h4>
<div class="ulist"><ul>
<li>
<p>
<span class="monospaced">target</span>: Determine which target should be notified if the matcher matches.
can be used multiple times to notify multiple targets.
</p>
</li>
<li>
<p>
<span class="monospaced">invert-match</span>: Inverts the result of the whole matcher
</p>
</li>
<li>
<p>
<span class="monospaced">mode</span>: Determines how the individual match rules are evaluated to compute
the result for the whole matcher. If set to <span class="monospaced">all</span>, all matching rules must
match. If set to <span class="monospaced">any</span>, at least one rule must match.
a matcher must be true. Defaults to <span class="monospaced">all</span>.
</p>
</li>
<li>
<p>
<span class="monospaced">match-calendar</span>: Match the notification’s timestamp against a schedule
</p>
</li>
<li>
<p>
<span class="monospaced">match-field</span>: Match the notification’s metadata fields
</p>
</li>
<li>
<p>
<span class="monospaced">match-severity</span>: Match the notification’s severity
</p>
</li>
<li>
<p>
<span class="monospaced">comment</span>: Comment for this matcher
</p>
</li>
</ul></div>
</div>
<div class="sect3">
<h4 id="notification_matchers_calendar">17.3.2. Calendar Matching Rules
 <a class="headerlink" href="#notification_matchers_calendar" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>A calendar matcher matches the time when a notification is sent agaist a
configurable schedule.</p></div>
<div class="ulist"><ul>
<li>
<p>
<span class="monospaced">match-calendar 8-12</span>
</p>
</li>
<li>
<p>
<span class="monospaced">match-calendar 8:00-15:30</span>
</p>
</li>
<li>
<p>
<span class="monospaced">match-calendar mon-fri 9:00-17:00</span>
</p>
</li>
<li>
<p>
<span class="monospaced">match-calendar sun,tue-wed,fri 9-17</span>
</p>
</li>
</ul></div>
</div>
<div class="sect3">
<h4 id="notification_matchers_field">17.3.3. Field Matching Rules
 <a class="headerlink" href="#notification_matchers_field" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Notifications have a selection of metadata fields that can be matched.</p></div>
<div class="ulist"><ul>
<li>
<p>
<span class="monospaced">match-field exact:type=vzdump</span> Only match notifications about backups.
</p>
</li>
<li>
<p>
<span class="monospaced">match-field regex:hostname=^.+\.example\.com$</span> Match the hostname of
the node.
</p>
</li>
</ul></div>
<div class="paragraph">
<p>If a matched metadata field does not exist, the notification will not be
matched.
For instance, a <span class="monospaced">match-field regex:hostname=.*</span> directive will only match
notifications that have an arbitraty <span class="monospaced">hostname</span> metadata field, but will
not match if the field does not exist.</p></div>
</div>
<div class="sect3">
<h4 id="notification_matchers_severity">17.3.4. Severity Matching Rules
 <a class="headerlink" href="#notification_matchers_severity" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>A notification has a associated severity that can be matched.</p></div>
<div class="ulist"><ul>
<li>
<p>
<span class="monospaced">match-severity error</span>: Only match errors
</p>
</li>
<li>
<p>
<span class="monospaced">match-severity warning,error</span>: Match warnings and error
</p>
</li>
</ul></div>
<div class="paragraph">
<p>The following severities are in use:
<span class="monospaced">info</span>, <span class="monospaced">notice</span>, <span class="monospaced">warning</span>, <span class="monospaced">error</span>, <span class="monospaced">unknown</span>.</p></div>
</div>
<div class="sect3">
<h4 id="_examples_11">17.3.5. Examples
 <a class="headerlink" href="#_examples_11" title="Permalink to this heading"></a>
</h4>
<div class="listingblock">
<div class="content monospaced">
<pre>matcher: workday
        match-calendar mon-fri 9-17
        target admin
        comment Notify admins during working hours

matcher: night-and-weekend
        match-calendar mon-fri 9-17
        invert-match true
        target on-call-admins
        comment Separate target for non-working hours</pre>
</div></div>
<div class="listingblock">
<div class="content monospaced">
<pre>matcher: backup-failures
        match-field exact:type=vzdump
        match-severity error
        target backup-admins
        comment Send notifications about backup failures to one group of admins

matcher: cluster-failures
        match-field exact:type=replication
        match-field exact:type=fencing
        mode any
        target cluster-admins
        comment Send cluster-related notifications to other group of admins</pre>
</div></div>
<div class="paragraph">
<p>The last matcher could also be rewritten using a field matcher with a regular
expression:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>matcher: cluster-failures
        match-field regex:type=^(replication|fencing)$
        target cluster-admins
        comment Send cluster-related notifications to other group of admins</pre>
</div></div>
</div>
</div>
<div class="sect2">
<h3 id="notification_events">
<span>17.4. Notification Events</span>
 <a class="headerlink" href="#notification_events" title="Permalink to this heading"></a>
</h3>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top"> Event                        </th>
<th class="tableblock halign-left valign-top"> <span class="monospaced">type</span>            </th>
<th class="tableblock halign-left valign-top"> Severity </th>
<th class="tableblock halign-left valign-top"> Metadata fields (in addition to <span class="monospaced">type</span>)</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">System updates available</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><span class="monospaced">package-updates</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><span class="monospaced">info</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><span class="monospaced">hostname</span></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">Cluster node fenced</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><span class="monospaced">fencing</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><span class="monospaced">error</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><span class="monospaced">hostname</span></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">Storage replication failed</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><span class="monospaced">replication</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><span class="monospaced">error</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">-</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">Backup finished</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><span class="monospaced">vzdump</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><span class="monospaced">info</span> (<span class="monospaced">error</span> on failure)</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><span class="monospaced">hostname</span></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">Mail for root</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><span class="monospaced">system-mail</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><span class="monospaced">unknown</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">-</p></td>
</tr>
</tbody>
</table>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:50%;">
<col style="width:50%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top"> Field name </th>
<th class="tableblock halign-left valign-top"> Description</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><span class="monospaced">type</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Type of the notification</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><span class="monospaced">hostname</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Hostname, without domain (e.g. <span class="monospaced">pve1</span>)</p></td>
</tr>
</tbody>
</table>
</div>
<div class="sect2">
<h3 id="_system_mail_forwarding">
<span>17.5. System Mail Forwarding</span>
 <a class="headerlink" href="#_system_mail_forwarding" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Certain local system daemons, such as <span class="monospaced">smartd</span>, generate notification emails
that are initially directed to the local <span class="monospaced">root</span> user. Proxmox VE will
feed these mails into the notification system as a notification of
type <span class="monospaced">system-mail</span> and with severity <span class="monospaced">unknown</span>.</p></div>
<div class="paragraph">
<p>When the email is forwarded to a sendmail target, the mail’s content and headers
are forwarded as-is. For all other targets,
the system tries to extract both a subject line and the main text body
from the email content. In instances where emails solely consist of HTML
content, they will be transformed into plain text format during this process.</p></div>
</div>
<div class="sect2">
<h3 id="_permissions_2">
<span>17.6. Permissions</span>
 <a class="headerlink" href="#_permissions_2" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>To modify/view the configuration for notification targets,
the <span class="monospaced">Mapping.Modify/Mapping.Audit</span> permissions are required for the
<span class="monospaced">/mapping/notifications</span> ACL node.</p></div>
<div class="paragraph">
<p>Testing a target requires <span class="monospaced">Mapping.Use</span>, <span class="monospaced">Mapping.Audit</span> or <span class="monospaced">Mapping.Modify</span>
permissions on <span class="monospaced">/mapping/notifications</span></p></div>
</div>
<div class="sect2">
<h3 id="notification_mode">
<span>17.7. Notification Mode</span>
 <a class="headerlink" href="#notification_mode" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>A backup job configuration has the <span class="monospaced">notification-mode</span>
option which can have one of three values.</p></div>
<div class="ulist"><ul>
<li>
<p>
<span class="monospaced">auto</span>: Use the <span class="monospaced">legacy-sendmail</span> mode if no email address is entered in the
  <span class="monospaced">mailto</span>/<span class="monospaced">Send email to</span> field. If no email address is entered,
  the <span class="monospaced">notification-system</span> mode is used.
</p>
</li>
<li>
<p>
<span class="monospaced">legacy-sendmail</span>: Send notification emails via the system’s <span class="monospaced">sendmail</span> command.
  The notification system will be bypassed and any configured
  targets/matchers will be ignored.
  This mode is equivalent to the notification behavior for version before
  Proxmox VE 8.1 .
</p>
</li>
<li>
<p>
<span class="monospaced">notification-system</span>: Use the new, flexible notification system.
</p>
</li>
</ul></div>
<div class="paragraph">
<p>If the <span class="monospaced">notification-mode</span> option is not set, Proxmox VE will default
to <span class="monospaced">auto</span>.</p></div>
<div class="paragraph">
<p>The <span class="monospaced">legacy-sendmail</span> mode might be removed in a later release of
Proxmox VE.</p></div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_important_service_daemons">
18. Important Service Daemons
 <a class="headerlink" href="#_important_service_daemons" title="Permalink to this heading"></a>
</h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="_pvedaemon_proxmox_ve_api_daemon">
<span>18.1. pvedaemon - Proxmox VE API Daemon</span>
 <a class="headerlink" href="#_pvedaemon_proxmox_ve_api_daemon" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>This daemon exposes the whole Proxmox VE API on <span class="monospaced">127.0.0.1:85</span>. It runs as
<span class="monospaced">root</span> and has permission to do all privileged operations.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">The daemon listens to a local address only, so you cannot access
it from outside. The <span class="monospaced">pveproxy</span> daemon exposes the API to the outside
world.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect2">
<h3 id="_pveproxy_proxmox_ve_api_proxy_daemon">
<span>18.2. pveproxy - Proxmox VE API Proxy Daemon</span>
 <a class="headerlink" href="#_pveproxy_proxmox_ve_api_proxy_daemon" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>This daemon exposes the whole Proxmox VE API on TCP port 8006 using HTTPS. It runs
as user <span class="monospaced">www-data</span> and has very limited permissions.  Operation requiring more
permissions are forwarded to the local <span class="monospaced">pvedaemon</span>.</p></div>
<div class="paragraph">
<p>Requests targeted for other nodes are automatically forwarded to those nodes.
This means that you can manage your whole cluster by connecting to a single
Proxmox VE node.</p></div>
<div class="sect3">
<h4 id="pveproxy_host_acls">18.2.1. Host based Access Control
 <a class="headerlink" href="#pveproxy_host_acls" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>It is possible to configure “apache2”-like access control lists. Values are
read from file <span class="monospaced">/etc/default/pveproxy</span>. For example:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>ALLOW_FROM="10.0.0.1-10.0.0.5,192.168.0.0/22"
DENY_FROM="all"
POLICY="allow"</pre>
</div></div>
<div class="paragraph">
<p>IP addresses can be specified using any syntax understood by <span class="monospaced">Net::IP</span>. The
name <span class="monospaced">all</span> is an alias for <span class="monospaced">0/0</span> and <span class="monospaced">::/0</span> (meaning all IPv4 and IPv6
addresses).</p></div>
<div class="paragraph">
<p>The default policy is <span class="monospaced">allow</span>.</p></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:33%;">
<col style="width:33%;">
<col style="width:33%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top"> Match                      </th>
<th class="tableblock halign-left valign-top"> POLICY=deny </th>
<th class="tableblock halign-left valign-top"> POLICY=allow</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">Match Allow only</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">allow</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">allow</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">Match Deny only</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">deny</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">deny</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">No match</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">deny</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">allow</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">Match Both Allow &amp; Deny</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">deny</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">allow</p></td>
</tr>
</tbody>
</table>
</div>
<div class="sect3">
<h4 id="pveproxy_listening_address">18.2.2. Listening IP Address
 <a class="headerlink" href="#pveproxy_listening_address" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>By default the <span class="monospaced">pveproxy</span> and <span class="monospaced">spiceproxy</span> daemons listen on the wildcard
address and accept connections from both IPv4 and IPv6 clients.</p></div>
<div class="paragraph">
<p>By setting <span class="monospaced">LISTEN_IP</span> in <span class="monospaced">/etc/default/pveproxy</span> you can control to which IP
address the <span class="monospaced">pveproxy</span> and <span class="monospaced">spiceproxy</span> daemons bind. The IP-address needs to
be configured on the system.</p></div>
<div class="paragraph">
<p>Setting the <span class="monospaced">sysctl</span> <span class="monospaced">net.ipv6.bindv6only</span> to the non-default <span class="monospaced">1</span> will cause
the daemons to only accept connection from IPv6 clients, while  usually also
causing lots of other issues. If you set this configuration we recommend to
either remove the <span class="monospaced">sysctl</span> setting, or set the <span class="monospaced">LISTEN_IP</span> to <span class="monospaced">0.0.0.0</span> (which
will only allow IPv4 clients).</p></div>
<div class="paragraph">
<p><span class="monospaced">LISTEN_IP</span> can be used to only to restricting the socket to an internal
interface and thus have less exposure to the public internet, for example:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>LISTEN_IP="192.0.2.1"</pre>
</div></div>
<div class="paragraph">
<p>Similarly, you can also set an IPv6 address:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>LISTEN_IP="2001:db8:85a3::1"</pre>
</div></div>
<div class="paragraph">
<p>Note that if you want to specify a link-local IPv6 address, you need to provide
the interface name itself. For example:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>LISTEN_IP="fe80::c463:8cff:feb9:6a4e%vmbr0"</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">The nodes in a cluster need access to <span class="monospaced">pveproxy</span> for communication,
possibly on different sub-nets. It is <strong>not recommended</strong> to set <span class="monospaced">LISTEN_IP</span> on
clustered systems.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>To apply the change you need to either reboot your node or fully restart the
<span class="monospaced">pveproxy</span> and <span class="monospaced">spiceproxy</span> service:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>systemctl restart pveproxy.service spiceproxy.service</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Unlike <span class="monospaced">reload</span>, a <span class="monospaced">restart</span> of the pveproxy service can interrupt some
long-running worker processes, for example a running console or shell from a
virtual guest. So, please use a maintenance window to bring this change in
effect.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="_ssl_cipher_suite">18.2.3. SSL Cipher Suite
 <a class="headerlink" href="#_ssl_cipher_suite" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>You can define the cipher list in <span class="monospaced">/etc/default/pveproxy</span> via the <span class="monospaced">CIPHERS</span>
(TLS ⇐ 1.2) and <span class="monospaced">CIPHERSUITES</span> (TLS &gt;= 1.3) keys. For example</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>CIPHERS="ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256"
CIPHERSUITES="TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256"</pre>
</div></div>
<div class="paragraph">
<p>Above is the default. See the ciphers(1) man page from the openssl
package for a list of all available options.</p></div>
<div class="paragraph">
<p>Additionally, you can set the client to choose the cipher used in
<span class="monospaced">/etc/default/pveproxy</span> (default is the first cipher in the list available to
both client and <span class="monospaced">pveproxy</span>):</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>HONOR_CIPHER_ORDER=0</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_supported_tls_versions">18.2.4. Supported TLS versions
 <a class="headerlink" href="#_supported_tls_versions" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The insecure SSL versions 2 and 3 are unconditionally disabled for pveproxy.
TLS versions below 1.1 are disabled by default on recent OpenSSL versions,
which is honored by <span class="monospaced">pveproxy</span> (see <span class="monospaced">/etc/ssl/openssl.cnf</span>).</p></div>
<div class="paragraph">
<p>To disable TLS version 1.2 or 1.3, set the following in <span class="monospaced">/etc/default/pveproxy</span>:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>DISABLE_TLS_1_2=1</pre>
</div></div>
<div class="paragraph">
<p>or, respectively:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>DISABLE_TLS_1_3=1</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Unless there is a specific reason to do so, it is not recommended to
manually adjust the supported TLS versions.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="_diffie_hellman_parameters">18.2.5. Diffie-Hellman Parameters
 <a class="headerlink" href="#_diffie_hellman_parameters" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>You can define the used Diffie-Hellman parameters in
<span class="monospaced">/etc/default/pveproxy</span> by setting <span class="monospaced">DHPARAMS</span> to the path of a file
containing DH parameters in PEM format, for example</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>DHPARAMS="/path/to/dhparams.pem"</pre>
</div></div>
<div class="paragraph">
<p>If this option is not set, the built-in <span class="monospaced">skip2048</span> parameters will be
used.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">DH parameters are only used if a cipher suite utilizing the DH key
exchange algorithm is negotiated.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect3">
<h4 id="pveproxy_custom_tls_cert">18.2.6. Alternative HTTPS certificate
 <a class="headerlink" href="#pveproxy_custom_tls_cert" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>You can change the certificate used to an external one or to one obtained via
ACME.</p></div>
<div class="paragraph">
<p>pveproxy uses <span class="monospaced">/etc/pve/local/pveproxy-ssl.pem</span> and
<span class="monospaced">/etc/pve/local/pveproxy-ssl.key</span>, if present, and falls back to
<span class="monospaced">/etc/pve/local/pve-ssl.pem</span> and <span class="monospaced">/etc/pve/local/pve-ssl.key</span>.
The private key may not use a passphrase.</p></div>
<div class="paragraph">
<p>It is possible to override the location of the certificate private key
<span class="monospaced">/etc/pve/local/pveproxy-ssl.key</span> by setting <span class="monospaced">TLS_KEY_FILE</span> in
<span class="monospaced">/etc/default/pveproxy</span>, for example:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>TLS_KEY_FILE="/secrets/pveproxy.key"</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">The included ACME integration does not honor this setting.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>See the Host System Administration chapter of the documentation for details.</p></div>
</div>
<div class="sect3">
<h4 id="pveproxy_response_compression">18.2.7. Response Compression
 <a class="headerlink" href="#pveproxy_response_compression" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>By default <span class="monospaced">pveproxy</span> uses gzip HTTP-level compression for compressible
content, if the client supports it. This can disabled in <span class="monospaced">/etc/default/pveproxy</span></p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>COMPRESSION=0</pre>
</div></div>
</div>
</div>
<div class="sect2">
<h3 id="_pvestatd_proxmox_ve_status_daemon">
<span>18.3. pvestatd - Proxmox VE Status Daemon</span>
 <a class="headerlink" href="#_pvestatd_proxmox_ve_status_daemon" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>This daemon queries the status of VMs, storages and containers at
regular intervals. The result is sent to all nodes in the cluster.</p></div>
</div>
<div class="sect2">
<h3 id="_spiceproxy_spice_proxy_service">
<span>18.4. spiceproxy - SPICE Proxy Service</span>
 <a class="headerlink" href="#_spiceproxy_spice_proxy_service" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p><a href="http://www.spice-space.org">SPICE</a> (the Simple Protocol for Independent
Computing Environments) is an open remote computing solution,
providing client access to remote displays and devices (e.g. keyboard,
mouse, audio). The main use case is to get remote access to virtual
machines and container.</p></div>
<div class="paragraph">
<p>This daemon listens on TCP port 3128, and implements an HTTP proxy to
forward <em>CONNECT</em> request from the SPICE client to the correct Proxmox VE
VM. It runs as user <span class="monospaced">www-data</span> and has very limited permissions.</p></div>
<div class="sect3">
<h4 id="_host_based_access_control">18.4.1. Host based Access Control
 <a class="headerlink" href="#_host_based_access_control" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>It is possible to configure "apache2" like access control
lists. Values are read from file <span class="monospaced">/etc/default/pveproxy</span>.
See <span class="monospaced">pveproxy</span> documentation for details.</p></div>
</div>
</div>
<div class="sect2">
<h3 id="_pvescheduler_proxmox_ve_scheduler_daemon">
<span>18.5. pvescheduler - Proxmox VE Scheduler Daemon</span>
 <a class="headerlink" href="#_pvescheduler_proxmox_ve_scheduler_daemon" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>This deamon is responsible for starting jobs according to the schedule,
such as replication and vzdump jobs.</p></div>
<div class="paragraph">
<p>For vzdump jobs, it gets its configuration from the file <span class="monospaced">/etc/pve/jobs.cfg</span></p></div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_useful_command_line_tools">
19. Useful Command-line Tools
 <a class="headerlink" href="#_useful_command_line_tools" title="Permalink to this heading"></a>
</h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="_pvesubscription_subscription_management">
<span>19.1. pvesubscription - Subscription Management</span>
 <a class="headerlink" href="#_pvesubscription_subscription_management" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>This tool is used to handle Proxmox VE subscriptions.</p></div>
</div>
<div class="sect2">
<h3 id="_pveperf_proxmox_ve_benchmark_script">
<span>19.2. pveperf - Proxmox VE Benchmark Script</span>
 <a class="headerlink" href="#_pveperf_proxmox_ve_benchmark_script" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Tries to gather some CPU/hard disk performance data on the hard disk
mounted at <span class="monospaced">PATH</span> (<span class="monospaced">/</span> is used as default):</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
CPU BOGOMIPS
</dt>
<dd>
<p>
bogomips sum of all CPUs
</p>
</dd>
<dt class="hdlist1">
REGEX/SECOND
</dt>
<dd>
<p>
regular expressions per second (perl performance test), should
be above 300000
</p>
</dd>
<dt class="hdlist1">
HD SIZE
</dt>
<dd>
<p>
hard disk size
</p>
</dd>
<dt class="hdlist1">
BUFFERED READS
</dt>
<dd>
<p>
simple HD read test. Modern HDs should reach at least 40
MB/sec
</p>
</dd>
<dt class="hdlist1">
AVERAGE SEEK TIME
</dt>
<dd>
<p>
tests average seek time. Fast SCSI HDs reach values &lt; 8
milliseconds. Common IDE/SATA disks get values from 15 to 20 ms.
</p>
</dd>
<dt class="hdlist1">
FSYNCS/SECOND
</dt>
<dd>
<p>
value should be greater than 200 (you should enable <span class="monospaced">write
back</span> cache mode on you RAID controller - needs a battery backed cache (BBWC)).
</p>
</dd>
<dt class="hdlist1">
DNS EXT
</dt>
<dd>
<p>
average time to resolve an external DNS name
</p>
</dd>
<dt class="hdlist1">
DNS INT
</dt>
<dd>
<p>
average time to resolve a local DNS name
</p>
</dd>
</dl></div>
</div>
<div class="sect2">
<h3 id="_shell_interface_for_the_proxmox_ve_api">
<span>19.3. Shell interface for the Proxmox VE API</span>
 <a class="headerlink" href="#_shell_interface_for_the_proxmox_ve_api" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>The Proxmox VE management tool (<span class="monospaced">pvesh</span>) allows to directly invoke API
function, without using the REST/HTTPS server.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Only <em>root</em> is allowed to do that.</td>
</tr></tbody></table>
</div>
<div class="sect3">
<h4 id="_examples_12">19.3.1. EXAMPLES
 <a class="headerlink" href="#_examples_12" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Get the list of nodes in my cluster</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># pvesh get /nodes</pre>
</div></div>
<div class="paragraph">
<p>Get a list of available options for the datacenter</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># pvesh usage cluster/options -v</pre>
</div></div>
<div class="paragraph">
<p>Set the HTMl5 NoVNC console as the default console for the datacenter</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre># pvesh set cluster/options -console html5</pre>
</div></div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_frequently_asked_questions_2">
20. Frequently Asked Questions
 <a class="headerlink" href="#_frequently_asked_questions_2" title="Permalink to this heading"></a>
</h2>
<div class="sectionbody">
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">New FAQs are appended to the bottom of this section.</td>
</tr></tbody></table>
</div>
<div class="qlist qanda"><ol>
<li>
<p><em>
What distribution is Proxmox VE based on?
</em></p>
<p>
Proxmox VE is based on <a href="https://www.debian.org">Debian GNU/Linux</a>
</p>
</li>
<li>
<p><em>
What license does the Proxmox VE project use?
</em></p>
<p>
Proxmox VE code is licensed under the GNU Affero General Public License,
version 3.
</p>
</li>
<li>
<p><em>
Will Proxmox VE run on a 32bit processor?
</em></p>
<p>
Proxmox VE works only on 64-bit CPUs (AMD or Intel). There is no plan
for 32-bit for the platform.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">VMs and Containers can be both 32-bit and 64-bit.</td>
</tr></tbody></table>
</div>
</li>
<li>
<p><em>
Does my CPU support virtualization?
</em></p>
<p>
To check if your CPU is virtualization compatible, check for the <span class="monospaced">vmx</span>
or <span class="monospaced">svm</span> tag in this command output:
</p>
<div class="listingblock">
<div class="content monospaced">
<pre>egrep '(vmx|svm)' /proc/cpuinfo</pre>
</div></div>
</li>
<li>
<p><em>
Supported Intel CPUs
</em></p>
<p>
64-bit processors with
<a href="https://en.wikipedia.org/wiki/Virtualization_Technology#Intel_virtualization_.28VT-x.29">Intel
Virtualization Technology (Intel VT-x)</a> support.
(<a href="https://ark.intel.com/content/www/us/en/ark/search/featurefilter.html?productType=873&amp;2_VTX=True&amp;2_InstructionSet=64-bit">List of processors with Intel VT and 64-bit</a>)
</p>
</li>
<li>
<p><em>
Supported AMD CPUs
</em></p>
<p>
64-bit processors with
<a href="https://en.wikipedia.org/wiki/Virtualization_Technology#AMD_virtualization_.28AMD-V.29">AMD
Virtualization Technology (AMD-V)</a> support.
</p>
</li>
<li>
<p><em>
What is a container/virtual environment (VE)/virtual private server (VPS)?
</em></p>
<p>
In the context of containers, these terms all refer to the concept of
operating-system-level virtualization. Operating-system-level virtualization is
a method of virtualization, in which the kernel of an operating system
allows for multiple isolated instances, that all share the kernel. When
referring to LXC, we call such instances containers. Because containers use the
host’s kernel rather than emulating a full operating system, they require less
overhead, but are limited to Linux guests.
</p>
</li>
<li>
<p><em>
What is a QEMU/KVM guest (or VM)?
</em></p>
<p>
A QEMU/KVM guest (or VM) is a guest system running virtualized under
Proxmox VE using QEMU and the Linux KVM kernel module.
</p>
</li>
<li>
<p><em>
What is QEMU?
</em></p>
<p>
QEMU is a generic and open source machine emulator and
virtualizer. QEMU uses the Linux KVM kernel module to achieve near
native performance by executing the guest code directly on the host
CPU.
It is not limited to Linux guests but allows arbitrary operating systems
to run.
</p>
</li>
<li>
<p><em>
How long will my Proxmox VE version be supported?
</em></p>
<p>
Proxmox VE versions are supported at least as long as the corresponding
Debian Version is
<a href="https://wiki.debian.org/DebianOldStable">oldstable</a>. Proxmox VE uses a
rolling release model and using the latest stable version is always
recommended.
</p>
<table class="tableblock frame-all grid-all" id="faq-support-table" style="
width:100%;
">
<colgroup><col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
<col style="width:20%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top"> Proxmox VE Version </th>
<th class="tableblock halign-left valign-top"> Debian Version      </th>
<th class="tableblock halign-left valign-top"> First Release </th>
<th class="tableblock halign-left valign-top"> Debian EOL </th>
<th class="tableblock halign-left valign-top"> Proxmox EOL</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">Proxmox VE 8</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Debian 12 (Bookworm)</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2023-06</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tba</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tba</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">Proxmox VE 7</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Debian 11 (Bullseye)</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2021-07</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2024-07</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2024-07</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">Proxmox VE 6</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Debian 10 (Buster)</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2019-07</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2022-09</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2022-09</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">Proxmox VE 5</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Debian 9 (Stretch)</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2017-07</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2020-07</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2020-07</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">Proxmox VE 4</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Debian 8 (Jessie)</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2015-10</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2018-06</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2018-06</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">Proxmox VE 3</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Debian 7 (Wheezy)</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2013-05</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2016-04</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2017-02</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">Proxmox VE 2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Debian 6 (Squeeze)</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2012-04</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2014-05</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2014-05</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">Proxmox VE 1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Debian 5 (Lenny)</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2008-10</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2012-03</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2013-01</p></td>
</tr>
</tbody>
</table>
</li>
<li>
<p><em>
How can I upgrade Proxmox VE to the next point release?
</em></p>
<p>
Minor version upgrades, for example upgrading from Proxmox VE in version 7.1
to 7.2 or 7.3, can be done just like any normal update.
But you should still check the <a href="https://pve.proxmox.com/wiki/Roadmap">release notes</a>
for any relevant noteable, or breaking change.
</p>
<div class="paragraph" id="faq-upgrade">
<p>For the update itself use either the Web UI <em>Node → Updates</em> panel or
through the CLI with:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>apt update
apt full-upgrade</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Always ensure you correctly setup the
<a href="#sysadmin_package_repositories">package repositories</a> and only
continue with the actual upgrade if <span class="monospaced">apt update</span> did not hit any error.</td>
</tr></tbody></table>
</div>
</li>
<li>
<p><em>
How can I upgrade Proxmox VE to the next major release?
</em></p>
<p>
Major version upgrades, for example going from Proxmox VE 4.4 to 5.0, are also
supported.
They must be carefully planned and tested and should <strong>never</strong> be started without
having a current backup ready.
</p>
<div class="paragraph" id="faq-upgrade-major">
<p>Although the specific upgrade steps depend on your respective setup, we provide
general instructions and advice of how a upgrade should be performed:</p></div>
<div class="ulist"><ul>
<li>
<p>
<a href="https://pve.proxmox.com/wiki/Upgrade_from_7_to_8">Upgrade from Proxmox VE 7 to 8</a>
</p>
</li>
<li>
<p>
<a href="https://pve.proxmox.com/wiki/Upgrade_from_6.x_to_7.0">Upgrade from Proxmox VE 6 to 7</a>
</p>
</li>
<li>
<p>
<a href="https://pve.proxmox.com/wiki/Upgrade_from_5.x_to_6.0">Upgrade from Proxmox VE 5 to 6</a>
</p>
</li>
<li>
<p>
<a href="https://pve.proxmox.com/wiki/Upgrade_from_4.x_to_5.0">Upgrade from Proxmox VE 4 to 5</a>
</p>
</li>
<li>
<p>
<a href="https://pve.proxmox.com/wiki/Upgrade_from_3.x_to_4.0">Upgrade from Proxmox VE 3 to 4</a>
</p>
</li>
</ul></div>
</li>
<li>
<p><em>
LXC vs LXD vs Proxmox Containers vs Docker
</em></p>
<p>
LXC is a userspace interface for the Linux kernel containment
features. Through a powerful API and simple tools, it lets Linux users
easily create and manage system containers. LXC, as well as the former
OpenVZ, aims at <strong>system virtualization</strong>. Thus, it allows you to run a
complete OS inside a container, where you log in using ssh, add users,
run apache, etc…
</p>
<div class="paragraph">
<p>LXD is built on top of LXC to provide a new, better user
experience. Under the hood, LXD uses LXC through <span class="monospaced">liblxc</span> and its Go
binding to create and manage the containers. It’s basically an
alternative to LXC’s tools and distribution template system with the
added features that come from being controllable over the network.</p></div>
<div class="paragraph">
<p>Proxmox Containers are how we refer to containers that are created and managed
using the Proxmox Container Toolkit (<span class="monospaced">pct</span>). They also target <strong>system
virtualization</strong> and use LXC as the basis of the container offering. The
Proxmox Container Toolkit (<span class="monospaced">pct</span>) is tightly coupled with Proxmox VE. This means
that it is aware of cluster setups, and it can use the same network
and storage resources as QEMU virtual machines (VMs). You can even use the
Proxmox VE firewall, create and restore backups, or manage containers using
the HA framework. Everything can be controlled over the network using
the Proxmox VE API.</p></div>
<div class="paragraph">
<p>Docker aims at running a <strong>single</strong> application in an isolated, self-contained
environment. These are generally referred to as “Application Containers”, rather
than “System Containers”. You manage a Docker instance from the host, using the
Docker Engine command-line interface. It is not recommended to run docker
directly on your Proxmox VE host.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">If you want to run application containers, for example, <em>Docker</em> images, it
is best to run them inside a Proxmox QEMU VM.</td>
</tr></tbody></table>
</div>
</li>
</ol></div>
</div>
</div>
<div class="sect1">
<h2 id="_bibliography">
21. Bibliography
 <a class="headerlink" href="#_bibliography" title="Permalink to this heading"></a>
</h2>
<div class="sectionbody">
<div class="ulist bibliography"><div class="title">Books about Proxmox VE</div><ul>
<li>
<p>
<a id="Ahmed16"></a>[Ahmed16] Wasim Ahmed.
  <em>Mastering Proxmox - Third Edition</em>.
  Packt Publishing, 2017.
  ISBN 978-1788397605
</p>
</li>
<li>
<p>
<a id="Ahmed15"></a>[Ahmed15] Wasim Ahmed.
  <em>Proxmox Cookbook</em>.
  Packt Publishing, 2015.
  ISBN 978-1783980901
</p>
</li>
<li>
<p>
<a id="Cheng14"></a>[Cheng14] Simon M.C. Cheng.
  <em>Proxmox High Availability</em>.
  Packt Publishing, 2014.
  ISBN 978-1783980888
</p>
</li>
<li>
<p>
<a id="Goldman16"></a>[Goldman16] Rik Goldman.
  <em>Learning Proxmox VE</em>.
  Packt Publishing, 2016.
  ISBN 978-1783981786
</p>
</li>
<li>
<p>
<a id="Surber16"></a>[Surber16]] Lee R. Surber.
  <em>Virtualization Complete: Business Basic Edition</em>.
  Linux Solutions (LRS-TEK), 2016.
  ASIN B01BBVQZT6
</p>
</li>
</ul></div>
<div class="ulist bibliography"><div class="title">Books about related technology</div><ul>
<li>
<p>
<a id="Hertzog13"></a>[Hertzog13] Raphaël Hertzog, Roland Mas., Freexian SARL
  <a href="https://debian-handbook.info/get">The Debian Administrator's Handbook: Debian Bullseye from Discovery to Mastery</a>,
  Freexian, 2021.
  ISBN 979-10-91414-20-3
</p>
</li>
<li>
<p>
<a id="Bir96"></a>[Bir96] Kenneth P. Birman.
  <em>Building Secure and Reliable Network Applications</em>.
  Manning Publications Co, 1996.
  ISBN 978-1884777295
</p>
</li>
<li>
<p>
<a id="Walsh10"></a>[Walsh10] Norman Walsh.
  <em>DocBook 5: The Definitive Guide</em>.
  O’Reilly &amp; Associates, 2010.
  ISBN 978-0596805029
</p>
</li>
<li>
<p>
<a id="Richardson07"></a>[Richardson07] Leonard Richardson &amp; Sam Ruby.
  <em>RESTful Web Services</em>.
  O’Reilly Media, 2007.
  ISBN 978-0596529260
</p>
</li>
<li>
<p>
<a id="Singh15"></a>[Singh15] Karan Singh.
  <em>Learning Ceph</em>.
  Packt Publishing, 2015.
  ISBN 978-1783985623
</p>
</li>
<li>
<p>
<a id="Singh16"></a>[Singh16] Karan Signh.
  <em>Ceph Cookbook</em>
  Packt Publishing, 2016.
  ISBN 978-1784393502
</p>
</li>
<li>
<p>
<a id="Mauerer08"></a>[Mauerer08] Wolfgang Mauerer.
  <em>Professional Linux Kernel Architecture</em>.
  John Wiley &amp; Sons, 2008.
  ISBN 978-0470343432
</p>
</li>
<li>
<p>
<a id="Loshin03"></a>[Loshin03] Pete Loshin,
  <em>IPv6: Theory, Protocol, and Practice, 2nd Edition</em>.
  Morgan Kaufmann, 2003.
  ISBN 978-1558608108
</p>
</li>
<li>
<p>
<a id="Loeliger12"></a>[Loeliger12] Jon Loeliger &amp; Matthew McCullough.
  <em>Version Control with Git: Powerful tools and techniques for
   collaborative software development</em>.
  O’Reilly and Associates, 2012.
  ISBN 978-1449316389
</p>
</li>
<li>
<p>
<a id="Kreibich10"></a>[Kreibich10] Jay A. Kreibich.
  <em>Using SQLite</em>,
  O’Reilly and Associates, 2010.
  ISBN 978-0596521189
</p>
</li>
</ul></div>
<div class="ulist bibliography"><div class="title">Books about related topics</div><ul>
<li>
<p>
<a id="Bessen09"></a>[Bessen09] James Bessen &amp; Michael J. Meurer,
  <em>Patent Failure: How Judges, Bureaucrats, and Lawyers Put Innovators at Risk</em>.
  Princeton Univ Press, 2009.
  ISBN 978-0691143217
</p>
</li>
</ul></div>
</div>
</div>
<div class="sect1">
<h2 id="_command_line_interface">22. Appendix A: Command-line Interface</h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="_output_format_options_span_class_monospaced_format_options_span">
<span>22.1. Output format options <span class="monospaced">[FORMAT_OPTIONS]</span></span>
 <a class="headerlink" href="#_output_format_options_span_class_monospaced_format_options_span" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>It is possible to specify the output format using the
<span class="monospaced">--output-format</span> parameter. The default format <em>text</em> uses ASCII-art
to draw nice borders around tables. It additionally transforms some
values into human-readable text, for example:</p></div>
<div class="ulist"><ul>
<li>
<p>
Unix epoch is displayed as ISO 8601 date string.
</p>
</li>
<li>
<p>
Durations are displayed as week/day/hour/minute/second count, i.e <span class="monospaced">1d 5h</span>.
</p>
</li>
<li>
<p>
Byte sizes value include units (<span class="monospaced">B</span>, <span class="monospaced">KiB</span>, <span class="monospaced">MiB</span>, <span class="monospaced">GiB</span>, <span class="monospaced">TiB</span>, <span class="monospaced">PiB</span>).
</p>
</li>
<li>
<p>
Fractions are display as percentage, i.e. 1.0 is displayed as 100%.
</p>
</li>
</ul></div>
<div class="paragraph">
<p>You can also completely suppress output using option <span class="monospaced">--quiet</span>.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--human-readable</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Call output rendering functions to produce human readable text.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--noborder</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Do not draw borders (for <em>text</em> format).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--noheader</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Do not show column headers (for <em>text</em> format).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--output-format</span> <span class="monospaced">&lt;json | json-pretty | text | yaml&gt;</span> (<em>default =</em> <span class="monospaced">text</span>)
</dt>
<dd>
<p>
Output format.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--quiet</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Suppress printing results.
</p>
</dd>
</dl></div>
</div>
<div class="sect2">
<h3 id="_strong_pvesm_strong_proxmox_ve_storage_manager">
<span>22.2. <strong>pvesm</strong> - Proxmox VE Storage Manager</span>
 <a class="headerlink" href="#_strong_pvesm_strong_proxmox_ve_storage_manager" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p><strong>pvesm</strong> <span class="monospaced">&lt;COMMAND&gt; [ARGS] [OPTIONS]</span></p></div>
<div class="paragraph">
<p><strong>pvesm add</strong> <span class="monospaced">&lt;type&gt; &lt;storage&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Create a new storage.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;type&gt;</span>: <span class="monospaced">&lt;btrfs | cephfs | cifs | dir | esxi | glusterfs | iscsi | iscsidirect | lvm | lvmthin | nfs | pbs | rbd | zfs | zfspool&gt;</span> 
</dt>
<dd>
<p>
Storage type.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;storage&gt;</span>: <span class="monospaced">&lt;storage ID&gt;</span> 
</dt>
<dd>
<p>
The storage identifier.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--authsupported</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Authsupported.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--base</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Base volume. This volume is automatically activated.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--blocksize</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
block size
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--bwlimit</span> <span class="monospaced">[clone=&lt;LIMIT&gt;] [,default=&lt;LIMIT&gt;] [,migration=&lt;LIMIT&gt;] [,move=&lt;LIMIT&gt;] [,restore=&lt;LIMIT&gt;]</span> 
</dt>
<dd>
<p>
Set I/O bandwidth limit for various operations (in KiB/s).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--comstar_hg</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
host group for comstar views
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--comstar_tg</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
target group for comstar views
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--content</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Allowed content types.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">the value <em>rootdir</em> is used for Containers, and value <em>images</em> for VMs.</td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">--content-dirs</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Overrides for default content type directories.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--create-base-path</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">yes</span>)
</dt>
<dd>
<p>
Create the base directory if it doesn’t exist.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--create-subdirs</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">yes</span>)
</dt>
<dd>
<p>
Populate the directory with the default structure.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--data-pool</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Data Pool (for erasure coding only)
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--datastore</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Proxmox Backup Server datastore name.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--disable</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Flag to disable the storage.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--domain</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
CIFS domain.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--encryption-key</span> <span class="monospaced">a file containing an encryption key, or the special value "autogen"</span> 
</dt>
<dd>
<p>
Encryption key. Use <em>autogen</em> to generate one automatically without passphrase.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--export</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
NFS export path.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--fingerprint</span> <span class="monospaced">([A-Fa-f0-9]{2}:){31}[A-Fa-f0-9]{2}</span> 
</dt>
<dd>
<p>
Certificate SHA 256 fingerprint.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--format</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Default image format.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--fs-name</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The Ceph filesystem name.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--fuse</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Mount CephFS through FUSE.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--is_mountpoint</span> <span class="monospaced">&lt;string&gt;</span> (<em>default =</em> <span class="monospaced">no</span>)
</dt>
<dd>
<p>
Assume the given path is an externally managed mountpoint and consider the storage offline if it is not mounted. Using a boolean (yes/no) value serves as a shortcut to using the target path in this field.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--iscsiprovider</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
iscsi provider
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--keyring</span> <span class="monospaced">file containing the keyring to authenticate in the Ceph cluster</span> 
</dt>
<dd>
<p>
Client keyring contents (for external clusters).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--krbd</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Always access rbd through krbd kernel module.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--lio_tpg</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
target portal group for Linux LIO targets
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--master-pubkey</span> <span class="monospaced">a file containing a PEM-formatted master public key</span> 
</dt>
<dd>
<p>
Base64-encoded, PEM-formatted public RSA key. Used to encrypt a copy of the encryption-key which will be added to each encrypted backup.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--max-protected-backups</span> <span class="monospaced">&lt;integer&gt; (-1 - N)</span> (<em>default =</em> <span class="monospaced">Unlimited for users with Datastore.Allocate privilege, 5 for other users</span>)
</dt>
<dd>
<p>
Maximal number of protected backups per guest. Use <em>-1</em> for unlimited.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--maxfiles</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> 
</dt>
<dd>
<p>
Deprecated: use <em>prune-backups</em> instead. Maximal number of backup files per VM. Use <em>0</em> for unlimited.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--mkdir</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">yes</span>)
</dt>
<dd>
<p>
Create the directory if it doesn’t exist and populate it with default sub-dirs. NOTE: Deprecated, use the <em>create-base-path</em> and <em>create-subdirs</em> options instead.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--monhost</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
IP addresses of monitors (for external clusters).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--mountpoint</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
mount point
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--namespace</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Namespace.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--nocow</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Set the NOCOW flag on files. Disables data checksumming and causes data errors to be unrecoverable from while allowing direct I/O. Only use this if data does not need to be any more safe than on a single ext4 formatted disk with no underlying raid system.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--nodes</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
List of nodes for which the storage configuration applies.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--nowritecache</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
disable write caching on the target
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--options</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
NFS/CIFS mount options (see <em>man nfs</em> or <em>man mount.cifs</em>)
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--password</span> <span class="monospaced">&lt;password&gt;</span> 
</dt>
<dd>
<p>
Password for accessing the share/datastore.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--path</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
File system path.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--pool</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Pool.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--port</span> <span class="monospaced">&lt;integer&gt; (1 - 65535)</span> 
</dt>
<dd>
<p>
For non default port.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--portal</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
iSCSI portal (IP or DNS name with optional port).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--preallocation</span> <span class="monospaced">&lt;falloc | full | metadata | off&gt;</span> (<em>default =</em> <span class="monospaced">metadata</span>)
</dt>
<dd>
<p>
Preallocation mode for raw and qcow2 images. Using <em>metadata</em> on raw images results in preallocation=off.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--prune-backups</span> <span class="monospaced">[keep-all=&lt;1|0&gt;] [,keep-daily=&lt;N&gt;] [,keep-hourly=&lt;N&gt;] [,keep-last=&lt;N&gt;] [,keep-monthly=&lt;N&gt;] [,keep-weekly=&lt;N&gt;] [,keep-yearly=&lt;N&gt;]</span> 
</dt>
<dd>
<p>
The retention options with shorter intervals are processed first with --keep-last being the very first one. Each option covers a specific period of time. We say that backups within this period are covered by this option. The next option does not take care of already covered backups and only considers older backups.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--saferemove</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Zero-out data when removing LVs.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--saferemove_throughput</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Wipe throughput (cstream -t parameter value).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--server</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Server IP or DNS name.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--server2</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Backup volfile server IP or DNS name.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Requires option(s): <span class="monospaced">server</span></td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">--share</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
CIFS share.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--shared</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Indicate that this is a single storage with the same contents on all nodes (or all listed in the <em>nodes</em> option). It will not make the contents of a local storage automatically accessible to other nodes, it just marks an already shared storage as such!
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--skip-cert-verification</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">false</span>)
</dt>
<dd>
<p>
Disable TLS certificate verification, only enable on fully trusted networks!
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--smbversion</span> <span class="monospaced">&lt;2.0 | 2.1 | 3 | 3.0 | 3.11 | default&gt;</span> (<em>default =</em> <span class="monospaced">default</span>)
</dt>
<dd>
<p>
SMB protocol version. <em>default</em> if not set, negotiates the highest SMB2+ version supported by both the client and server.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--sparse</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
use sparse volumes
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--subdir</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Subdir to mount.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--tagged_only</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Only use logical volumes tagged with <em>pve-vm-ID</em>.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--target</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
iSCSI target.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--thinpool</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
LVM thin pool LV name.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--transport</span> <span class="monospaced">&lt;rdma | tcp | unix&gt;</span> 
</dt>
<dd>
<p>
Gluster transport: tcp or rdma
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--username</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
RBD Id.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--vgname</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Volume group name.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--volume</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Glusterfs Volume.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesm alloc</strong> <span class="monospaced">&lt;storage&gt; &lt;vmid&gt; &lt;filename&gt; &lt;size&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Allocate disk images.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;storage&gt;</span>: <span class="monospaced">&lt;storage ID&gt;</span> 
</dt>
<dd>
<p>
The storage identifier.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
Specify owner VM
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;filename&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The name of the file to create.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;size&gt;</span>: <span class="monospaced">\d+[MG]?</span> 
</dt>
<dd>
<p>
Size in kilobyte (1024 bytes). Optional suffixes <em>M</em> (megabyte, 1024K) and <em>G</em> (gigabyte, 1024M)
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--format</span> <span class="monospaced">&lt;qcow2 | raw | subvol&gt;</span> 
</dt>
<dd>
<p>
no description available
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Requires option(s): <span class="monospaced">size</span></td>
</tr></tbody></table>
</div>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesm apiinfo</strong></p></div>
<div class="paragraph">
<p>Returns APIVER and APIAGE.</p></div>
<div class="paragraph">
<p><strong>pvesm cifsscan</strong></p></div>
<div class="paragraph">
<p>An alias for <em>pvesm scan cifs</em>.</p></div>
<div class="paragraph">
<p><strong>pvesm export</strong> <span class="monospaced">&lt;volume&gt; &lt;format&gt; &lt;filename&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Used internally to export a volume.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;volume&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Volume identifier
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;format&gt;</span>: <span class="monospaced">&lt;btrfs | qcow2+size | raw+size | tar+size | vmdk+size | zfs&gt;</span> 
</dt>
<dd>
<p>
Export stream format
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;filename&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Destination file name
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--base</span> <span class="monospaced">(?^i:[a-z0-9_\-]{1,40})</span> 
</dt>
<dd>
<p>
Snapshot to start an incremental stream from
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--snapshot</span> <span class="monospaced">(?^i:[a-z0-9_\-]{1,40})</span> 
</dt>
<dd>
<p>
Snapshot to export
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--snapshot-list</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Ordered list of snapshots to transfer
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--with-snapshots</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Whether to include intermediate snapshots in the stream
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesm extractconfig</strong> <span class="monospaced">&lt;volume&gt;</span></p></div>
<div class="paragraph">
<p>Extract configuration from vzdump backup archive.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;volume&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Volume identifier
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesm free</strong> <span class="monospaced">&lt;volume&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Delete volume</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;volume&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Volume identifier
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--delay</span> <span class="monospaced">&lt;integer&gt; (1 - 30)</span> 
</dt>
<dd>
<p>
Time to wait for the task to finish. We return <em>null</em> if the task finish within that time.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--storage</span> <span class="monospaced">&lt;storage ID&gt;</span> 
</dt>
<dd>
<p>
The storage identifier.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesm glusterfsscan</strong></p></div>
<div class="paragraph">
<p>An alias for <em>pvesm scan glusterfs</em>.</p></div>
<div class="paragraph">
<p><strong>pvesm help</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Get help about specified command.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--extra-args</span> <span class="monospaced">&lt;array&gt;</span> 
</dt>
<dd>
<p>
Shows help for a specific command
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--verbose</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Verbose output format.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesm import</strong> <span class="monospaced">&lt;volume&gt; &lt;format&gt; &lt;filename&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Used internally to import a volume.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;volume&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Volume identifier
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;format&gt;</span>: <span class="monospaced">&lt;btrfs | qcow2+size | raw+size | tar+size | vmdk+size | zfs&gt;</span> 
</dt>
<dd>
<p>
Import stream format
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;filename&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Source file name. For <em>-</em> stdin is used, the tcp://&lt;IP-or-CIDR&gt; format allows to use a TCP connection, the unix://PATH-TO-SOCKET format a UNIX socket as input.Else, the file is treated as common file.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--allow-rename</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Choose a new volume ID if the requested volume ID already exists, instead of throwing an error.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--base</span> <span class="monospaced">(?^i:[a-z0-9_\-]{1,40})</span> 
</dt>
<dd>
<p>
Base snapshot of an incremental stream
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--delete-snapshot</span> <span class="monospaced">(?^i:[a-z0-9_\-]{1,80})</span> 
</dt>
<dd>
<p>
A snapshot to delete on success
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--snapshot</span> <span class="monospaced">(?^i:[a-z0-9_\-]{1,40})</span> 
</dt>
<dd>
<p>
The current-state snapshot if the stream contains snapshots
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--with-snapshots</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Whether the stream includes intermediate snapshots
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesm iscsiscan</strong></p></div>
<div class="paragraph">
<p>An alias for <em>pvesm scan iscsi</em>.</p></div>
<div class="paragraph">
<p><strong>pvesm list</strong> <span class="monospaced">&lt;storage&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>List storage content.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;storage&gt;</span>: <span class="monospaced">&lt;storage ID&gt;</span> 
</dt>
<dd>
<p>
The storage identifier.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--content</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Only list content of this type.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--vmid</span> <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
Only list images for this VM
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesm lvmscan</strong></p></div>
<div class="paragraph">
<p>An alias for <em>pvesm scan lvm</em>.</p></div>
<div class="paragraph">
<p><strong>pvesm lvmthinscan</strong></p></div>
<div class="paragraph">
<p>An alias for <em>pvesm scan lvmthin</em>.</p></div>
<div class="paragraph">
<p><strong>pvesm nfsscan</strong></p></div>
<div class="paragraph">
<p>An alias for <em>pvesm scan nfs</em>.</p></div>
<div class="paragraph">
<p><strong>pvesm path</strong> <span class="monospaced">&lt;volume&gt;</span></p></div>
<div class="paragraph">
<p>Get filesystem path for specified volume</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;volume&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Volume identifier
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesm prune-backups</strong> <span class="monospaced">&lt;storage&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Prune backups. Only those using the standard naming scheme are considered.
If no keep options are specified, those from the storage configuration are
used.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;storage&gt;</span>: <span class="monospaced">&lt;storage ID&gt;</span> 
</dt>
<dd>
<p>
The storage identifier.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--dry-run</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Only show what would be pruned, don’t delete anything.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--keep-all</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Keep all backups. Conflicts with the other options when true.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--keep-daily</span> <span class="monospaced">&lt;N&gt;</span> 
</dt>
<dd>
<p>
Keep backups for the last &lt;N&gt; different days. If there is morethan one backup for a single day, only the latest one is kept.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--keep-hourly</span> <span class="monospaced">&lt;N&gt;</span> 
</dt>
<dd>
<p>
Keep backups for the last &lt;N&gt; different hours. If there is morethan one backup for a single hour, only the latest one is kept.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--keep-last</span> <span class="monospaced">&lt;N&gt;</span> 
</dt>
<dd>
<p>
Keep the last &lt;N&gt; backups.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--keep-monthly</span> <span class="monospaced">&lt;N&gt;</span> 
</dt>
<dd>
<p>
Keep backups for the last &lt;N&gt; different months. If there is morethan one backup for a single month, only the latest one is kept.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--keep-weekly</span> <span class="monospaced">&lt;N&gt;</span> 
</dt>
<dd>
<p>
Keep backups for the last &lt;N&gt; different weeks. If there is morethan one backup for a single week, only the latest one is kept.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--keep-yearly</span> <span class="monospaced">&lt;N&gt;</span> 
</dt>
<dd>
<p>
Keep backups for the last &lt;N&gt; different years. If there is morethan one backup for a single year, only the latest one is kept.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--type</span> <span class="monospaced">&lt;lxc | qemu&gt;</span> 
</dt>
<dd>
<p>
Either <em>qemu</em> or <em>lxc</em>. Only consider backups for guests of this type.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--vmid</span> <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
Only consider backups for this guest.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesm remove</strong> <span class="monospaced">&lt;storage&gt;</span></p></div>
<div class="paragraph">
<p>Delete storage configuration.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;storage&gt;</span>: <span class="monospaced">&lt;storage ID&gt;</span> 
</dt>
<dd>
<p>
The storage identifier.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesm scan cifs</strong> <span class="monospaced">&lt;server&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Scan remote CIFS server.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;server&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The server address (name or IP).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--domain</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
SMB domain (Workgroup).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--password</span> <span class="monospaced">&lt;password&gt;</span> 
</dt>
<dd>
<p>
User password.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--username</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
User name.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesm scan glusterfs</strong> <span class="monospaced">&lt;server&gt;</span></p></div>
<div class="paragraph">
<p>Scan remote GlusterFS server.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;server&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The server address (name or IP).
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesm scan iscsi</strong> <span class="monospaced">&lt;portal&gt;</span></p></div>
<div class="paragraph">
<p>Scan remote iSCSI server.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;portal&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The iSCSI portal (IP or DNS name with optional port).
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesm scan lvm</strong></p></div>
<div class="paragraph">
<p>List local LVM volume groups.</p></div>
<div class="paragraph">
<p><strong>pvesm scan lvmthin</strong> <span class="monospaced">&lt;vg&gt;</span></p></div>
<div class="paragraph">
<p>List local LVM Thin Pools.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vg&gt;</span>: <span class="monospaced">[a-zA-Z0-9\.\+\_][a-zA-Z0-9\.\+\_\-]+</span> 
</dt>
<dd>
<p>
no description available
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesm scan nfs</strong> <span class="monospaced">&lt;server&gt;</span></p></div>
<div class="paragraph">
<p>Scan remote NFS server.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;server&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The server address (name or IP).
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesm scan pbs</strong> <span class="monospaced">&lt;server&gt; &lt;username&gt; --password &lt;string&gt;</span> <span class="monospaced">[OPTIONS]</span> <span class="monospaced">[FORMAT_OPTIONS]</span></p></div>
<div class="paragraph">
<p>Scan remote Proxmox Backup Server.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;server&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The server address (name or IP).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;username&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
User-name or API token-ID.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--fingerprint</span> <span class="monospaced">([A-Fa-f0-9]{2}:){31}[A-Fa-f0-9]{2}</span> 
</dt>
<dd>
<p>
Certificate SHA 256 fingerprint.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--password</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
User password or API token secret.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--port</span> <span class="monospaced">&lt;integer&gt; (1 - 65535)</span> (<em>default =</em> <span class="monospaced">8007</span>)
</dt>
<dd>
<p>
Optional port.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesm scan zfs</strong></p></div>
<div class="paragraph">
<p>Scan zfs pool list on local node.</p></div>
<div class="paragraph">
<p><strong>pvesm set</strong> <span class="monospaced">&lt;storage&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Update storage configuration.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;storage&gt;</span>: <span class="monospaced">&lt;storage ID&gt;</span> 
</dt>
<dd>
<p>
The storage identifier.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--blocksize</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
block size
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--bwlimit</span> <span class="monospaced">[clone=&lt;LIMIT&gt;] [,default=&lt;LIMIT&gt;] [,migration=&lt;LIMIT&gt;] [,move=&lt;LIMIT&gt;] [,restore=&lt;LIMIT&gt;]</span> 
</dt>
<dd>
<p>
Set I/O bandwidth limit for various operations (in KiB/s).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--comstar_hg</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
host group for comstar views
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--comstar_tg</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
target group for comstar views
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--content</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Allowed content types.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">the value <em>rootdir</em> is used for Containers, and value <em>images</em> for VMs.</td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">--content-dirs</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Overrides for default content type directories.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--create-base-path</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">yes</span>)
</dt>
<dd>
<p>
Create the base directory if it doesn’t exist.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--create-subdirs</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">yes</span>)
</dt>
<dd>
<p>
Populate the directory with the default structure.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--data-pool</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Data Pool (for erasure coding only)
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--delete</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
A list of settings you want to delete.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--digest</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Prevent changes if current configuration file has a different digest. This can be used to prevent concurrent modifications.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--disable</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Flag to disable the storage.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--domain</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
CIFS domain.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--encryption-key</span> <span class="monospaced">a file containing an encryption key, or the special value "autogen"</span> 
</dt>
<dd>
<p>
Encryption key. Use <em>autogen</em> to generate one automatically without passphrase.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--fingerprint</span> <span class="monospaced">([A-Fa-f0-9]{2}:){31}[A-Fa-f0-9]{2}</span> 
</dt>
<dd>
<p>
Certificate SHA 256 fingerprint.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--format</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Default image format.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--fs-name</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The Ceph filesystem name.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--fuse</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Mount CephFS through FUSE.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--is_mountpoint</span> <span class="monospaced">&lt;string&gt;</span> (<em>default =</em> <span class="monospaced">no</span>)
</dt>
<dd>
<p>
Assume the given path is an externally managed mountpoint and consider the storage offline if it is not mounted. Using a boolean (yes/no) value serves as a shortcut to using the target path in this field.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--keyring</span> <span class="monospaced">file containing the keyring to authenticate in the Ceph cluster</span> 
</dt>
<dd>
<p>
Client keyring contents (for external clusters).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--krbd</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Always access rbd through krbd kernel module.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--lio_tpg</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
target portal group for Linux LIO targets
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--master-pubkey</span> <span class="monospaced">a file containing a PEM-formatted master public key</span> 
</dt>
<dd>
<p>
Base64-encoded, PEM-formatted public RSA key. Used to encrypt a copy of the encryption-key which will be added to each encrypted backup.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--max-protected-backups</span> <span class="monospaced">&lt;integer&gt; (-1 - N)</span> (<em>default =</em> <span class="monospaced">Unlimited for users with Datastore.Allocate privilege, 5 for other users</span>)
</dt>
<dd>
<p>
Maximal number of protected backups per guest. Use <em>-1</em> for unlimited.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--maxfiles</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> 
</dt>
<dd>
<p>
Deprecated: use <em>prune-backups</em> instead. Maximal number of backup files per VM. Use <em>0</em> for unlimited.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--mkdir</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">yes</span>)
</dt>
<dd>
<p>
Create the directory if it doesn’t exist and populate it with default sub-dirs. NOTE: Deprecated, use the <em>create-base-path</em> and <em>create-subdirs</em> options instead.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--monhost</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
IP addresses of monitors (for external clusters).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--mountpoint</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
mount point
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--namespace</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Namespace.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--nocow</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Set the NOCOW flag on files. Disables data checksumming and causes data errors to be unrecoverable from while allowing direct I/O. Only use this if data does not need to be any more safe than on a single ext4 formatted disk with no underlying raid system.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--nodes</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
List of nodes for which the storage configuration applies.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--nowritecache</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
disable write caching on the target
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--options</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
NFS/CIFS mount options (see <em>man nfs</em> or <em>man mount.cifs</em>)
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--password</span> <span class="monospaced">&lt;password&gt;</span> 
</dt>
<dd>
<p>
Password for accessing the share/datastore.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--pool</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Pool.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--port</span> <span class="monospaced">&lt;integer&gt; (1 - 65535)</span> 
</dt>
<dd>
<p>
For non default port.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--preallocation</span> <span class="monospaced">&lt;falloc | full | metadata | off&gt;</span> (<em>default =</em> <span class="monospaced">metadata</span>)
</dt>
<dd>
<p>
Preallocation mode for raw and qcow2 images. Using <em>metadata</em> on raw images results in preallocation=off.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--prune-backups</span> <span class="monospaced">[keep-all=&lt;1|0&gt;] [,keep-daily=&lt;N&gt;] [,keep-hourly=&lt;N&gt;] [,keep-last=&lt;N&gt;] [,keep-monthly=&lt;N&gt;] [,keep-weekly=&lt;N&gt;] [,keep-yearly=&lt;N&gt;]</span> 
</dt>
<dd>
<p>
The retention options with shorter intervals are processed first with --keep-last being the very first one. Each option covers a specific period of time. We say that backups within this period are covered by this option. The next option does not take care of already covered backups and only considers older backups.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--saferemove</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Zero-out data when removing LVs.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--saferemove_throughput</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Wipe throughput (cstream -t parameter value).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--server</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Server IP or DNS name.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--server2</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Backup volfile server IP or DNS name.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Requires option(s): <span class="monospaced">server</span></td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">--shared</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Indicate that this is a single storage with the same contents on all nodes (or all listed in the <em>nodes</em> option). It will not make the contents of a local storage automatically accessible to other nodes, it just marks an already shared storage as such!
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--skip-cert-verification</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">false</span>)
</dt>
<dd>
<p>
Disable TLS certificate verification, only enable on fully trusted networks!
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--smbversion</span> <span class="monospaced">&lt;2.0 | 2.1 | 3 | 3.0 | 3.11 | default&gt;</span> (<em>default =</em> <span class="monospaced">default</span>)
</dt>
<dd>
<p>
SMB protocol version. <em>default</em> if not set, negotiates the highest SMB2+ version supported by both the client and server.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--sparse</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
use sparse volumes
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--subdir</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Subdir to mount.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--tagged_only</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Only use logical volumes tagged with <em>pve-vm-ID</em>.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--transport</span> <span class="monospaced">&lt;rdma | tcp | unix&gt;</span> 
</dt>
<dd>
<p>
Gluster transport: tcp or rdma
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--username</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
RBD Id.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesm status</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Get status for all datastores.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--content</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Only list stores which support this content type.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--enabled</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Only list stores which are enabled (not disabled in config).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--format</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Include information about formats
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--storage</span> <span class="monospaced">&lt;storage ID&gt;</span> 
</dt>
<dd>
<p>
Only list status for  specified storage
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--target</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
If target is different to <em>node</em>, we only lists shared storages which content is accessible on this <em>node</em> and the specified <em>target</em> node.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesm zfsscan</strong></p></div>
<div class="paragraph">
<p>An alias for <em>pvesm scan zfs</em>.</p></div>
</div>
<div class="sect2">
<h3 id="_strong_pvesubscription_strong_proxmox_ve_subscription_manager">
<span>22.3. <strong>pvesubscription</strong> - Proxmox VE Subscription Manager</span>
 <a class="headerlink" href="#_strong_pvesubscription_strong_proxmox_ve_subscription_manager" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p><strong>pvesubscription</strong> <span class="monospaced">&lt;COMMAND&gt; [ARGS] [OPTIONS]</span></p></div>
<div class="paragraph">
<p><strong>pvesubscription delete</strong></p></div>
<div class="paragraph">
<p>Delete subscription key of this node.</p></div>
<div class="paragraph">
<p><strong>pvesubscription get</strong></p></div>
<div class="paragraph">
<p>Read subscription info.</p></div>
<div class="paragraph">
<p><strong>pvesubscription help</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Get help about specified command.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--extra-args</span> <span class="monospaced">&lt;array&gt;</span> 
</dt>
<dd>
<p>
Shows help for a specific command
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--verbose</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Verbose output format.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesubscription set</strong> <span class="monospaced">&lt;key&gt;</span></p></div>
<div class="paragraph">
<p>Set subscription key.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;key&gt;</span>: <span class="monospaced">\s*pve([1248])([cbsp])-[0-9a-f]{10}\s*</span> 
</dt>
<dd>
<p>
Proxmox VE subscription key
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesubscription set-offline-key</strong> <span class="monospaced">&lt;data&gt;</span></p></div>
<div class="paragraph">
<p>Internal use only! To set an offline key, use the package
proxmox-offline-mirror-helper instead.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;data&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
A signed subscription info blob
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesubscription update</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Update subscription info.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--force</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Always connect to server, even if local cache is still valid.
</p>
</dd>
</dl></div>
</div>
<div class="sect2">
<h3 id="_strong_pveperf_strong_proxmox_ve_benchmark_script">
<span>22.4. <strong>pveperf</strong> - Proxmox VE Benchmark Script</span>
 <a class="headerlink" href="#_strong_pveperf_strong_proxmox_ve_benchmark_script" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p><strong>pveperf</strong> <span class="monospaced">[PATH]</span></p></div>
</div>
<div class="sect2">
<h3 id="_strong_pveceph_strong_manage_ceph_services_on_proxmox_ve_nodes">
<span>22.5. <strong>pveceph</strong> - Manage CEPH Services on Proxmox VE Nodes</span>
 <a class="headerlink" href="#_strong_pveceph_strong_manage_ceph_services_on_proxmox_ve_nodes" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p><strong>pveceph</strong> <span class="monospaced">&lt;COMMAND&gt; [ARGS] [OPTIONS]</span></p></div>
<div class="paragraph">
<p><strong>pveceph createmgr</strong></p></div>
<div class="paragraph">
<p>An alias for <em>pveceph mgr create</em>.</p></div>
<div class="paragraph">
<p><strong>pveceph createmon</strong></p></div>
<div class="paragraph">
<p>An alias for <em>pveceph mon create</em>.</p></div>
<div class="paragraph">
<p><strong>pveceph createosd</strong></p></div>
<div class="paragraph">
<p>An alias for <em>pveceph osd create</em>.</p></div>
<div class="paragraph">
<p><strong>pveceph createpool</strong></p></div>
<div class="paragraph">
<p>An alias for <em>pveceph pool create</em>.</p></div>
<div class="paragraph">
<p><strong>pveceph destroymgr</strong></p></div>
<div class="paragraph">
<p>An alias for <em>pveceph mgr destroy</em>.</p></div>
<div class="paragraph">
<p><strong>pveceph destroymon</strong></p></div>
<div class="paragraph">
<p>An alias for <em>pveceph mon destroy</em>.</p></div>
<div class="paragraph">
<p><strong>pveceph destroyosd</strong></p></div>
<div class="paragraph">
<p>An alias for <em>pveceph osd destroy</em>.</p></div>
<div class="paragraph">
<p><strong>pveceph destroypool</strong></p></div>
<div class="paragraph">
<p>An alias for <em>pveceph pool destroy</em>.</p></div>
<div class="paragraph">
<p><strong>pveceph fs create</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Create a Ceph filesystem</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--add-storage</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Configure the created CephFS as storage for this cluster.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--name</span> <span class="monospaced">&lt;string&gt;</span> (<em>default =</em> <span class="monospaced">cephfs</span>)
</dt>
<dd>
<p>
The ceph filesystem name.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--pg_num</span> <span class="monospaced">&lt;integer&gt; (8 - 32768)</span> (<em>default =</em> <span class="monospaced">128</span>)
</dt>
<dd>
<p>
Number of placement groups for the backing data pool. The metadata pool will use a quarter of this.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveceph fs destroy</strong> <span class="monospaced">&lt;name&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Destroy a Ceph filesystem</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;name&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The ceph filesystem name.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--remove-pools</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Remove data and metadata pools configured for this fs.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--remove-storages</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Remove all pveceph-managed storages configured for this fs.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveceph help</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Get help about specified command.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--extra-args</span> <span class="monospaced">&lt;array&gt;</span> 
</dt>
<dd>
<p>
Shows help for a specific command
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--verbose</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Verbose output format.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveceph init</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Create initial ceph default configuration and setup symlinks.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--cluster-network</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Declare a separate cluster network, OSDs will routeheartbeat, object replication and recovery traffic over it
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Requires option(s): <span class="monospaced">network</span></td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">--disable_cephx</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Disable cephx authentication.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content">cephx is a security feature protecting against man-in-the-middle attacks. Only consider disabling cephx if your network is private!</td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">--min_size</span> <span class="monospaced">&lt;integer&gt; (1 - 7)</span> (<em>default =</em> <span class="monospaced">2</span>)
</dt>
<dd>
<p>
Minimum number of available replicas per object to allow I/O
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--network</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Use specific network for all ceph related traffic
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--pg_bits</span> <span class="monospaced">&lt;integer&gt; (6 - 14)</span> (<em>default =</em> <span class="monospaced">6</span>)
</dt>
<dd>
<p>
Placement group bits, used to specify the default number of placement groups.
</p>
<div class="paragraph">
<p>Depreacted. This setting was deprecated in recent Ceph versions.</p></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">--size</span> <span class="monospaced">&lt;integer&gt; (1 - 7)</span> (<em>default =</em> <span class="monospaced">3</span>)
</dt>
<dd>
<p>
Targeted number of replicas per object
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveceph install</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Install ceph related packages.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--allow-experimental</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Allow experimental versions. Use with care!
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--repository</span> <span class="monospaced">&lt;enterprise | no-subscription | test&gt;</span> (<em>default =</em> <span class="monospaced">enterprise</span>)
</dt>
<dd>
<p>
Ceph repository to use.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--version</span> <span class="monospaced">&lt;quincy | reef&gt;</span> (<em>default =</em> <span class="monospaced">quincy</span>)
</dt>
<dd>
<p>
Ceph version to install.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveceph lspools</strong></p></div>
<div class="paragraph">
<p>An alias for <em>pveceph pool ls</em>.</p></div>
<div class="paragraph">
<p><strong>pveceph mds create</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Create Ceph Metadata Server (MDS)</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--hotstandby</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Determines whether a ceph-mds daemon should poll and replay the log of an active MDS. Faster switch on MDS failure, but needs more idle resources.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--name</span> <span class="monospaced">[a-zA-Z0-9]([a-zA-Z0-9\-]*[a-zA-Z0-9])?</span> (<em>default =</em> <span class="monospaced">nodename</span>)
</dt>
<dd>
<p>
The ID for the mds, when omitted the same as the nodename
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveceph mds destroy</strong> <span class="monospaced">&lt;name&gt;</span></p></div>
<div class="paragraph">
<p>Destroy Ceph Metadata Server</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;name&gt;</span>: <span class="monospaced">[a-zA-Z0-9]([a-zA-Z0-9\-]*[a-zA-Z0-9])?</span> 
</dt>
<dd>
<p>
The name (ID) of the mds
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveceph mgr create</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Create Ceph Manager</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--id</span> <span class="monospaced">[a-zA-Z0-9]([a-zA-Z0-9\-]*[a-zA-Z0-9])?</span> 
</dt>
<dd>
<p>
The ID for the manager, when omitted the same as the nodename
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveceph mgr destroy</strong> <span class="monospaced">&lt;id&gt;</span></p></div>
<div class="paragraph">
<p>Destroy Ceph Manager.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;id&gt;</span>: <span class="monospaced">[a-zA-Z0-9]([a-zA-Z0-9\-]*[a-zA-Z0-9])?</span> 
</dt>
<dd>
<p>
The ID of the manager
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveceph mon create</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Create Ceph Monitor and Manager</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--mon-address</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Overwrites autodetected monitor IP address(es). Must be in the public network(s) of Ceph.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--monid</span> <span class="monospaced">[a-zA-Z0-9]([a-zA-Z0-9\-]*[a-zA-Z0-9])?</span> 
</dt>
<dd>
<p>
The ID for the monitor, when omitted the same as the nodename
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveceph mon destroy</strong> <span class="monospaced">&lt;monid&gt;</span></p></div>
<div class="paragraph">
<p>Destroy Ceph Monitor and Manager.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;monid&gt;</span>: <span class="monospaced">[a-zA-Z0-9]([a-zA-Z0-9\-]*[a-zA-Z0-9])?</span> 
</dt>
<dd>
<p>
Monitor ID
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveceph osd create</strong> <span class="monospaced">&lt;dev&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Create OSD</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;dev&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Block device name.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--crush-device-class</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Set the device class of the OSD in crush.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--db_dev</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Block device name for block.db.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--db_dev_size</span> <span class="monospaced">&lt;number&gt; (1 - N)</span> (<em>default =</em> <span class="monospaced">bluestore_block_db_size or 10% of OSD size</span>)
</dt>
<dd>
<p>
Size in GiB for block.db.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Requires option(s): <span class="monospaced">db_dev</span></td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">--encrypted</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Enables encryption of the OSD.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--osds-per-device</span> <span class="monospaced">&lt;integer&gt; (1 - N)</span> 
</dt>
<dd>
<p>
OSD services per physical device. Only useful for fast NVMe devices"
                    ." to utilize their performance better.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--wal_dev</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Block device name for block.wal.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--wal_dev_size</span> <span class="monospaced">&lt;number&gt; (0.5 - N)</span> (<em>default =</em> <span class="monospaced">bluestore_block_wal_size or 1% of OSD size</span>)
</dt>
<dd>
<p>
Size in GiB for block.wal.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Requires option(s): <span class="monospaced">wal_dev</span></td>
</tr></tbody></table>
</div>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveceph osd destroy</strong> <span class="monospaced">&lt;osdid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Destroy OSD</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;osdid&gt;</span>: <span class="monospaced">&lt;integer&gt;</span> 
</dt>
<dd>
<p>
OSD ID
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--cleanup</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
If set, we remove partition table entries.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveceph osd details</strong> <span class="monospaced">&lt;osdid&gt;</span> <span class="monospaced">[OPTIONS]</span> <span class="monospaced">[FORMAT_OPTIONS]</span></p></div>
<div class="paragraph">
<p>Get OSD details.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;osdid&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
ID of the OSD
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--verbose</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Print verbose information, same as json-pretty output format.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveceph pool create</strong> <span class="monospaced">&lt;name&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Create Ceph pool</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;name&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The name of the pool. It must be unique.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--add_storages</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0; for erasure coded pools: 1</span>)
</dt>
<dd>
<p>
Configure VM and CT storage using the new pool.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--application</span> <span class="monospaced">&lt;cephfs | rbd | rgw&gt;</span> (<em>default =</em> <span class="monospaced">rbd</span>)
</dt>
<dd>
<p>
The application of the pool.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--crush_rule</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The rule to use for mapping object placement in the cluster.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--erasure-coding</span> <span class="monospaced">k=&lt;integer&gt; ,m=&lt;integer&gt; [,device-class=&lt;class&gt;] [,failure-domain=&lt;domain&gt;] [,profile=&lt;profile&gt;]</span> 
</dt>
<dd>
<p>
Create an erasure coded pool for RBD with an accompaning replicated pool for metadata storage. With EC, the common ceph options <em>size</em>, <em>min_size</em> and <em>crush_rule</em> parameters will be applied to the metadata pool.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--min_size</span> <span class="monospaced">&lt;integer&gt; (1 - 7)</span> (<em>default =</em> <span class="monospaced">2</span>)
</dt>
<dd>
<p>
Minimum number of replicas per object
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--pg_autoscale_mode</span> <span class="monospaced">&lt;off | on | warn&gt;</span> (<em>default =</em> <span class="monospaced">warn</span>)
</dt>
<dd>
<p>
The automatic PG scaling mode of the pool.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--pg_num</span> <span class="monospaced">&lt;integer&gt; (1 - 32768)</span> (<em>default =</em> <span class="monospaced">128</span>)
</dt>
<dd>
<p>
Number of placement groups.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--pg_num_min</span> <span class="monospaced">&lt;integer&gt; (-N - 32768)</span> 
</dt>
<dd>
<p>
Minimal number of placement groups.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--size</span> <span class="monospaced">&lt;integer&gt; (1 - 7)</span> (<em>default =</em> <span class="monospaced">3</span>)
</dt>
<dd>
<p>
Number of replicas per object
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--target_size</span> <span class="monospaced">^(\d+(\.\d+)?)([KMGT])?$</span> 
</dt>
<dd>
<p>
The estimated target size of the pool for the PG autoscaler.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--target_size_ratio</span> <span class="monospaced">&lt;number&gt;</span> 
</dt>
<dd>
<p>
The estimated target ratio of the pool for the PG autoscaler.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveceph pool destroy</strong> <span class="monospaced">&lt;name&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Destroy pool</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;name&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The name of the pool. It must be unique.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--force</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
If true, destroys pool even if in use
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--remove_ecprofile</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Remove the erasure code profile. Defaults to true, if applicable.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--remove_storages</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Remove all pveceph-managed storages configured for this pool
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveceph pool get</strong> <span class="monospaced">&lt;name&gt;</span> <span class="monospaced">[OPTIONS]</span> <span class="monospaced">[FORMAT_OPTIONS]</span></p></div>
<div class="paragraph">
<p>Show the current pool status.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;name&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The name of the pool. It must be unique.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--verbose</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
If enabled, will display additional data(eg. statistics).
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveceph pool ls</strong> <span class="monospaced">[FORMAT_OPTIONS]</span></p></div>
<div class="paragraph">
<p>List all pools and their settings (which are settable by the POST/PUT
endpoints).</p></div>
<div class="paragraph">
<p><strong>pveceph pool set</strong> <span class="monospaced">&lt;name&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Change POOL settings</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;name&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The name of the pool. It must be unique.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--application</span> <span class="monospaced">&lt;cephfs | rbd | rgw&gt;</span> 
</dt>
<dd>
<p>
The application of the pool.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--crush_rule</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The rule to use for mapping object placement in the cluster.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--min_size</span> <span class="monospaced">&lt;integer&gt; (1 - 7)</span> 
</dt>
<dd>
<p>
Minimum number of replicas per object
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--pg_autoscale_mode</span> <span class="monospaced">&lt;off | on | warn&gt;</span> 
</dt>
<dd>
<p>
The automatic PG scaling mode of the pool.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--pg_num</span> <span class="monospaced">&lt;integer&gt; (1 - 32768)</span> 
</dt>
<dd>
<p>
Number of placement groups.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--pg_num_min</span> <span class="monospaced">&lt;integer&gt; (-N - 32768)</span> 
</dt>
<dd>
<p>
Minimal number of placement groups.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--size</span> <span class="monospaced">&lt;integer&gt; (1 - 7)</span> 
</dt>
<dd>
<p>
Number of replicas per object
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--target_size</span> <span class="monospaced">^(\d+(\.\d+)?)([KMGT])?$</span> 
</dt>
<dd>
<p>
The estimated target size of the pool for the PG autoscaler.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--target_size_ratio</span> <span class="monospaced">&lt;number&gt;</span> 
</dt>
<dd>
<p>
The estimated target ratio of the pool for the PG autoscaler.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveceph purge</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Destroy ceph related data and configuration files.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--crash</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Additionally purge Ceph crash logs, /var/lib/ceph/crash.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--logs</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Additionally purge Ceph logs, /var/log/ceph.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveceph start</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Start ceph services.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--service</span> <span class="monospaced">(ceph|mon|mds|osd|mgr)(\.[a-zA-Z0-9]([a-zA-Z0-9\-]*[a-zA-Z0-9])?)?</span> (<em>default =</em> <span class="monospaced">ceph.target</span>)
</dt>
<dd>
<p>
Ceph service name.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveceph status</strong></p></div>
<div class="paragraph">
<p>Get Ceph Status.</p></div>
<div class="paragraph">
<p><strong>pveceph stop</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Stop ceph services.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--service</span> <span class="monospaced">(ceph|mon|mds|osd|mgr)(\.[a-zA-Z0-9]([a-zA-Z0-9\-]*[a-zA-Z0-9])?)?</span> (<em>default =</em> <span class="monospaced">ceph.target</span>)
</dt>
<dd>
<p>
Ceph service name.
</p>
</dd>
</dl></div>
</div>
<div class="sect2">
<h3 id="_strong_pvenode_strong_proxmox_ve_node_management">
<span>22.6. <strong>pvenode</strong> - Proxmox VE Node Management</span>
 <a class="headerlink" href="#_strong_pvenode_strong_proxmox_ve_node_management" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p><strong>pvenode</strong> <span class="monospaced">&lt;COMMAND&gt; [ARGS] [OPTIONS]</span></p></div>
<div class="paragraph">
<p><strong>pvenode acme account deactivate</strong> <span class="monospaced">[&lt;name&gt;]</span></p></div>
<div class="paragraph">
<p>Deactivate existing ACME account at CA.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;name&gt;</span>: <span class="monospaced">&lt;name&gt;</span> (<em>default =</em> <span class="monospaced">default</span>)
</dt>
<dd>
<p>
ACME account config file name.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvenode acme account info</strong> <span class="monospaced">[&lt;name&gt;]</span> <span class="monospaced">[FORMAT_OPTIONS]</span></p></div>
<div class="paragraph">
<p>Return existing ACME account information.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;name&gt;</span>: <span class="monospaced">&lt;name&gt;</span> (<em>default =</em> <span class="monospaced">default</span>)
</dt>
<dd>
<p>
ACME account config file name.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvenode acme account list</strong></p></div>
<div class="paragraph">
<p>ACMEAccount index.</p></div>
<div class="paragraph">
<p><strong>pvenode acme account register</strong> <span class="monospaced">[&lt;name&gt;] {&lt;contact&gt;}</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Register a new ACME account with a compatible CA.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;name&gt;</span>: <span class="monospaced">&lt;name&gt;</span> (<em>default =</em> <span class="monospaced">default</span>)
</dt>
<dd>
<p>
ACME account config file name.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;contact&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Contact email addresses.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--directory</span> <span class="monospaced">^https?://.*</span> 
</dt>
<dd>
<p>
URL of ACME CA directory endpoint.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvenode acme account update</strong> <span class="monospaced">[&lt;name&gt;]</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Update existing ACME account information with CA. Note: not specifying any
new account information triggers a refresh.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;name&gt;</span>: <span class="monospaced">&lt;name&gt;</span> (<em>default =</em> <span class="monospaced">default</span>)
</dt>
<dd>
<p>
ACME account config file name.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--contact</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Contact email addresses.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvenode acme cert order</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Order a new certificate from ACME-compatible CA.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--force</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Overwrite existing custom certificate.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvenode acme cert renew</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Renew existing certificate from CA.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--force</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Force renewal even if expiry is more than 30 days away.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvenode acme cert revoke</strong></p></div>
<div class="paragraph">
<p>Revoke existing certificate from CA.</p></div>
<div class="paragraph">
<p><strong>pvenode acme plugin add</strong> <span class="monospaced">&lt;type&gt; &lt;id&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Add ACME plugin configuration.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;type&gt;</span>: <span class="monospaced">&lt;dns | standalone&gt;</span> 
</dt>
<dd>
<p>
ACME challenge type.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;id&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
ACME Plugin ID name
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--api</span> <span class="monospaced">&lt;1984hosting | acmedns | acmeproxy | active24 | ad | ali | anx | artfiles | arvan | aurora | autodns | aws | azion | azure | bookmyname | bunny | cf | clouddns | cloudns | cn | conoha | constellix | cpanel | curanet | cyon | da | ddnss | desec | df | dgon | dnsexit | dnshome | dnsimple | dnsservices | do | doapi | domeneshop | dp | dpi | dreamhost | duckdns | durabledns | dyn | dynu | dynv6 | easydns | edgedns | euserv | exoscale | fornex | freedns | gandi_livedns | gcloud | gcore | gd | geoscaling | googledomains | he | hetzner | hexonet | hostingde | huaweicloud | infoblox | infomaniak | internetbs | inwx | ionos | ipv64 | ispconfig | jd | joker | kappernet | kas | kinghost | knot | la | leaseweb | lexicon | linode | linode_v4 | loopia | lua | maradns | me | miab | misaka | myapi | mydevil | mydnsjp | mythic_beasts | namecheap | namecom | namesilo | nanelo | nederhost | neodigit | netcup | netlify | nic | njalla | nm | nsd | nsone | nsupdate | nw | oci | one | online | openprovider | openstack | opnsense | ovh | pdns | pleskxml | pointhq | porkbun | rackcorp | rackspace | rage4 | rcode0 | regru | scaleway | schlundtech | selectel | selfhost | servercow | simply | tele3 | tencent | transip | udr | ultra | unoeuro | variomedia | veesp | vercel | vscale | vultr | websupport | world4you | yandex | yc | zilore | zone | zonomi&gt;</span> 
</dt>
<dd>
<p>
API plugin name
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--data</span> <span class="monospaced">File with one key-value pair per line, will be base64url encode for storage in plugin config.</span> 
</dt>
<dd>
<p>
DNS plugin data. (base64 encoded)
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--disable</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Flag to disable the config.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--nodes</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
List of cluster node names.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--validation-delay</span> <span class="monospaced">&lt;integer&gt; (0 - 172800)</span> (<em>default =</em> <span class="monospaced">30</span>)
</dt>
<dd>
<p>
Extra delay in seconds to wait before requesting validation. Allows to cope with a long TTL of DNS records.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvenode acme plugin config</strong> <span class="monospaced">&lt;id&gt;</span> <span class="monospaced">[FORMAT_OPTIONS]</span></p></div>
<div class="paragraph">
<p>Get ACME plugin configuration.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;id&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Unique identifier for ACME plugin instance.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvenode acme plugin list</strong> <span class="monospaced">[OPTIONS]</span> <span class="monospaced">[FORMAT_OPTIONS]</span></p></div>
<div class="paragraph">
<p>ACME plugin index.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--type</span> <span class="monospaced">&lt;dns | standalone&gt;</span> 
</dt>
<dd>
<p>
Only list ACME plugins of a specific type
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvenode acme plugin remove</strong> <span class="monospaced">&lt;id&gt;</span></p></div>
<div class="paragraph">
<p>Delete ACME plugin configuration.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;id&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Unique identifier for ACME plugin instance.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvenode acme plugin set</strong> <span class="monospaced">&lt;id&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Update ACME plugin configuration.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;id&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
ACME Plugin ID name
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--api</span> <span class="monospaced">&lt;1984hosting | acmedns | acmeproxy | active24 | ad | ali | anx | artfiles | arvan | aurora | autodns | aws | azion | azure | bookmyname | bunny | cf | clouddns | cloudns | cn | conoha | constellix | cpanel | curanet | cyon | da | ddnss | desec | df | dgon | dnsexit | dnshome | dnsimple | dnsservices | do | doapi | domeneshop | dp | dpi | dreamhost | duckdns | durabledns | dyn | dynu | dynv6 | easydns | edgedns | euserv | exoscale | fornex | freedns | gandi_livedns | gcloud | gcore | gd | geoscaling | googledomains | he | hetzner | hexonet | hostingde | huaweicloud | infoblox | infomaniak | internetbs | inwx | ionos | ipv64 | ispconfig | jd | joker | kappernet | kas | kinghost | knot | la | leaseweb | lexicon | linode | linode_v4 | loopia | lua | maradns | me | miab | misaka | myapi | mydevil | mydnsjp | mythic_beasts | namecheap | namecom | namesilo | nanelo | nederhost | neodigit | netcup | netlify | nic | njalla | nm | nsd | nsone | nsupdate | nw | oci | one | online | openprovider | openstack | opnsense | ovh | pdns | pleskxml | pointhq | porkbun | rackcorp | rackspace | rage4 | rcode0 | regru | scaleway | schlundtech | selectel | selfhost | servercow | simply | tele3 | tencent | transip | udr | ultra | unoeuro | variomedia | veesp | vercel | vscale | vultr | websupport | world4you | yandex | yc | zilore | zone | zonomi&gt;</span> 
</dt>
<dd>
<p>
API plugin name
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--data</span> <span class="monospaced">File with one key-value pair per line, will be base64url encode for storage in plugin config.</span> 
</dt>
<dd>
<p>
DNS plugin data. (base64 encoded)
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--delete</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
A list of settings you want to delete.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--digest</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Prevent changes if current configuration file has a different digest. This can be used to prevent concurrent modifications.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--disable</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Flag to disable the config.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--nodes</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
List of cluster node names.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--validation-delay</span> <span class="monospaced">&lt;integer&gt; (0 - 172800)</span> (<em>default =</em> <span class="monospaced">30</span>)
</dt>
<dd>
<p>
Extra delay in seconds to wait before requesting validation. Allows to cope with a long TTL of DNS records.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvenode cert delete</strong> <span class="monospaced">[&lt;restart&gt;]</span></p></div>
<div class="paragraph">
<p>DELETE custom certificate chain and key.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;restart&gt;</span>: <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Restart pveproxy.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvenode cert info</strong> <span class="monospaced">[FORMAT_OPTIONS]</span></p></div>
<div class="paragraph">
<p>Get information about node’s certificates.</p></div>
<div class="paragraph">
<p><strong>pvenode cert set</strong> <span class="monospaced">&lt;certificates&gt; [&lt;key&gt;]</span> <span class="monospaced">[OPTIONS]</span> <span class="monospaced">[FORMAT_OPTIONS]</span></p></div>
<div class="paragraph">
<p>Upload or update custom certificate chain and key.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;certificates&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
PEM encoded certificate (chain).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;key&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
PEM encoded private key.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--force</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Overwrite existing custom or ACME certificate files.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--restart</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Restart pveproxy.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvenode config get</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Get node configuration options.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--property</span> <span class="monospaced">&lt;acme | acmedomain0 | acmedomain1 | acmedomain2 | acmedomain3 | acmedomain4 | acmedomain5 | description | startall-onboot-delay | wakeonlan&gt;</span> (<em>default =</em> <span class="monospaced">all</span>)
</dt>
<dd>
<p>
Return only a specific property from the node configuration.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvenode config set</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Set node configuration options.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--acme</span> <span class="monospaced">[account=&lt;name&gt;] [,domains=&lt;domain[;domain;...]&gt;]</span> 
</dt>
<dd>
<p>
Node specific ACME settings.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--acmedomain[n]</span> <span class="monospaced">[domain=]&lt;domain&gt; [,alias=&lt;domain&gt;] [,plugin=&lt;name of the plugin configuration&gt;]</span> 
</dt>
<dd>
<p>
ACME domain and validation plugin
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--delete</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
A list of settings you want to delete.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--description</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Description for the Node. Shown in the web-interface node notes panel. This is saved as comment inside the configuration file.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--digest</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Prevent changes if current configuration file has different SHA1 digest. This can be used to prevent concurrent modifications.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--startall-onboot-delay</span> <span class="monospaced">&lt;integer&gt; (0 - 300)</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Initial delay in seconds, before starting all the Virtual Guests with on-boot enabled.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--wakeonlan</span> <span class="monospaced">[mac=]&lt;MAC address&gt; [,bind-interface=&lt;bind interface&gt;] [,broadcast-address=&lt;IPv4 broadcast address&gt;]</span> 
</dt>
<dd>
<p>
Node specific wake on LAN settings.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvenode help</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Get help about specified command.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--extra-args</span> <span class="monospaced">&lt;array&gt;</span> 
</dt>
<dd>
<p>
Shows help for a specific command
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--verbose</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Verbose output format.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvenode migrateall</strong> <span class="monospaced">&lt;target&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Migrate all VMs and Containers.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;target&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Target node.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--maxworkers</span> <span class="monospaced">&lt;integer&gt; (1 - N)</span> 
</dt>
<dd>
<p>
Maximal number of parallel migration job. If not set, uses’max_workers' from datacenter.cfg. One of both must be set!
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--vms</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Only consider Guests with these IDs.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--with-local-disks</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Enable live storage migration for local disk
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvenode startall</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Start all VMs and containers located on this node (by default only those
with onboot=1).</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--force</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">off</span>)
</dt>
<dd>
<p>
Issue start command even if virtual guest have <em>onboot</em> not set or set to off.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--vms</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Only consider guests from this comma separated list of VMIDs.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvenode stopall</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Stop all VMs and Containers.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--force-stop</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Force a hard-stop after the timeout.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--timeout</span> <span class="monospaced">&lt;integer&gt; (0 - 7200)</span> (<em>default =</em> <span class="monospaced">180</span>)
</dt>
<dd>
<p>
Timeout for each guest shutdown task. Depending on <span class="monospaced">force-stop</span>, the shutdown gets then simply aborted or a hard-stop is forced.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--vms</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Only consider Guests with these IDs.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvenode task list</strong> <span class="monospaced">[OPTIONS]</span> <span class="monospaced">[FORMAT_OPTIONS]</span></p></div>
<div class="paragraph">
<p>Read task list for one node (finished tasks).</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--errors</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Only list tasks with a status of ERROR.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--limit</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">50</span>)
</dt>
<dd>
<p>
Only list this amount of tasks.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--since</span> <span class="monospaced">&lt;integer&gt;</span> 
</dt>
<dd>
<p>
Only list tasks since this UNIX epoch.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--source</span> <span class="monospaced">&lt;active | all | archive&gt;</span> (<em>default =</em> <span class="monospaced">archive</span>)
</dt>
<dd>
<p>
List archived, active or all tasks.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--start</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
List tasks beginning from this offset.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--statusfilter</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
List of Task States that should be returned.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--typefilter</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Only list tasks of this type (e.g., vzstart, vzdump).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--until</span> <span class="monospaced">&lt;integer&gt;</span> 
</dt>
<dd>
<p>
Only list tasks until this UNIX epoch.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--userfilter</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Only list tasks from this user.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--vmid</span> <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
Only list tasks for this VM.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvenode task log</strong> <span class="monospaced">&lt;upid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Read task log.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;upid&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The task’s unique ID.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--download</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Whether the tasklog file should be downloaded. This parameter can’t be used in conjunction with other parameters
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--start</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Start at this line when reading the tasklog
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvenode task status</strong> <span class="monospaced">&lt;upid&gt;</span> <span class="monospaced">[FORMAT_OPTIONS]</span></p></div>
<div class="paragraph">
<p>Read task status.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;upid&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The task’s unique ID.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvenode wakeonlan</strong> <span class="monospaced">&lt;node&gt;</span></p></div>
<div class="paragraph">
<p>Try to wake a node via <em>wake on LAN</em> network packet.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;node&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
target node for wake on LAN packet
</p>
</dd>
</dl></div>
</div>
<div class="sect2">
<h3 id="_strong_pvesh_strong_shell_interface_for_the_proxmox_ve_api">
<span>22.7. <strong>pvesh</strong> - Shell interface for the Proxmox VE API</span>
 <a class="headerlink" href="#_strong_pvesh_strong_shell_interface_for_the_proxmox_ve_api" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p><strong>pvesh</strong> <span class="monospaced">&lt;COMMAND&gt; [ARGS] [OPTIONS]</span></p></div>
<div class="paragraph">
<p><strong>pvesh create</strong> <span class="monospaced">&lt;api_path&gt;</span> <span class="monospaced">[OPTIONS]</span> <span class="monospaced">[FORMAT_OPTIONS]</span></p></div>
<div class="paragraph">
<p>Call API POST on &lt;api_path&gt;.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;api_path&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
API path.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--noproxy</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Disable automatic proxying.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesh delete</strong> <span class="monospaced">&lt;api_path&gt;</span> <span class="monospaced">[OPTIONS]</span> <span class="monospaced">[FORMAT_OPTIONS]</span></p></div>
<div class="paragraph">
<p>Call API DELETE on &lt;api_path&gt;.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;api_path&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
API path.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--noproxy</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Disable automatic proxying.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesh get</strong> <span class="monospaced">&lt;api_path&gt;</span> <span class="monospaced">[OPTIONS]</span> <span class="monospaced">[FORMAT_OPTIONS]</span></p></div>
<div class="paragraph">
<p>Call API GET on &lt;api_path&gt;.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;api_path&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
API path.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--noproxy</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Disable automatic proxying.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesh help</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Get help about specified command.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--extra-args</span> <span class="monospaced">&lt;array&gt;</span> 
</dt>
<dd>
<p>
Shows help for a specific command
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--verbose</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Verbose output format.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesh ls</strong> <span class="monospaced">&lt;api_path&gt;</span> <span class="monospaced">[OPTIONS]</span> <span class="monospaced">[FORMAT_OPTIONS]</span></p></div>
<div class="paragraph">
<p>List child objects on &lt;api_path&gt;.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;api_path&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
API path.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--noproxy</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Disable automatic proxying.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesh set</strong> <span class="monospaced">&lt;api_path&gt;</span> <span class="monospaced">[OPTIONS]</span> <span class="monospaced">[FORMAT_OPTIONS]</span></p></div>
<div class="paragraph">
<p>Call API PUT on &lt;api_path&gt;.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;api_path&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
API path.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--noproxy</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Disable automatic proxying.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesh usage</strong> <span class="monospaced">&lt;api_path&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>print API usage information for &lt;api_path&gt;.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;api_path&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
API path.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--command</span> <span class="monospaced">&lt;create | delete | get | set&gt;</span> 
</dt>
<dd>
<p>
API command.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--returns</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Including schema for returned data.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--verbose</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Verbose output format.
</p>
</dd>
</dl></div>
</div>
<div class="sect2">
<h3 id="_strong_qm_strong_qemu_kvm_virtual_machine_manager">
<span>22.8. <strong>qm</strong> - QEMU/KVM Virtual Machine Manager</span>
 <a class="headerlink" href="#_strong_qm_strong_qemu_kvm_virtual_machine_manager" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p><strong>qm</strong> <span class="monospaced">&lt;COMMAND&gt; [ARGS] [OPTIONS]</span></p></div>
<div class="paragraph">
<p><strong>qm agent</strong></p></div>
<div class="paragraph">
<p>An alias for <em>qm guest cmd</em>.</p></div>
<div class="paragraph">
<p><strong>qm cleanup</strong> <span class="monospaced">&lt;vmid&gt; &lt;clean-shutdown&gt; &lt;guest-requested&gt;</span></p></div>
<div class="paragraph">
<p>Cleans up resources like tap devices, vgpus, etc. Called after a vm shuts
down, crashes, etc.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;clean-shutdown&gt;</span>: <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Indicates if qemu shutdown cleanly.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;guest-requested&gt;</span>: <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Indicates if the shutdown was requested by the guest or via qmp.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm clone</strong> <span class="monospaced">&lt;vmid&gt; &lt;newid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Create a copy of virtual machine/template.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;newid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
VMID for the clone.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--bwlimit</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">clone limit from datacenter or storage config</span>)
</dt>
<dd>
<p>
Override I/O bandwidth limit (in KiB/s).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--description</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Description for the new VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--format</span> <span class="monospaced">&lt;qcow2 | raw | vmdk&gt;</span> 
</dt>
<dd>
<p>
Target format for file storage. Only valid for full clone.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--full</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Create a full copy of all disks. This is always done when you clone a normal VM. For VM templates, we try to create a linked clone by default.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--name</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Set a name for the new VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--pool</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Add the new VM to the specified pool.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--snapname</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The name of the snapshot.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--storage</span> <span class="monospaced">&lt;storage ID&gt;</span> 
</dt>
<dd>
<p>
Target storage for full clone.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--target</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Target node. Only allowed if the original VM is on shared storage.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm cloudinit dump</strong> <span class="monospaced">&lt;vmid&gt; &lt;type&gt;</span></p></div>
<div class="paragraph">
<p>Get automatically generated cloudinit config.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;type&gt;</span>: <span class="monospaced">&lt;meta | network | user&gt;</span> 
</dt>
<dd>
<p>
Config type.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm cloudinit pending</strong> <span class="monospaced">&lt;vmid&gt;</span></p></div>
<div class="paragraph">
<p>Get the cloudinit configuration with both current and pending values.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm cloudinit update</strong> <span class="monospaced">&lt;vmid&gt;</span></p></div>
<div class="paragraph">
<p>Regenerate and change cloudinit config drive.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm config</strong> <span class="monospaced">&lt;vmid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Get the virtual machine configuration with pending configuration changes
applied. Set the <em>current</em> parameter to get the current configuration
instead.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--current</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Get current values (instead of pending values).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--snapshot</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Fetch config values from given snapshot.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm create</strong> <span class="monospaced">&lt;vmid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Create or restore a virtual machine.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--acpi</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Enable/disable ACPI.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--affinity</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
List of host cores used to execute guest processes, for example: 0,5,8-11
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--agent</span> <span class="monospaced">[enabled=]&lt;1|0&gt; [,freeze-fs-on-backup=&lt;1|0&gt;] [,fstrim_cloned_disks=&lt;1|0&gt;] [,type=&lt;virtio|isa&gt;]</span> 
</dt>
<dd>
<p>
Enable/disable communication with the QEMU Guest Agent and its properties.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--arch</span> <span class="monospaced">&lt;aarch64 | x86_64&gt;</span> 
</dt>
<dd>
<p>
Virtual processor architecture. Defaults to the host.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--archive</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The backup archive. Either the file system path to a .tar or .vma file (use <em>-</em> to pipe data from stdin) or a proxmox storage backup volume identifier.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--args</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Arbitrary arguments passed to kvm.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--audio0</span> <span class="monospaced">device=&lt;ich9-intel-hda|intel-hda|AC97&gt; [,driver=&lt;spice|none&gt;]</span> 
</dt>
<dd>
<p>
Configure a audio device, useful in combination with QXL/Spice.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--autostart</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Automatic restart after crash (currently ignored).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--balloon</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> 
</dt>
<dd>
<p>
Amount of target RAM for the VM in MiB. Using zero disables the ballon driver.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--bios</span> <span class="monospaced">&lt;ovmf | seabios&gt;</span> (<em>default =</em> <span class="monospaced">seabios</span>)
</dt>
<dd>
<p>
Select BIOS implementation.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--boot</span> <span class="monospaced">[[legacy=]&lt;[acdn]{1,4}&gt;] [,order=&lt;device[;device...]&gt;]</span> 
</dt>
<dd>
<p>
Specify guest boot order. Use the <em>order=</em> sub-property as usage with no key or <em>legacy=</em> is deprecated.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--bootdisk</span> <span class="monospaced">(ide|sata|scsi|virtio)\d+</span> 
</dt>
<dd>
<p>
Enable booting from specified disk. Deprecated: Use <em>boot: order=foo;bar</em> instead.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--bwlimit</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">restore limit from datacenter or storage config</span>)
</dt>
<dd>
<p>
Override I/O bandwidth limit (in KiB/s).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--cdrom</span> <span class="monospaced">&lt;volume&gt;</span> 
</dt>
<dd>
<p>
This is an alias for option -ide2
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--cicustom</span> <span class="monospaced">[meta=&lt;volume&gt;] [,network=&lt;volume&gt;] [,user=&lt;volume&gt;] [,vendor=&lt;volume&gt;]</span> 
</dt>
<dd>
<p>
cloud-init: Specify custom files to replace the automatically generated ones at start.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--cipassword</span> <span class="monospaced">&lt;password&gt;</span> 
</dt>
<dd>
<p>
cloud-init: Password to assign the user. Using this is generally not recommended. Use ssh keys instead. Also note that older cloud-init versions do not support hashed passwords.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--citype</span> <span class="monospaced">&lt;configdrive2 | nocloud | opennebula&gt;</span> 
</dt>
<dd>
<p>
Specifies the cloud-init configuration format. The default depends on the configured operating system type (<span class="monospaced">ostype</span>. We use the <span class="monospaced">nocloud</span> format for Linux, and <span class="monospaced">configdrive2</span> for windows.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--ciupgrade</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
cloud-init: do an automatic package upgrade after the first boot.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--ciuser</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
cloud-init: User name to change ssh keys and password for instead of the image’s configured default user.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--cores</span> <span class="monospaced">&lt;integer&gt; (1 - N)</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
The number of cores per socket.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--cpu</span> <span class="monospaced">[[cputype=]&lt;string&gt;] [,flags=&lt;+FLAG[;-FLAG...]&gt;] [,hidden=&lt;1|0&gt;] [,hv-vendor-id=&lt;vendor-id&gt;] [,phys-bits=&lt;8-64|host&gt;] [,reported-model=&lt;enum&gt;]</span> 
</dt>
<dd>
<p>
Emulated CPU type.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--cpulimit</span> <span class="monospaced">&lt;number&gt; (0 - 128)</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Limit of CPU usage.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--cpuunits</span> <span class="monospaced">&lt;integer&gt; (1 - 262144)</span> (<em>default =</em> <span class="monospaced">cgroup v1: 1024, cgroup v2: 100</span>)
</dt>
<dd>
<p>
CPU weight for a VM, will be clamped to [1, 10000] in cgroup v2.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--description</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Description for the VM. Shown in the web-interface VM’s summary. This is saved as comment inside the configuration file.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--efidisk0</span> <span class="monospaced">[file=]&lt;volume&gt; [,efitype=&lt;2m|4m&gt;] [,format=&lt;enum&gt;] [,import-from=&lt;source volume&gt;] [,pre-enrolled-keys=&lt;1|0&gt;] [,size=&lt;DiskSize&gt;]</span> 
</dt>
<dd>
<p>
Configure a disk for storing EFI vars. Use the special syntax STORAGE_ID:SIZE_IN_GiB to allocate a new volume. Note that SIZE_IN_GiB is ignored here and that the default EFI vars are copied to the volume instead. Use STORAGE_ID:0 and the <em>import-from</em> parameter to import from an existing volume.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--force</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Allow to overwrite existing VM.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Requires option(s): <span class="monospaced">archive</span></td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">--freeze</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Freeze CPU at startup (use <em>c</em> monitor command to start execution).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--hookscript</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Script that will be executed during various steps in the vms lifetime.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--hostpci[n]</span> <span class="monospaced">[[host=]&lt;HOSTPCIID[;HOSTPCIID2...]&gt;] [,device-id=&lt;hex id&gt;] [,legacy-igd=&lt;1|0&gt;] [,mapping=&lt;mapping-id&gt;] [,mdev=&lt;string&gt;] [,pcie=&lt;1|0&gt;] [,rombar=&lt;1|0&gt;] [,romfile=&lt;string&gt;] [,sub-device-id=&lt;hex id&gt;] [,sub-vendor-id=&lt;hex id&gt;] [,vendor-id=&lt;hex id&gt;] [,x-vga=&lt;1|0&gt;]</span> 
</dt>
<dd>
<p>
Map host PCI devices into guest.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--hotplug</span> <span class="monospaced">&lt;string&gt;</span> (<em>default =</em> <span class="monospaced">network,disk,usb</span>)
</dt>
<dd>
<p>
Selectively enable hotplug features. This is a comma separated list of hotplug features: <em>network</em>, <em>disk</em>, <em>cpu</em>, <em>memory</em>, <em>usb</em> and <em>cloudinit</em>. Use <em>0</em> to disable hotplug completely. Using <em>1</em> as value is an alias for the default <span class="monospaced">network,disk,usb</span>. USB hotplugging is possible for guests with machine version &gt;= 7.1 and ostype l26 or windows &gt; 7.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--hugepages</span> <span class="monospaced">&lt;1024 | 2 | any&gt;</span> 
</dt>
<dd>
<p>
Enable/disable hugepages memory.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--ide[n]</span> <span class="monospaced">[file=]&lt;volume&gt; [,aio=&lt;native|threads|io_uring&gt;] [,backup=&lt;1|0&gt;] [,bps=&lt;bps&gt;] [,bps_max_length=&lt;seconds&gt;] [,bps_rd=&lt;bps&gt;] [,bps_rd_max_length=&lt;seconds&gt;] [,bps_wr=&lt;bps&gt;] [,bps_wr_max_length=&lt;seconds&gt;] [,cache=&lt;enum&gt;] [,cyls=&lt;integer&gt;] [,detect_zeroes=&lt;1|0&gt;] [,discard=&lt;ignore|on&gt;] [,format=&lt;enum&gt;] [,heads=&lt;integer&gt;] [,import-from=&lt;source volume&gt;] [,iops=&lt;iops&gt;] [,iops_max=&lt;iops&gt;] [,iops_max_length=&lt;seconds&gt;] [,iops_rd=&lt;iops&gt;] [,iops_rd_max=&lt;iops&gt;] [,iops_rd_max_length=&lt;seconds&gt;] [,iops_wr=&lt;iops&gt;] [,iops_wr_max=&lt;iops&gt;] [,iops_wr_max_length=&lt;seconds&gt;] [,mbps=&lt;mbps&gt;] [,mbps_max=&lt;mbps&gt;] [,mbps_rd=&lt;mbps&gt;] [,mbps_rd_max=&lt;mbps&gt;] [,mbps_wr=&lt;mbps&gt;] [,mbps_wr_max=&lt;mbps&gt;] [,media=&lt;cdrom|disk&gt;] [,model=&lt;model&gt;] [,replicate=&lt;1|0&gt;] [,rerror=&lt;ignore|report|stop&gt;] [,secs=&lt;integer&gt;] [,serial=&lt;serial&gt;] [,shared=&lt;1|0&gt;] [,size=&lt;DiskSize&gt;] [,snapshot=&lt;1|0&gt;] [,ssd=&lt;1|0&gt;] [,trans=&lt;none|lba|auto&gt;] [,werror=&lt;enum&gt;] [,wwn=&lt;wwn&gt;]</span> 
</dt>
<dd>
<p>
Use volume as IDE hard disk or CD-ROM (n is 0 to 3). Use the special syntax STORAGE_ID:SIZE_IN_GiB to allocate a new volume. Use STORAGE_ID:0 and the <em>import-from</em> parameter to import from an existing volume.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--ipconfig[n]</span> <span class="monospaced">[gw=&lt;GatewayIPv4&gt;] [,gw6=&lt;GatewayIPv6&gt;] [,ip=&lt;IPv4Format/CIDR&gt;] [,ip6=&lt;IPv6Format/CIDR&gt;]</span> 
</dt>
<dd>
<p>
cloud-init: Specify IP addresses and gateways for the corresponding interface.
</p>
<div class="paragraph">
<p>IP addresses use CIDR notation, gateways are optional but need an IP of the same type specified.</p></div>
<div class="paragraph">
<p>The special string <em>dhcp</em> can be used for IP addresses to use DHCP, in which case no explicit
gateway should be provided.
For IPv6 the special string <em>auto</em> can be used to use stateless autoconfiguration. This requires
cloud-init 19.4 or newer.</p></div>
<div class="paragraph">
<p>If cloud-init is enabled and neither an IPv4 nor an IPv6 address is specified, it defaults to using
dhcp on IPv4.</p></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">--ivshmem</span> <span class="monospaced">size=&lt;integer&gt; [,name=&lt;string&gt;]</span> 
</dt>
<dd>
<p>
Inter-VM shared memory. Useful for direct communication between VMs, or to the host.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--keephugepages</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Use together with hugepages. If enabled, hugepages will not not be deleted after VM shutdown and can be used for subsequent starts.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--keyboard</span> <span class="monospaced">&lt;da | de | de-ch | en-gb | en-us | es | fi | fr | fr-be | fr-ca | fr-ch | hu | is | it | ja | lt | mk | nl | no | pl | pt | pt-br | sl | sv | tr&gt;</span> 
</dt>
<dd>
<p>
Keyboard layout for VNC server. This option is generally not required and is often better handled from within the guest OS.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--kvm</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Enable/disable KVM hardware virtualization.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--live-restore</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Start the VM immediately while importing or restoring in the background.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--localtime</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Set the real time clock (RTC) to local time. This is enabled by default if the <span class="monospaced">ostype</span> indicates a Microsoft Windows OS.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--lock</span> <span class="monospaced">&lt;backup | clone | create | migrate | rollback | snapshot | snapshot-delete | suspended | suspending&gt;</span> 
</dt>
<dd>
<p>
Lock/unlock the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--machine</span> <span class="monospaced">[[type=]&lt;machine type&gt;] [,viommu=&lt;intel|virtio&gt;]</span> 
</dt>
<dd>
<p>
Specify the QEMU machine.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--memory</span> <span class="monospaced">[current=]&lt;integer&gt;</span> 
</dt>
<dd>
<p>
Memory properties.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--migrate_downtime</span> <span class="monospaced">&lt;number&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">0.1</span>)
</dt>
<dd>
<p>
Set maximum tolerated downtime (in seconds) for migrations. Should the migration not be able to converge in the very end, because too much newly dirtied RAM needs to be transferred, the limit will be increased automatically step-by-step until migration can converge.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--migrate_speed</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Set maximum speed (in MB/s) for migrations. Value 0 is no limit.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--name</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Set a name for the VM. Only used on the configuration web interface.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--nameserver</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
cloud-init: Sets DNS server IP address for a container. Create will automatically use the setting from the host if neither searchdomain nor nameserver are set.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--net[n]</span> <span class="monospaced">[model=]&lt;enum&gt; [,bridge=&lt;bridge&gt;] [,firewall=&lt;1|0&gt;] [,link_down=&lt;1|0&gt;] [,macaddr=&lt;XX:XX:XX:XX:XX:XX&gt;] [,mtu=&lt;integer&gt;] [,queues=&lt;integer&gt;] [,rate=&lt;number&gt;] [,tag=&lt;integer&gt;] [,trunks=&lt;vlanid[;vlanid...]&gt;] [,&lt;model&gt;=&lt;macaddr&gt;]</span> 
</dt>
<dd>
<p>
Specify network devices.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--numa</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Enable/disable NUMA.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--numa[n]</span> <span class="monospaced">cpus=&lt;id[-id];...&gt; [,hostnodes=&lt;id[-id];...&gt;] [,memory=&lt;number&gt;] [,policy=&lt;preferred|bind|interleave&gt;]</span> 
</dt>
<dd>
<p>
NUMA topology.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--onboot</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Specifies whether a VM will be started during system bootup.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--ostype</span> <span class="monospaced">&lt;l24 | l26 | other | solaris | w2k | w2k3 | w2k8 | win10 | win11 | win7 | win8 | wvista | wxp&gt;</span> 
</dt>
<dd>
<p>
Specify guest operating system.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--parallel[n]</span> <span class="monospaced">/dev/parport\d+|/dev/usb/lp\d+</span> 
</dt>
<dd>
<p>
Map host parallel devices (n is 0 to 2).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--pool</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Add the VM to the specified pool.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--protection</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Sets the protection flag of the VM. This will disable the remove VM and remove disk operations.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--reboot</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Allow reboot. If set to <em>0</em> the VM exit on reboot.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--rng0</span> <span class="monospaced">[source=]&lt;/dev/urandom|/dev/random|/dev/hwrng&gt; [,max_bytes=&lt;integer&gt;] [,period=&lt;integer&gt;]</span> 
</dt>
<dd>
<p>
Configure a VirtIO-based Random Number Generator.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--sata[n]</span> <span class="monospaced">[file=]&lt;volume&gt; [,aio=&lt;native|threads|io_uring&gt;] [,backup=&lt;1|0&gt;] [,bps=&lt;bps&gt;] [,bps_max_length=&lt;seconds&gt;] [,bps_rd=&lt;bps&gt;] [,bps_rd_max_length=&lt;seconds&gt;] [,bps_wr=&lt;bps&gt;] [,bps_wr_max_length=&lt;seconds&gt;] [,cache=&lt;enum&gt;] [,cyls=&lt;integer&gt;] [,detect_zeroes=&lt;1|0&gt;] [,discard=&lt;ignore|on&gt;] [,format=&lt;enum&gt;] [,heads=&lt;integer&gt;] [,import-from=&lt;source volume&gt;] [,iops=&lt;iops&gt;] [,iops_max=&lt;iops&gt;] [,iops_max_length=&lt;seconds&gt;] [,iops_rd=&lt;iops&gt;] [,iops_rd_max=&lt;iops&gt;] [,iops_rd_max_length=&lt;seconds&gt;] [,iops_wr=&lt;iops&gt;] [,iops_wr_max=&lt;iops&gt;] [,iops_wr_max_length=&lt;seconds&gt;] [,mbps=&lt;mbps&gt;] [,mbps_max=&lt;mbps&gt;] [,mbps_rd=&lt;mbps&gt;] [,mbps_rd_max=&lt;mbps&gt;] [,mbps_wr=&lt;mbps&gt;] [,mbps_wr_max=&lt;mbps&gt;] [,media=&lt;cdrom|disk&gt;] [,replicate=&lt;1|0&gt;] [,rerror=&lt;ignore|report|stop&gt;] [,secs=&lt;integer&gt;] [,serial=&lt;serial&gt;] [,shared=&lt;1|0&gt;] [,size=&lt;DiskSize&gt;] [,snapshot=&lt;1|0&gt;] [,ssd=&lt;1|0&gt;] [,trans=&lt;none|lba|auto&gt;] [,werror=&lt;enum&gt;] [,wwn=&lt;wwn&gt;]</span> 
</dt>
<dd>
<p>
Use volume as SATA hard disk or CD-ROM (n is 0 to 5). Use the special syntax STORAGE_ID:SIZE_IN_GiB to allocate a new volume. Use STORAGE_ID:0 and the <em>import-from</em> parameter to import from an existing volume.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--scsi[n]</span> <span class="monospaced">[file=]&lt;volume&gt; [,aio=&lt;native|threads|io_uring&gt;] [,backup=&lt;1|0&gt;] [,bps=&lt;bps&gt;] [,bps_max_length=&lt;seconds&gt;] [,bps_rd=&lt;bps&gt;] [,bps_rd_max_length=&lt;seconds&gt;] [,bps_wr=&lt;bps&gt;] [,bps_wr_max_length=&lt;seconds&gt;] [,cache=&lt;enum&gt;] [,cyls=&lt;integer&gt;] [,detect_zeroes=&lt;1|0&gt;] [,discard=&lt;ignore|on&gt;] [,format=&lt;enum&gt;] [,heads=&lt;integer&gt;] [,import-from=&lt;source volume&gt;] [,iops=&lt;iops&gt;] [,iops_max=&lt;iops&gt;] [,iops_max_length=&lt;seconds&gt;] [,iops_rd=&lt;iops&gt;] [,iops_rd_max=&lt;iops&gt;] [,iops_rd_max_length=&lt;seconds&gt;] [,iops_wr=&lt;iops&gt;] [,iops_wr_max=&lt;iops&gt;] [,iops_wr_max_length=&lt;seconds&gt;] [,iothread=&lt;1|0&gt;] [,mbps=&lt;mbps&gt;] [,mbps_max=&lt;mbps&gt;] [,mbps_rd=&lt;mbps&gt;] [,mbps_rd_max=&lt;mbps&gt;] [,mbps_wr=&lt;mbps&gt;] [,mbps_wr_max=&lt;mbps&gt;] [,media=&lt;cdrom|disk&gt;] [,product=&lt;product&gt;] [,queues=&lt;integer&gt;] [,replicate=&lt;1|0&gt;] [,rerror=&lt;ignore|report|stop&gt;] [,ro=&lt;1|0&gt;] [,scsiblock=&lt;1|0&gt;] [,secs=&lt;integer&gt;] [,serial=&lt;serial&gt;] [,shared=&lt;1|0&gt;] [,size=&lt;DiskSize&gt;] [,snapshot=&lt;1|0&gt;] [,ssd=&lt;1|0&gt;] [,trans=&lt;none|lba|auto&gt;] [,vendor=&lt;vendor&gt;] [,werror=&lt;enum&gt;] [,wwn=&lt;wwn&gt;]</span> 
</dt>
<dd>
<p>
Use volume as SCSI hard disk or CD-ROM (n is 0 to 30). Use the special syntax STORAGE_ID:SIZE_IN_GiB to allocate a new volume. Use STORAGE_ID:0 and the <em>import-from</em> parameter to import from an existing volume.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--scsihw</span> <span class="monospaced">&lt;lsi | lsi53c810 | megasas | pvscsi | virtio-scsi-pci | virtio-scsi-single&gt;</span> (<em>default =</em> <span class="monospaced">lsi</span>)
</dt>
<dd>
<p>
SCSI controller model
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--searchdomain</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
cloud-init: Sets DNS search domains for a container. Create will automatically use the setting from the host if neither searchdomain nor nameserver are set.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--serial[n]</span> <span class="monospaced">(/dev/.+|socket)</span> 
</dt>
<dd>
<p>
Create a serial device inside the VM (n is 0 to 3)
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--shares</span> <span class="monospaced">&lt;integer&gt; (0 - 50000)</span> (<em>default =</em> <span class="monospaced">1000</span>)
</dt>
<dd>
<p>
Amount of memory shares for auto-ballooning. The larger the number is, the more memory this VM gets. Number is relative to weights of all other running VMs. Using zero disables auto-ballooning. Auto-ballooning is done by pvestatd.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--smbios1</span> <span class="monospaced">[base64=&lt;1|0&gt;] [,family=&lt;Base64 encoded string&gt;] [,manufacturer=&lt;Base64 encoded string&gt;] [,product=&lt;Base64 encoded string&gt;] [,serial=&lt;Base64 encoded string&gt;] [,sku=&lt;Base64 encoded string&gt;] [,uuid=&lt;UUID&gt;] [,version=&lt;Base64 encoded string&gt;]</span> 
</dt>
<dd>
<p>
Specify SMBIOS type 1 fields.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--smp</span> <span class="monospaced">&lt;integer&gt; (1 - N)</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
The number of CPUs. Please use option -sockets instead.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--sockets</span> <span class="monospaced">&lt;integer&gt; (1 - N)</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
The number of CPU sockets.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--spice_enhancements</span> <span class="monospaced">[foldersharing=&lt;1|0&gt;] [,videostreaming=&lt;off|all|filter&gt;]</span> 
</dt>
<dd>
<p>
Configure additional enhancements for SPICE.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--sshkeys</span> <span class="monospaced">&lt;filepath&gt;</span> 
</dt>
<dd>
<p>
cloud-init: Setup public SSH keys (one key per line, OpenSSH format).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--start</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Start VM after it was created successfully.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--startdate</span> <span class="monospaced">(now | YYYY-MM-DD | YYYY-MM-DDTHH:MM:SS)</span> (<em>default =</em> <span class="monospaced">now</span>)
</dt>
<dd>
<p>
Set the initial date of the real time clock. Valid format for date are:'now' or <em>2006-06-17T16:01:21</em> or <em>2006-06-17</em>.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--startup</span> `[[order=]\d+] [,up=\d+] [,down=\d+] ` 
</dt>
<dd>
<p>
Startup and shutdown behavior. Order is a non-negative number defining the general startup order. Shutdown in done with reverse ordering. Additionally you can set the <em>up</em> or <em>down</em> delay in seconds, which specifies a delay to wait before the next VM is started or stopped.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--storage</span> <span class="monospaced">&lt;storage ID&gt;</span> 
</dt>
<dd>
<p>
Default storage.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--tablet</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Enable/disable the USB tablet device.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--tags</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Tags of the VM. This is only meta information.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--tdf</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Enable/disable time drift fix.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--template</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Enable/disable Template.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--tpmstate0</span> <span class="monospaced">[file=]&lt;volume&gt; [,import-from=&lt;source volume&gt;] [,size=&lt;DiskSize&gt;] [,version=&lt;v1.2|v2.0&gt;]</span> 
</dt>
<dd>
<p>
Configure a Disk for storing TPM state. The format is fixed to <em>raw</em>. Use the special syntax STORAGE_ID:SIZE_IN_GiB to allocate a new volume. Note that SIZE_IN_GiB is ignored here and 4 MiB will be used instead. Use STORAGE_ID:0 and the <em>import-from</em> parameter to import from an existing volume.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--unique</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Assign a unique random ethernet address.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Requires option(s): <span class="monospaced">archive</span></td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">--unused[n]</span> <span class="monospaced">[file=]&lt;volume&gt;</span> 
</dt>
<dd>
<p>
Reference to unused volumes. This is used internally, and should not be modified manually.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--usb[n]</span> <span class="monospaced">[[host=]&lt;HOSTUSBDEVICE|spice&gt;] [,mapping=&lt;mapping-id&gt;] [,usb3=&lt;1|0&gt;]</span> 
</dt>
<dd>
<p>
Configure an USB device (n is 0 to 4, for machine version &gt;= 7.1 and ostype l26 or windows &gt; 7, n can be up to 14).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--vcpus</span> <span class="monospaced">&lt;integer&gt; (1 - N)</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Number of hotplugged vcpus.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--vga</span> <span class="monospaced">[[type=]&lt;enum&gt;] [,clipboard=&lt;vnc&gt;] [,memory=&lt;integer&gt;]</span> 
</dt>
<dd>
<p>
Configure the VGA hardware.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--virtio[n]</span> <span class="monospaced">[file=]&lt;volume&gt; [,aio=&lt;native|threads|io_uring&gt;] [,backup=&lt;1|0&gt;] [,bps=&lt;bps&gt;] [,bps_max_length=&lt;seconds&gt;] [,bps_rd=&lt;bps&gt;] [,bps_rd_max_length=&lt;seconds&gt;] [,bps_wr=&lt;bps&gt;] [,bps_wr_max_length=&lt;seconds&gt;] [,cache=&lt;enum&gt;] [,cyls=&lt;integer&gt;] [,detect_zeroes=&lt;1|0&gt;] [,discard=&lt;ignore|on&gt;] [,format=&lt;enum&gt;] [,heads=&lt;integer&gt;] [,import-from=&lt;source volume&gt;] [,iops=&lt;iops&gt;] [,iops_max=&lt;iops&gt;] [,iops_max_length=&lt;seconds&gt;] [,iops_rd=&lt;iops&gt;] [,iops_rd_max=&lt;iops&gt;] [,iops_rd_max_length=&lt;seconds&gt;] [,iops_wr=&lt;iops&gt;] [,iops_wr_max=&lt;iops&gt;] [,iops_wr_max_length=&lt;seconds&gt;] [,iothread=&lt;1|0&gt;] [,mbps=&lt;mbps&gt;] [,mbps_max=&lt;mbps&gt;] [,mbps_rd=&lt;mbps&gt;] [,mbps_rd_max=&lt;mbps&gt;] [,mbps_wr=&lt;mbps&gt;] [,mbps_wr_max=&lt;mbps&gt;] [,media=&lt;cdrom|disk&gt;] [,replicate=&lt;1|0&gt;] [,rerror=&lt;ignore|report|stop&gt;] [,ro=&lt;1|0&gt;] [,secs=&lt;integer&gt;] [,serial=&lt;serial&gt;] [,shared=&lt;1|0&gt;] [,size=&lt;DiskSize&gt;] [,snapshot=&lt;1|0&gt;] [,trans=&lt;none|lba|auto&gt;] [,werror=&lt;enum&gt;]</span> 
</dt>
<dd>
<p>
Use volume as VIRTIO hard disk (n is 0 to 15). Use the special syntax STORAGE_ID:SIZE_IN_GiB to allocate a new volume. Use STORAGE_ID:0 and the <em>import-from</em> parameter to import from an existing volume.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--vmgenid</span> <span class="monospaced">&lt;UUID&gt;</span> (<em>default =</em> <span class="monospaced">1 (autogenerated)</span>)
</dt>
<dd>
<p>
Set VM Generation ID. Use <em>1</em> to autogenerate on create or update, pass <em>0</em> to disable explicitly.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--vmstatestorage</span> <span class="monospaced">&lt;storage ID&gt;</span> 
</dt>
<dd>
<p>
Default storage for VM state volumes/files.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--watchdog</span> <span class="monospaced">[[model=]&lt;i6300esb|ib700&gt;] [,action=&lt;enum&gt;]</span> 
</dt>
<dd>
<p>
Create a virtual hardware watchdog device.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm delsnapshot</strong> <span class="monospaced">&lt;vmid&gt; &lt;snapname&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Delete a VM snapshot.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;snapname&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The name of the snapshot.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--force</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
For removal from config file, even if removing disk snapshots fails.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm destroy</strong> <span class="monospaced">&lt;vmid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Destroy the VM and  all used/owned volumes. Removes any VM specific
permissions and firewall rules</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--destroy-unreferenced-disks</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
If set, destroy additionally all disks not referenced in the config but with a matching VMID from all enabled storages.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--purge</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Remove VMID from configurations, like backup &amp; replication jobs and HA.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--skiplock</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Ignore locks - only root is allowed to use this option.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm disk import</strong> <span class="monospaced">&lt;vmid&gt; &lt;source&gt; &lt;storage&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Import an external disk image as an unused disk in a VM. The
 image format has to be supported by qemu-img(1).</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;source&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Path to the disk image to import
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;storage&gt;</span>: <span class="monospaced">&lt;storage ID&gt;</span> 
</dt>
<dd>
<p>
Target storage ID
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--format</span> <span class="monospaced">&lt;qcow2 | raw | vmdk&gt;</span> 
</dt>
<dd>
<p>
Target format
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm disk move</strong> <span class="monospaced">&lt;vmid&gt; &lt;disk&gt; [&lt;storage&gt;]</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Move volume to different storage or to a different VM.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;disk&gt;</span>: <span class="monospaced">&lt;efidisk0 | ide0 | ide1 | ide2 | ide3 | sata0 | sata1 | sata2 | sata3 | sata4 | sata5 | scsi0 | scsi1 | scsi10 | scsi11 | scsi12 | scsi13 | scsi14 | scsi15 | scsi16 | scsi17 | scsi18 | scsi19 | scsi2 | scsi20 | scsi21 | scsi22 | scsi23 | scsi24 | scsi25 | scsi26 | scsi27 | scsi28 | scsi29 | scsi3 | scsi30 | scsi4 | scsi5 | scsi6 | scsi7 | scsi8 | scsi9 | tpmstate0 | unused0 | unused1 | unused10 | unused100 | unused101 | unused102 | unused103 | unused104 | unused105 | unused106 | unused107 | unused108 | unused109 | unused11 | unused110 | unused111 | unused112 | unused113 | unused114 | unused115 | unused116 | unused117 | unused118 | unused119 | unused12 | unused120 | unused121 | unused122 | unused123 | unused124 | unused125 | unused126 | unused127 | unused128 | unused129 | unused13 | unused130 | unused131 | unused132 | unused133 | unused134 | unused135 | unused136 | unused137 | unused138 | unused139 | unused14 | unused140 | unused141 | unused142 | unused143 | unused144 | unused145 | unused146 | unused147 | unused148 | unused149 | unused15 | unused150 | unused151 | unused152 | unused153 | unused154 | unused155 | unused156 | unused157 | unused158 | unused159 | unused16 | unused160 | unused161 | unused162 | unused163 | unused164 | unused165 | unused166 | unused167 | unused168 | unused169 | unused17 | unused170 | unused171 | unused172 | unused173 | unused174 | unused175 | unused176 | unused177 | unused178 | unused179 | unused18 | unused180 | unused181 | unused182 | unused183 | unused184 | unused185 | unused186 | unused187 | unused188 | unused189 | unused19 | unused190 | unused191 | unused192 | unused193 | unused194 | unused195 | unused196 | unused197 | unused198 | unused199 | unused2 | unused20 | unused200 | unused201 | unused202 | unused203 | unused204 | unused205 | unused206 | unused207 | unused208 | unused209 | unused21 | unused210 | unused211 | unused212 | unused213 | unused214 | unused215 | unused216 | unused217 | unused218 | unused219 | unused22 | unused220 | unused221 | unused222 | unused223 | unused224 | unused225 | unused226 | unused227 | unused228 | unused229 | unused23 | unused230 | unused231 | unused232 | unused233 | unused234 | unused235 | unused236 | unused237 | unused238 | unused239 | unused24 | unused240 | unused241 | unused242 | unused243 | unused244 | unused245 | unused246 | unused247 | unused248 | unused249 | unused25 | unused250 | unused251 | unused252 | unused253 | unused254 | unused255 | unused26 | unused27 | unused28 | unused29 | unused3 | unused30 | unused31 | unused32 | unused33 | unused34 | unused35 | unused36 | unused37 | unused38 | unused39 | unused4 | unused40 | unused41 | unused42 | unused43 | unused44 | unused45 | unused46 | unused47 | unused48 | unused49 | unused5 | unused50 | unused51 | unused52 | unused53 | unused54 | unused55 | unused56 | unused57 | unused58 | unused59 | unused6 | unused60 | unused61 | unused62 | unused63 | unused64 | unused65 | unused66 | unused67 | unused68 | unused69 | unused7 | unused70 | unused71 | unused72 | unused73 | unused74 | unused75 | unused76 | unused77 | unused78 | unused79 | unused8 | unused80 | unused81 | unused82 | unused83 | unused84 | unused85 | unused86 | unused87 | unused88 | unused89 | unused9 | unused90 | unused91 | unused92 | unused93 | unused94 | unused95 | unused96 | unused97 | unused98 | unused99 | virtio0 | virtio1 | virtio10 | virtio11 | virtio12 | virtio13 | virtio14 | virtio15 | virtio2 | virtio3 | virtio4 | virtio5 | virtio6 | virtio7 | virtio8 | virtio9&gt;</span> 
</dt>
<dd>
<p>
The disk you want to move.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;storage&gt;</span>: <span class="monospaced">&lt;storage ID&gt;</span> 
</dt>
<dd>
<p>
Target storage.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--bwlimit</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">move limit from datacenter or storage config</span>)
</dt>
<dd>
<p>
Override I/O bandwidth limit (in KiB/s).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--delete</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Delete the original disk after successful copy. By default the original disk is kept as unused disk.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--digest</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Prevent changes if current configuration file has different SHA1"
                    ." digest. This can be used to prevent concurrent modifications.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--format</span> <span class="monospaced">&lt;qcow2 | raw | vmdk&gt;</span> 
</dt>
<dd>
<p>
Target Format.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--target-digest</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Prevent changes if the current config file of the target VM has a"
                    ." different SHA1 digest. This can be used to detect concurrent modifications.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--target-disk</span> <span class="monospaced">&lt;efidisk0 | ide0 | ide1 | ide2 | ide3 | sata0 | sata1 | sata2 | sata3 | sata4 | sata5 | scsi0 | scsi1 | scsi10 | scsi11 | scsi12 | scsi13 | scsi14 | scsi15 | scsi16 | scsi17 | scsi18 | scsi19 | scsi2 | scsi20 | scsi21 | scsi22 | scsi23 | scsi24 | scsi25 | scsi26 | scsi27 | scsi28 | scsi29 | scsi3 | scsi30 | scsi4 | scsi5 | scsi6 | scsi7 | scsi8 | scsi9 | tpmstate0 | unused0 | unused1 | unused10 | unused100 | unused101 | unused102 | unused103 | unused104 | unused105 | unused106 | unused107 | unused108 | unused109 | unused11 | unused110 | unused111 | unused112 | unused113 | unused114 | unused115 | unused116 | unused117 | unused118 | unused119 | unused12 | unused120 | unused121 | unused122 | unused123 | unused124 | unused125 | unused126 | unused127 | unused128 | unused129 | unused13 | unused130 | unused131 | unused132 | unused133 | unused134 | unused135 | unused136 | unused137 | unused138 | unused139 | unused14 | unused140 | unused141 | unused142 | unused143 | unused144 | unused145 | unused146 | unused147 | unused148 | unused149 | unused15 | unused150 | unused151 | unused152 | unused153 | unused154 | unused155 | unused156 | unused157 | unused158 | unused159 | unused16 | unused160 | unused161 | unused162 | unused163 | unused164 | unused165 | unused166 | unused167 | unused168 | unused169 | unused17 | unused170 | unused171 | unused172 | unused173 | unused174 | unused175 | unused176 | unused177 | unused178 | unused179 | unused18 | unused180 | unused181 | unused182 | unused183 | unused184 | unused185 | unused186 | unused187 | unused188 | unused189 | unused19 | unused190 | unused191 | unused192 | unused193 | unused194 | unused195 | unused196 | unused197 | unused198 | unused199 | unused2 | unused20 | unused200 | unused201 | unused202 | unused203 | unused204 | unused205 | unused206 | unused207 | unused208 | unused209 | unused21 | unused210 | unused211 | unused212 | unused213 | unused214 | unused215 | unused216 | unused217 | unused218 | unused219 | unused22 | unused220 | unused221 | unused222 | unused223 | unused224 | unused225 | unused226 | unused227 | unused228 | unused229 | unused23 | unused230 | unused231 | unused232 | unused233 | unused234 | unused235 | unused236 | unused237 | unused238 | unused239 | unused24 | unused240 | unused241 | unused242 | unused243 | unused244 | unused245 | unused246 | unused247 | unused248 | unused249 | unused25 | unused250 | unused251 | unused252 | unused253 | unused254 | unused255 | unused26 | unused27 | unused28 | unused29 | unused3 | unused30 | unused31 | unused32 | unused33 | unused34 | unused35 | unused36 | unused37 | unused38 | unused39 | unused4 | unused40 | unused41 | unused42 | unused43 | unused44 | unused45 | unused46 | unused47 | unused48 | unused49 | unused5 | unused50 | unused51 | unused52 | unused53 | unused54 | unused55 | unused56 | unused57 | unused58 | unused59 | unused6 | unused60 | unused61 | unused62 | unused63 | unused64 | unused65 | unused66 | unused67 | unused68 | unused69 | unused7 | unused70 | unused71 | unused72 | unused73 | unused74 | unused75 | unused76 | unused77 | unused78 | unused79 | unused8 | unused80 | unused81 | unused82 | unused83 | unused84 | unused85 | unused86 | unused87 | unused88 | unused89 | unused9 | unused90 | unused91 | unused92 | unused93 | unused94 | unused95 | unused96 | unused97 | unused98 | unused99 | virtio0 | virtio1 | virtio10 | virtio11 | virtio12 | virtio13 | virtio14 | virtio15 | virtio2 | virtio3 | virtio4 | virtio5 | virtio6 | virtio7 | virtio8 | virtio9&gt;</span> 
</dt>
<dd>
<p>
The config key the disk will be moved to on the target VM (for example, ide0 or scsi1). Default is the source disk key.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--target-vmid</span> <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm disk rescan</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Rescan all storages and update disk sizes and unused disk images.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--dryrun</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Do not actually write changes out to VM config(s).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--vmid</span> <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm disk resize</strong> <span class="monospaced">&lt;vmid&gt; &lt;disk&gt; &lt;size&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Extend volume size.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;disk&gt;</span>: <span class="monospaced">&lt;efidisk0 | ide0 | ide1 | ide2 | ide3 | sata0 | sata1 | sata2 | sata3 | sata4 | sata5 | scsi0 | scsi1 | scsi10 | scsi11 | scsi12 | scsi13 | scsi14 | scsi15 | scsi16 | scsi17 | scsi18 | scsi19 | scsi2 | scsi20 | scsi21 | scsi22 | scsi23 | scsi24 | scsi25 | scsi26 | scsi27 | scsi28 | scsi29 | scsi3 | scsi30 | scsi4 | scsi5 | scsi6 | scsi7 | scsi8 | scsi9 | tpmstate0 | virtio0 | virtio1 | virtio10 | virtio11 | virtio12 | virtio13 | virtio14 | virtio15 | virtio2 | virtio3 | virtio4 | virtio5 | virtio6 | virtio7 | virtio8 | virtio9&gt;</span> 
</dt>
<dd>
<p>
The disk you want to resize.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;size&gt;</span>: <span class="monospaced">\+?\d+(\.\d+)?[KMGT]?</span> 
</dt>
<dd>
<p>
The new size. With the <span class="monospaced">+</span> sign the value is added to the actual size of the volume and without it, the value is taken as an absolute one. Shrinking disk size is not supported.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--digest</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Prevent changes if current configuration file has different SHA1 digest. This can be used to prevent concurrent modifications.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--skiplock</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Ignore locks - only root is allowed to use this option.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm disk unlink</strong> <span class="monospaced">&lt;vmid&gt; --idlist &lt;string&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Unlink/delete disk images.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--force</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Force physical removal. Without this, we simple remove the disk from the config file and create an additional configuration entry called <em>unused[n]</em>, which contains the volume ID. Unlink of unused[n] always cause physical removal.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--idlist</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
A list of disk IDs you want to delete.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm guest cmd</strong> <span class="monospaced">&lt;vmid&gt; &lt;command&gt;</span></p></div>
<div class="paragraph">
<p>Execute QEMU Guest Agent commands.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;command&gt;</span>: <span class="monospaced">&lt;fsfreeze-freeze | fsfreeze-status | fsfreeze-thaw | fstrim | get-fsinfo | get-host-name | get-memory-block-info | get-memory-blocks | get-osinfo | get-time | get-timezone | get-users | get-vcpus | info | network-get-interfaces | ping | shutdown | suspend-disk | suspend-hybrid | suspend-ram&gt;</span> 
</dt>
<dd>
<p>
The QGA command.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm guest exec</strong> <span class="monospaced">&lt;vmid&gt; [&lt;extra-args&gt;]</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Executes the given command via the guest agent</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;extra-args&gt;</span>: <span class="monospaced">&lt;array&gt;</span> 
</dt>
<dd>
<p>
Extra arguments as array
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--pass-stdin</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
When set, read STDIN until EOF and forward to guest agent via <em>input-data</em> (usually treated as STDIN to process launched by guest agent). Allows maximal 1 MiB.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--synchronous</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
If set to off, returns the pid immediately instead of waiting for the commmand to finish or the timeout.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--timeout</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">30</span>)
</dt>
<dd>
<p>
The maximum time to wait synchronously for the command to finish. If reached, the pid gets returned. Set to 0 to deactivate
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm guest exec-status</strong> <span class="monospaced">&lt;vmid&gt; &lt;pid&gt;</span></p></div>
<div class="paragraph">
<p>Gets the status of the given pid started by the guest-agent</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;pid&gt;</span>: <span class="monospaced">&lt;integer&gt;</span> 
</dt>
<dd>
<p>
The PID to query
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm guest passwd</strong> <span class="monospaced">&lt;vmid&gt; &lt;username&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Sets the password for the given user to the given password</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;username&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The user to set the password for.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--crypted</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
set to 1 if the password has already been passed through crypt()
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm help</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Get help about specified command.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--extra-args</span> <span class="monospaced">&lt;array&gt;</span> 
</dt>
<dd>
<p>
Shows help for a specific command
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--verbose</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Verbose output format.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm import</strong> <span class="monospaced">&lt;vmid&gt; &lt;source&gt; --storage &lt;string&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Import a foreign virtual guest from a supported import source, such as an
ESXi storage.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;source&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The import source volume id.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--acpi</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Enable/disable ACPI.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--affinity</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
List of host cores used to execute guest processes, for example: 0,5,8-11
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--agent</span> <span class="monospaced">[enabled=]&lt;1|0&gt; [,freeze-fs-on-backup=&lt;1|0&gt;] [,fstrim_cloned_disks=&lt;1|0&gt;] [,type=&lt;virtio|isa&gt;]</span> 
</dt>
<dd>
<p>
Enable/disable communication with the QEMU Guest Agent and its properties.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--arch</span> <span class="monospaced">&lt;aarch64 | x86_64&gt;</span> 
</dt>
<dd>
<p>
Virtual processor architecture. Defaults to the host.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--args</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Arbitrary arguments passed to kvm.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--audio0</span> <span class="monospaced">device=&lt;ich9-intel-hda|intel-hda|AC97&gt; [,driver=&lt;spice|none&gt;]</span> 
</dt>
<dd>
<p>
Configure a audio device, useful in combination with QXL/Spice.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--autostart</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Automatic restart after crash (currently ignored).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--balloon</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> 
</dt>
<dd>
<p>
Amount of target RAM for the VM in MiB. Using zero disables the ballon driver.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--bios</span> <span class="monospaced">&lt;ovmf | seabios&gt;</span> (<em>default =</em> <span class="monospaced">seabios</span>)
</dt>
<dd>
<p>
Select BIOS implementation.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--boot</span> <span class="monospaced">[[legacy=]&lt;[acdn]{1,4}&gt;] [,order=&lt;device[;device...]&gt;]</span> 
</dt>
<dd>
<p>
Specify guest boot order. Use the <em>order=</em> sub-property as usage with no key or <em>legacy=</em> is deprecated.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--bootdisk</span> <span class="monospaced">(ide|sata|scsi|virtio)\d+</span> 
</dt>
<dd>
<p>
Enable booting from specified disk. Deprecated: Use <em>boot: order=foo;bar</em> instead.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--cdrom</span> <span class="monospaced">&lt;volume&gt;</span> 
</dt>
<dd>
<p>
This is an alias for option -ide2
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--cicustom</span> <span class="monospaced">[meta=&lt;volume&gt;] [,network=&lt;volume&gt;] [,user=&lt;volume&gt;] [,vendor=&lt;volume&gt;]</span> 
</dt>
<dd>
<p>
cloud-init: Specify custom files to replace the automatically generated ones at start.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--cipassword</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
cloud-init: Password to assign the user. Using this is generally not recommended. Use ssh keys instead. Also note that older cloud-init versions do not support hashed passwords.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--citype</span> <span class="monospaced">&lt;configdrive2 | nocloud | opennebula&gt;</span> 
</dt>
<dd>
<p>
Specifies the cloud-init configuration format. The default depends on the configured operating system type (<span class="monospaced">ostype</span>. We use the <span class="monospaced">nocloud</span> format for Linux, and <span class="monospaced">configdrive2</span> for windows.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--ciupgrade</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
cloud-init: do an automatic package upgrade after the first boot.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--ciuser</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
cloud-init: User name to change ssh keys and password for instead of the image’s configured default user.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--cores</span> <span class="monospaced">&lt;integer&gt; (1 - N)</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
The number of cores per socket.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--cpu</span> <span class="monospaced">[[cputype=]&lt;string&gt;] [,flags=&lt;+FLAG[;-FLAG...]&gt;] [,hidden=&lt;1|0&gt;] [,hv-vendor-id=&lt;vendor-id&gt;] [,phys-bits=&lt;8-64|host&gt;] [,reported-model=&lt;enum&gt;]</span> 
</dt>
<dd>
<p>
Emulated CPU type.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--cpulimit</span> <span class="monospaced">&lt;number&gt; (0 - 128)</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Limit of CPU usage.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--cpuunits</span> <span class="monospaced">&lt;integer&gt; (1 - 262144)</span> (<em>default =</em> <span class="monospaced">cgroup v1: 1024, cgroup v2: 100</span>)
</dt>
<dd>
<p>
CPU weight for a VM, will be clamped to [1, 10000] in cgroup v2.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--delete</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
A list of settings you want to delete.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--description</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Description for the VM. Shown in the web-interface VM’s summary. This is saved as comment inside the configuration file.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--dryrun</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Show the create command and exit without doing anything.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--efidisk0</span> <span class="monospaced">[file=]&lt;volume&gt; [,efitype=&lt;2m|4m&gt;] [,format=&lt;enum&gt;] [,pre-enrolled-keys=&lt;1|0&gt;] [,size=&lt;DiskSize&gt;]</span> 
</dt>
<dd>
<p>
Configure a disk for storing EFI vars.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--format</span> <span class="monospaced">&lt;qcow2 | raw | vmdk&gt;</span> 
</dt>
<dd>
<p>
Target format
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--freeze</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Freeze CPU at startup (use <em>c</em> monitor command to start execution).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--hookscript</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Script that will be executed during various steps in the vms lifetime.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--hostpci[n]</span> <span class="monospaced">[[host=]&lt;HOSTPCIID[;HOSTPCIID2...]&gt;] [,device-id=&lt;hex id&gt;] [,legacy-igd=&lt;1|0&gt;] [,mapping=&lt;mapping-id&gt;] [,mdev=&lt;string&gt;] [,pcie=&lt;1|0&gt;] [,rombar=&lt;1|0&gt;] [,romfile=&lt;string&gt;] [,sub-device-id=&lt;hex id&gt;] [,sub-vendor-id=&lt;hex id&gt;] [,vendor-id=&lt;hex id&gt;] [,x-vga=&lt;1|0&gt;]</span> 
</dt>
<dd>
<p>
Map host PCI devices into guest.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--hotplug</span> <span class="monospaced">&lt;string&gt;</span> (<em>default =</em> <span class="monospaced">network,disk,usb</span>)
</dt>
<dd>
<p>
Selectively enable hotplug features. This is a comma separated list of hotplug features: <em>network</em>, <em>disk</em>, <em>cpu</em>, <em>memory</em>, <em>usb</em> and <em>cloudinit</em>. Use <em>0</em> to disable hotplug completely. Using <em>1</em> as value is an alias for the default <span class="monospaced">network,disk,usb</span>. USB hotplugging is possible for guests with machine version &gt;= 7.1 and ostype l26 or windows &gt; 7.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--hugepages</span> <span class="monospaced">&lt;1024 | 2 | any&gt;</span> 
</dt>
<dd>
<p>
Enable/disable hugepages memory.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--ide[n]</span> <span class="monospaced">[file=]&lt;volume&gt; [,aio=&lt;native|threads|io_uring&gt;] [,backup=&lt;1|0&gt;] [,bps=&lt;bps&gt;] [,bps_max_length=&lt;seconds&gt;] [,bps_rd=&lt;bps&gt;] [,bps_rd_max_length=&lt;seconds&gt;] [,bps_wr=&lt;bps&gt;] [,bps_wr_max_length=&lt;seconds&gt;] [,cache=&lt;enum&gt;] [,cyls=&lt;integer&gt;] [,detect_zeroes=&lt;1|0&gt;] [,discard=&lt;ignore|on&gt;] [,format=&lt;enum&gt;] [,heads=&lt;integer&gt;] [,iops=&lt;iops&gt;] [,iops_max=&lt;iops&gt;] [,iops_max_length=&lt;seconds&gt;] [,iops_rd=&lt;iops&gt;] [,iops_rd_max=&lt;iops&gt;] [,iops_rd_max_length=&lt;seconds&gt;] [,iops_wr=&lt;iops&gt;] [,iops_wr_max=&lt;iops&gt;] [,iops_wr_max_length=&lt;seconds&gt;] [,mbps=&lt;mbps&gt;] [,mbps_max=&lt;mbps&gt;] [,mbps_rd=&lt;mbps&gt;] [,mbps_rd_max=&lt;mbps&gt;] [,mbps_wr=&lt;mbps&gt;] [,mbps_wr_max=&lt;mbps&gt;] [,media=&lt;cdrom|disk&gt;] [,model=&lt;model&gt;] [,replicate=&lt;1|0&gt;] [,rerror=&lt;ignore|report|stop&gt;] [,secs=&lt;integer&gt;] [,serial=&lt;serial&gt;] [,shared=&lt;1|0&gt;] [,size=&lt;DiskSize&gt;] [,snapshot=&lt;1|0&gt;] [,ssd=&lt;1|0&gt;] [,trans=&lt;none|lba|auto&gt;] [,werror=&lt;enum&gt;] [,wwn=&lt;wwn&gt;]</span> 
</dt>
<dd>
<p>
Use volume as IDE hard disk or CD-ROM (n is 0 to 3).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--ipconfig[n]</span> <span class="monospaced">[gw=&lt;GatewayIPv4&gt;] [,gw6=&lt;GatewayIPv6&gt;] [,ip=&lt;IPv4Format/CIDR&gt;] [,ip6=&lt;IPv6Format/CIDR&gt;]</span> 
</dt>
<dd>
<p>
cloud-init: Specify IP addresses and gateways for the corresponding interface.
</p>
<div class="paragraph">
<p>IP addresses use CIDR notation, gateways are optional but need an IP of the same type specified.</p></div>
<div class="paragraph">
<p>The special string <em>dhcp</em> can be used for IP addresses to use DHCP, in which case no explicit
gateway should be provided.
For IPv6 the special string <em>auto</em> can be used to use stateless autoconfiguration. This requires
cloud-init 19.4 or newer.</p></div>
<div class="paragraph">
<p>If cloud-init is enabled and neither an IPv4 nor an IPv6 address is specified, it defaults to using
dhcp on IPv4.</p></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">--ivshmem</span> <span class="monospaced">size=&lt;integer&gt; [,name=&lt;string&gt;]</span> 
</dt>
<dd>
<p>
Inter-VM shared memory. Useful for direct communication between VMs, or to the host.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--keephugepages</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Use together with hugepages. If enabled, hugepages will not not be deleted after VM shutdown and can be used for subsequent starts.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--keyboard</span> <span class="monospaced">&lt;da | de | de-ch | en-gb | en-us | es | fi | fr | fr-be | fr-ca | fr-ch | hu | is | it | ja | lt | mk | nl | no | pl | pt | pt-br | sl | sv | tr&gt;</span> 
</dt>
<dd>
<p>
Keyboard layout for VNC server. This option is generally not required and is often better handled from within the guest OS.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--kvm</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Enable/disable KVM hardware virtualization.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--live-import</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Immediately start the VM and copy the data in the background.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--localtime</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Set the real time clock (RTC) to local time. This is enabled by default if the <span class="monospaced">ostype</span> indicates a Microsoft Windows OS.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--lock</span> <span class="monospaced">&lt;backup | clone | create | migrate | rollback | snapshot | snapshot-delete | suspended | suspending&gt;</span> 
</dt>
<dd>
<p>
Lock/unlock the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--machine</span> <span class="monospaced">[[type=]&lt;machine type&gt;] [,viommu=&lt;intel|virtio&gt;]</span> 
</dt>
<dd>
<p>
Specify the QEMU machine.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--memory</span> <span class="monospaced">[current=]&lt;integer&gt;</span> 
</dt>
<dd>
<p>
Memory properties.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--migrate_downtime</span> <span class="monospaced">&lt;number&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">0.1</span>)
</dt>
<dd>
<p>
Set maximum tolerated downtime (in seconds) for migrations. Should the migration not be able to converge in the very end, because too much newly dirtied RAM needs to be transferred, the limit will be increased automatically step-by-step until migration can converge.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--migrate_speed</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Set maximum speed (in MB/s) for migrations. Value 0 is no limit.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--name</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Set a name for the VM. Only used on the configuration web interface.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--nameserver</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
cloud-init: Sets DNS server IP address for a container. Create will automatically use the setting from the host if neither searchdomain nor nameserver are set.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--net[n]</span> <span class="monospaced">[model=]&lt;enum&gt; [,bridge=&lt;bridge&gt;] [,firewall=&lt;1|0&gt;] [,link_down=&lt;1|0&gt;] [,macaddr=&lt;XX:XX:XX:XX:XX:XX&gt;] [,mtu=&lt;integer&gt;] [,queues=&lt;integer&gt;] [,rate=&lt;number&gt;] [,tag=&lt;integer&gt;] [,trunks=&lt;vlanid[;vlanid...]&gt;] [,&lt;model&gt;=&lt;macaddr&gt;]</span> 
</dt>
<dd>
<p>
Specify network devices.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--numa</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Enable/disable NUMA.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--numa[n]</span> <span class="monospaced">cpus=&lt;id[-id];...&gt; [,hostnodes=&lt;id[-id];...&gt;] [,memory=&lt;number&gt;] [,policy=&lt;preferred|bind|interleave&gt;]</span> 
</dt>
<dd>
<p>
NUMA topology.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--onboot</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Specifies whether a VM will be started during system bootup.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--ostype</span> <span class="monospaced">&lt;l24 | l26 | other | solaris | w2k | w2k3 | w2k8 | win10 | win11 | win7 | win8 | wvista | wxp&gt;</span> 
</dt>
<dd>
<p>
Specify guest operating system.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--parallel[n]</span> <span class="monospaced">/dev/parport\d+|/dev/usb/lp\d+</span> 
</dt>
<dd>
<p>
Map host parallel devices (n is 0 to 2).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--protection</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Sets the protection flag of the VM. This will disable the remove VM and remove disk operations.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--reboot</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Allow reboot. If set to <em>0</em> the VM exit on reboot.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--rng0</span> <span class="monospaced">[source=]&lt;/dev/urandom|/dev/random|/dev/hwrng&gt; [,max_bytes=&lt;integer&gt;] [,period=&lt;integer&gt;]</span> 
</dt>
<dd>
<p>
Configure a VirtIO-based Random Number Generator.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--sata[n]</span> <span class="monospaced">[file=]&lt;volume&gt; [,aio=&lt;native|threads|io_uring&gt;] [,backup=&lt;1|0&gt;] [,bps=&lt;bps&gt;] [,bps_max_length=&lt;seconds&gt;] [,bps_rd=&lt;bps&gt;] [,bps_rd_max_length=&lt;seconds&gt;] [,bps_wr=&lt;bps&gt;] [,bps_wr_max_length=&lt;seconds&gt;] [,cache=&lt;enum&gt;] [,cyls=&lt;integer&gt;] [,detect_zeroes=&lt;1|0&gt;] [,discard=&lt;ignore|on&gt;] [,format=&lt;enum&gt;] [,heads=&lt;integer&gt;] [,iops=&lt;iops&gt;] [,iops_max=&lt;iops&gt;] [,iops_max_length=&lt;seconds&gt;] [,iops_rd=&lt;iops&gt;] [,iops_rd_max=&lt;iops&gt;] [,iops_rd_max_length=&lt;seconds&gt;] [,iops_wr=&lt;iops&gt;] [,iops_wr_max=&lt;iops&gt;] [,iops_wr_max_length=&lt;seconds&gt;] [,mbps=&lt;mbps&gt;] [,mbps_max=&lt;mbps&gt;] [,mbps_rd=&lt;mbps&gt;] [,mbps_rd_max=&lt;mbps&gt;] [,mbps_wr=&lt;mbps&gt;] [,mbps_wr_max=&lt;mbps&gt;] [,media=&lt;cdrom|disk&gt;] [,replicate=&lt;1|0&gt;] [,rerror=&lt;ignore|report|stop&gt;] [,secs=&lt;integer&gt;] [,serial=&lt;serial&gt;] [,shared=&lt;1|0&gt;] [,size=&lt;DiskSize&gt;] [,snapshot=&lt;1|0&gt;] [,ssd=&lt;1|0&gt;] [,trans=&lt;none|lba|auto&gt;] [,werror=&lt;enum&gt;] [,wwn=&lt;wwn&gt;]</span> 
</dt>
<dd>
<p>
Use volume as SATA hard disk or CD-ROM (n is 0 to 5).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--scsi[n]</span> <span class="monospaced">[file=]&lt;volume&gt; [,aio=&lt;native|threads|io_uring&gt;] [,backup=&lt;1|0&gt;] [,bps=&lt;bps&gt;] [,bps_max_length=&lt;seconds&gt;] [,bps_rd=&lt;bps&gt;] [,bps_rd_max_length=&lt;seconds&gt;] [,bps_wr=&lt;bps&gt;] [,bps_wr_max_length=&lt;seconds&gt;] [,cache=&lt;enum&gt;] [,cyls=&lt;integer&gt;] [,detect_zeroes=&lt;1|0&gt;] [,discard=&lt;ignore|on&gt;] [,format=&lt;enum&gt;] [,heads=&lt;integer&gt;] [,iops=&lt;iops&gt;] [,iops_max=&lt;iops&gt;] [,iops_max_length=&lt;seconds&gt;] [,iops_rd=&lt;iops&gt;] [,iops_rd_max=&lt;iops&gt;] [,iops_rd_max_length=&lt;seconds&gt;] [,iops_wr=&lt;iops&gt;] [,iops_wr_max=&lt;iops&gt;] [,iops_wr_max_length=&lt;seconds&gt;] [,iothread=&lt;1|0&gt;] [,mbps=&lt;mbps&gt;] [,mbps_max=&lt;mbps&gt;] [,mbps_rd=&lt;mbps&gt;] [,mbps_rd_max=&lt;mbps&gt;] [,mbps_wr=&lt;mbps&gt;] [,mbps_wr_max=&lt;mbps&gt;] [,media=&lt;cdrom|disk&gt;] [,product=&lt;product&gt;] [,queues=&lt;integer&gt;] [,replicate=&lt;1|0&gt;] [,rerror=&lt;ignore|report|stop&gt;] [,ro=&lt;1|0&gt;] [,scsiblock=&lt;1|0&gt;] [,secs=&lt;integer&gt;] [,serial=&lt;serial&gt;] [,shared=&lt;1|0&gt;] [,size=&lt;DiskSize&gt;] [,snapshot=&lt;1|0&gt;] [,ssd=&lt;1|0&gt;] [,trans=&lt;none|lba|auto&gt;] [,vendor=&lt;vendor&gt;] [,werror=&lt;enum&gt;] [,wwn=&lt;wwn&gt;]</span> 
</dt>
<dd>
<p>
Use volume as SCSI hard disk or CD-ROM (n is 0 to 30).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--scsihw</span> <span class="monospaced">&lt;lsi | lsi53c810 | megasas | pvscsi | virtio-scsi-pci | virtio-scsi-single&gt;</span> (<em>default =</em> <span class="monospaced">lsi</span>)
</dt>
<dd>
<p>
SCSI controller model
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--searchdomain</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
cloud-init: Sets DNS search domains for a container. Create will automatically use the setting from the host if neither searchdomain nor nameserver are set.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--serial[n]</span> <span class="monospaced">(/dev/.+|socket)</span> 
</dt>
<dd>
<p>
Create a serial device inside the VM (n is 0 to 3)
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--shares</span> <span class="monospaced">&lt;integer&gt; (0 - 50000)</span> (<em>default =</em> <span class="monospaced">1000</span>)
</dt>
<dd>
<p>
Amount of memory shares for auto-ballooning. The larger the number is, the more memory this VM gets. Number is relative to weights of all other running VMs. Using zero disables auto-ballooning. Auto-ballooning is done by pvestatd.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--smbios1</span> <span class="monospaced">[base64=&lt;1|0&gt;] [,family=&lt;Base64 encoded string&gt;] [,manufacturer=&lt;Base64 encoded string&gt;] [,product=&lt;Base64 encoded string&gt;] [,serial=&lt;Base64 encoded string&gt;] [,sku=&lt;Base64 encoded string&gt;] [,uuid=&lt;UUID&gt;] [,version=&lt;Base64 encoded string&gt;]</span> 
</dt>
<dd>
<p>
Specify SMBIOS type 1 fields.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--smp</span> <span class="monospaced">&lt;integer&gt; (1 - N)</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
The number of CPUs. Please use option -sockets instead.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--sockets</span> <span class="monospaced">&lt;integer&gt; (1 - N)</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
The number of CPU sockets.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--spice_enhancements</span> <span class="monospaced">[foldersharing=&lt;1|0&gt;] [,videostreaming=&lt;off|all|filter&gt;]</span> 
</dt>
<dd>
<p>
Configure additional enhancements for SPICE.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--sshkeys</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
cloud-init: Setup public SSH keys (one key per line, OpenSSH format).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--startdate</span> <span class="monospaced">(now | YYYY-MM-DD | YYYY-MM-DDTHH:MM:SS)</span> (<em>default =</em> <span class="monospaced">now</span>)
</dt>
<dd>
<p>
Set the initial date of the real time clock. Valid format for date are:'now' or <em>2006-06-17T16:01:21</em> or <em>2006-06-17</em>.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--startup</span> `[[order=]\d+] [,up=\d+] [,down=\d+] ` 
</dt>
<dd>
<p>
Startup and shutdown behavior. Order is a non-negative number defining the general startup order. Shutdown in done with reverse ordering. Additionally you can set the <em>up</em> or <em>down</em> delay in seconds, which specifies a delay to wait before the next VM is started or stopped.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--storage</span> <span class="monospaced">&lt;storage ID&gt;</span> 
</dt>
<dd>
<p>
Default storage.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--tablet</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Enable/disable the USB tablet device.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--tags</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Tags of the VM. This is only meta information.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--tdf</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Enable/disable time drift fix.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--template</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Enable/disable Template.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--tpmstate0</span> <span class="monospaced">[file=]&lt;volume&gt; [,size=&lt;DiskSize&gt;] [,version=&lt;v1.2|v2.0&gt;]</span> 
</dt>
<dd>
<p>
Configure a Disk for storing TPM state. The format is fixed to <em>raw</em>.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--unused[n]</span> <span class="monospaced">[file=]&lt;volume&gt;</span> 
</dt>
<dd>
<p>
Reference to unused volumes. This is used internally, and should not be modified manually.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--usb[n]</span> <span class="monospaced">[[host=]&lt;HOSTUSBDEVICE|spice&gt;] [,mapping=&lt;mapping-id&gt;] [,usb3=&lt;1|0&gt;]</span> 
</dt>
<dd>
<p>
Configure an USB device (n is 0 to 4, for machine version &gt;= 7.1 and ostype l26 or windows &gt; 7, n can be up to 14).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--vcpus</span> <span class="monospaced">&lt;integer&gt; (1 - N)</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Number of hotplugged vcpus.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--vga</span> <span class="monospaced">[[type=]&lt;enum&gt;] [,clipboard=&lt;vnc&gt;] [,memory=&lt;integer&gt;]</span> 
</dt>
<dd>
<p>
Configure the VGA hardware.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--virtio[n]</span> <span class="monospaced">[file=]&lt;volume&gt; [,aio=&lt;native|threads|io_uring&gt;] [,backup=&lt;1|0&gt;] [,bps=&lt;bps&gt;] [,bps_max_length=&lt;seconds&gt;] [,bps_rd=&lt;bps&gt;] [,bps_rd_max_length=&lt;seconds&gt;] [,bps_wr=&lt;bps&gt;] [,bps_wr_max_length=&lt;seconds&gt;] [,cache=&lt;enum&gt;] [,cyls=&lt;integer&gt;] [,detect_zeroes=&lt;1|0&gt;] [,discard=&lt;ignore|on&gt;] [,format=&lt;enum&gt;] [,heads=&lt;integer&gt;] [,iops=&lt;iops&gt;] [,iops_max=&lt;iops&gt;] [,iops_max_length=&lt;seconds&gt;] [,iops_rd=&lt;iops&gt;] [,iops_rd_max=&lt;iops&gt;] [,iops_rd_max_length=&lt;seconds&gt;] [,iops_wr=&lt;iops&gt;] [,iops_wr_max=&lt;iops&gt;] [,iops_wr_max_length=&lt;seconds&gt;] [,iothread=&lt;1|0&gt;] [,mbps=&lt;mbps&gt;] [,mbps_max=&lt;mbps&gt;] [,mbps_rd=&lt;mbps&gt;] [,mbps_rd_max=&lt;mbps&gt;] [,mbps_wr=&lt;mbps&gt;] [,mbps_wr_max=&lt;mbps&gt;] [,media=&lt;cdrom|disk&gt;] [,replicate=&lt;1|0&gt;] [,rerror=&lt;ignore|report|stop&gt;] [,ro=&lt;1|0&gt;] [,secs=&lt;integer&gt;] [,serial=&lt;serial&gt;] [,shared=&lt;1|0&gt;] [,size=&lt;DiskSize&gt;] [,snapshot=&lt;1|0&gt;] [,trans=&lt;none|lba|auto&gt;] [,werror=&lt;enum&gt;]</span> 
</dt>
<dd>
<p>
Use volume as VIRTIO hard disk (n is 0 to 15).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--vmgenid</span> <span class="monospaced">&lt;UUID&gt;</span> (<em>default =</em> <span class="monospaced">1 (autogenerated)</span>)
</dt>
<dd>
<p>
Set VM Generation ID. Use <em>1</em> to autogenerate on create or update, pass <em>0</em> to disable explicitly.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--vmstatestorage</span> <span class="monospaced">&lt;storage ID&gt;</span> 
</dt>
<dd>
<p>
Default storage for VM state volumes/files.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--watchdog</span> <span class="monospaced">[[model=]&lt;i6300esb|ib700&gt;] [,action=&lt;enum&gt;]</span> 
</dt>
<dd>
<p>
Create a virtual hardware watchdog device.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm importdisk</strong></p></div>
<div class="paragraph">
<p>An alias for <em>qm disk import</em>.</p></div>
<div class="paragraph">
<p><strong>qm importovf</strong> <span class="monospaced">&lt;vmid&gt; &lt;manifest&gt; &lt;storage&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Create a new VM using parameters read from an OVF manifest</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;manifest&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
path to the ovf file
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;storage&gt;</span>: <span class="monospaced">&lt;storage ID&gt;</span> 
</dt>
<dd>
<p>
Target storage ID
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--dryrun</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Print a parsed representation of the extracted OVF parameters, but do not create a VM
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--format</span> <span class="monospaced">&lt;qcow2 | raw | vmdk&gt;</span> 
</dt>
<dd>
<p>
Target format
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm list</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Virtual machine index (per node).</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--full</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Determine the full status of active VMs.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm listsnapshot</strong> <span class="monospaced">&lt;vmid&gt;</span></p></div>
<div class="paragraph">
<p>List all snapshots.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm migrate</strong> <span class="monospaced">&lt;vmid&gt; &lt;target&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Migrate virtual machine. Creates a new migration task.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;target&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Target node.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--bwlimit</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">migrate limit from datacenter or storage config</span>)
</dt>
<dd>
<p>
Override I/O bandwidth limit (in KiB/s).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--force</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Allow to migrate VMs which use local devices. Only root may use this option.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--migration_network</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
CIDR of the (sub) network that is used for migration.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--migration_type</span> <span class="monospaced">&lt;insecure | secure&gt;</span> 
</dt>
<dd>
<p>
Migration traffic is encrypted using an SSH tunnel by default. On secure, completely private networks this can be disabled to increase performance.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--online</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Use online/live migration if VM is running. Ignored if VM is stopped.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--targetstorage</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Mapping from source to target storages. Providing only a single storage ID maps all source storages to that storage. Providing the special value <em>1</em> will map each source storage to itself.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--with-local-disks</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Enable live storage migration for local disk
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm monitor</strong> <span class="monospaced">&lt;vmid&gt;</span></p></div>
<div class="paragraph">
<p>Enter QEMU Monitor interface.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm move-disk</strong></p></div>
<div class="paragraph">
<p>An alias for <em>qm disk move</em>.</p></div>
<div class="paragraph">
<p><strong>qm move_disk</strong></p></div>
<div class="paragraph">
<p>An alias for <em>qm disk move</em>.</p></div>
<div class="paragraph">
<p><strong>qm mtunnel</strong></p></div>
<div class="paragraph">
<p>Used by qmigrate - do not use manually.</p></div>
<div class="paragraph">
<p><strong>qm nbdstop</strong> <span class="monospaced">&lt;vmid&gt;</span></p></div>
<div class="paragraph">
<p>Stop embedded nbd server.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm pending</strong> <span class="monospaced">&lt;vmid&gt;</span></p></div>
<div class="paragraph">
<p>Get the virtual machine configuration with both current and pending values.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm reboot</strong> <span class="monospaced">&lt;vmid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Reboot the VM by shutting it down, and starting it again. Applies pending
changes.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--timeout</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> 
</dt>
<dd>
<p>
Wait maximal timeout seconds for the shutdown.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm remote-migrate</strong> <span class="monospaced">&lt;vmid&gt; [&lt;target-vmid&gt;] &lt;target-endpoint&gt; --target-bridge &lt;string&gt; --target-storage &lt;string&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Migrate virtual machine to a remote cluster. Creates a new migration task.
EXPERIMENTAL feature!</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;target-vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;target-endpoint&gt;</span>: <span class="monospaced">apitoken=&lt;PVEAPIToken=user@realm!token=SECRET&gt; ,host=&lt;ADDRESS&gt; [,fingerprint=&lt;FINGERPRINT&gt;] [,port=&lt;PORT&gt;]</span> 
</dt>
<dd>
<p>
Remote target endpoint
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--bwlimit</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">migrate limit from datacenter or storage config</span>)
</dt>
<dd>
<p>
Override I/O bandwidth limit (in KiB/s).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--delete</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Delete the original VM and related data after successful migration. By default the original VM is kept on the source cluster in a stopped state.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--online</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Use online/live migration if VM is running. Ignored if VM is stopped.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--target-bridge</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Mapping from source to target bridges. Providing only a single bridge ID maps all source bridges to that bridge. Providing the special value <em>1</em> will map each source bridge to itself.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--target-storage</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Mapping from source to target storages. Providing only a single storage ID maps all source storages to that storage. Providing the special value <em>1</em> will map each source storage to itself.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm rescan</strong></p></div>
<div class="paragraph">
<p>An alias for <em>qm disk rescan</em>.</p></div>
<div class="paragraph">
<p><strong>qm reset</strong> <span class="monospaced">&lt;vmid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Reset virtual machine.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--skiplock</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Ignore locks - only root is allowed to use this option.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm resize</strong></p></div>
<div class="paragraph">
<p>An alias for <em>qm disk resize</em>.</p></div>
<div class="paragraph">
<p><strong>qm resume</strong> <span class="monospaced">&lt;vmid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Resume virtual machine.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--nocheck</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
no description available
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--skiplock</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Ignore locks - only root is allowed to use this option.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm rollback</strong> <span class="monospaced">&lt;vmid&gt; &lt;snapname&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Rollback VM state to specified snapshot.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;snapname&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The name of the snapshot.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--start</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Whether the VM should get started after rolling back successfully. (Note: VMs will be automatically started if the snapshot includes RAM.)
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm sendkey</strong> <span class="monospaced">&lt;vmid&gt; &lt;key&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Send key event to virtual machine.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;key&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The key (qemu monitor encoding).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--skiplock</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Ignore locks - only root is allowed to use this option.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm set</strong> <span class="monospaced">&lt;vmid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Set virtual machine options (synchrounous API) - You should consider using
the POST method instead for any actions involving hotplug or storage
allocation.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--acpi</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Enable/disable ACPI.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--affinity</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
List of host cores used to execute guest processes, for example: 0,5,8-11
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--agent</span> <span class="monospaced">[enabled=]&lt;1|0&gt; [,freeze-fs-on-backup=&lt;1|0&gt;] [,fstrim_cloned_disks=&lt;1|0&gt;] [,type=&lt;virtio|isa&gt;]</span> 
</dt>
<dd>
<p>
Enable/disable communication with the QEMU Guest Agent and its properties.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--arch</span> <span class="monospaced">&lt;aarch64 | x86_64&gt;</span> 
</dt>
<dd>
<p>
Virtual processor architecture. Defaults to the host.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--args</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Arbitrary arguments passed to kvm.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--audio0</span> <span class="monospaced">device=&lt;ich9-intel-hda|intel-hda|AC97&gt; [,driver=&lt;spice|none&gt;]</span> 
</dt>
<dd>
<p>
Configure a audio device, useful in combination with QXL/Spice.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--autostart</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Automatic restart after crash (currently ignored).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--balloon</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> 
</dt>
<dd>
<p>
Amount of target RAM for the VM in MiB. Using zero disables the ballon driver.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--bios</span> <span class="monospaced">&lt;ovmf | seabios&gt;</span> (<em>default =</em> <span class="monospaced">seabios</span>)
</dt>
<dd>
<p>
Select BIOS implementation.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--boot</span> <span class="monospaced">[[legacy=]&lt;[acdn]{1,4}&gt;] [,order=&lt;device[;device...]&gt;]</span> 
</dt>
<dd>
<p>
Specify guest boot order. Use the <em>order=</em> sub-property as usage with no key or <em>legacy=</em> is deprecated.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--bootdisk</span> <span class="monospaced">(ide|sata|scsi|virtio)\d+</span> 
</dt>
<dd>
<p>
Enable booting from specified disk. Deprecated: Use <em>boot: order=foo;bar</em> instead.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--cdrom</span> <span class="monospaced">&lt;volume&gt;</span> 
</dt>
<dd>
<p>
This is an alias for option -ide2
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--cicustom</span> <span class="monospaced">[meta=&lt;volume&gt;] [,network=&lt;volume&gt;] [,user=&lt;volume&gt;] [,vendor=&lt;volume&gt;]</span> 
</dt>
<dd>
<p>
cloud-init: Specify custom files to replace the automatically generated ones at start.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--cipassword</span> <span class="monospaced">&lt;password&gt;</span> 
</dt>
<dd>
<p>
cloud-init: Password to assign the user. Using this is generally not recommended. Use ssh keys instead. Also note that older cloud-init versions do not support hashed passwords.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--citype</span> <span class="monospaced">&lt;configdrive2 | nocloud | opennebula&gt;</span> 
</dt>
<dd>
<p>
Specifies the cloud-init configuration format. The default depends on the configured operating system type (<span class="monospaced">ostype</span>. We use the <span class="monospaced">nocloud</span> format for Linux, and <span class="monospaced">configdrive2</span> for windows.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--ciupgrade</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
cloud-init: do an automatic package upgrade after the first boot.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--ciuser</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
cloud-init: User name to change ssh keys and password for instead of the image’s configured default user.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--cores</span> <span class="monospaced">&lt;integer&gt; (1 - N)</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
The number of cores per socket.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--cpu</span> <span class="monospaced">[[cputype=]&lt;string&gt;] [,flags=&lt;+FLAG[;-FLAG...]&gt;] [,hidden=&lt;1|0&gt;] [,hv-vendor-id=&lt;vendor-id&gt;] [,phys-bits=&lt;8-64|host&gt;] [,reported-model=&lt;enum&gt;]</span> 
</dt>
<dd>
<p>
Emulated CPU type.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--cpulimit</span> <span class="monospaced">&lt;number&gt; (0 - 128)</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Limit of CPU usage.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--cpuunits</span> <span class="monospaced">&lt;integer&gt; (1 - 262144)</span> (<em>default =</em> <span class="monospaced">cgroup v1: 1024, cgroup v2: 100</span>)
</dt>
<dd>
<p>
CPU weight for a VM, will be clamped to [1, 10000] in cgroup v2.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--delete</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
A list of settings you want to delete.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--description</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Description for the VM. Shown in the web-interface VM’s summary. This is saved as comment inside the configuration file.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--digest</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Prevent changes if current configuration file has different SHA1 digest. This can be used to prevent concurrent modifications.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--efidisk0</span> <span class="monospaced">[file=]&lt;volume&gt; [,efitype=&lt;2m|4m&gt;] [,format=&lt;enum&gt;] [,import-from=&lt;source volume&gt;] [,pre-enrolled-keys=&lt;1|0&gt;] [,size=&lt;DiskSize&gt;]</span> 
</dt>
<dd>
<p>
Configure a disk for storing EFI vars. Use the special syntax STORAGE_ID:SIZE_IN_GiB to allocate a new volume. Note that SIZE_IN_GiB is ignored here and that the default EFI vars are copied to the volume instead. Use STORAGE_ID:0 and the <em>import-from</em> parameter to import from an existing volume.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--force</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Force physical removal. Without this, we simple remove the disk from the config file and create an additional configuration entry called <em>unused[n]</em>, which contains the volume ID. Unlink of unused[n] always cause physical removal.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Requires option(s): <span class="monospaced">delete</span></td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">--freeze</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Freeze CPU at startup (use <em>c</em> monitor command to start execution).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--hookscript</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Script that will be executed during various steps in the vms lifetime.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--hostpci[n]</span> <span class="monospaced">[[host=]&lt;HOSTPCIID[;HOSTPCIID2...]&gt;] [,device-id=&lt;hex id&gt;] [,legacy-igd=&lt;1|0&gt;] [,mapping=&lt;mapping-id&gt;] [,mdev=&lt;string&gt;] [,pcie=&lt;1|0&gt;] [,rombar=&lt;1|0&gt;] [,romfile=&lt;string&gt;] [,sub-device-id=&lt;hex id&gt;] [,sub-vendor-id=&lt;hex id&gt;] [,vendor-id=&lt;hex id&gt;] [,x-vga=&lt;1|0&gt;]</span> 
</dt>
<dd>
<p>
Map host PCI devices into guest.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--hotplug</span> <span class="monospaced">&lt;string&gt;</span> (<em>default =</em> <span class="monospaced">network,disk,usb</span>)
</dt>
<dd>
<p>
Selectively enable hotplug features. This is a comma separated list of hotplug features: <em>network</em>, <em>disk</em>, <em>cpu</em>, <em>memory</em>, <em>usb</em> and <em>cloudinit</em>. Use <em>0</em> to disable hotplug completely. Using <em>1</em> as value is an alias for the default <span class="monospaced">network,disk,usb</span>. USB hotplugging is possible for guests with machine version &gt;= 7.1 and ostype l26 or windows &gt; 7.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--hugepages</span> <span class="monospaced">&lt;1024 | 2 | any&gt;</span> 
</dt>
<dd>
<p>
Enable/disable hugepages memory.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--ide[n]</span> <span class="monospaced">[file=]&lt;volume&gt; [,aio=&lt;native|threads|io_uring&gt;] [,backup=&lt;1|0&gt;] [,bps=&lt;bps&gt;] [,bps_max_length=&lt;seconds&gt;] [,bps_rd=&lt;bps&gt;] [,bps_rd_max_length=&lt;seconds&gt;] [,bps_wr=&lt;bps&gt;] [,bps_wr_max_length=&lt;seconds&gt;] [,cache=&lt;enum&gt;] [,cyls=&lt;integer&gt;] [,detect_zeroes=&lt;1|0&gt;] [,discard=&lt;ignore|on&gt;] [,format=&lt;enum&gt;] [,heads=&lt;integer&gt;] [,import-from=&lt;source volume&gt;] [,iops=&lt;iops&gt;] [,iops_max=&lt;iops&gt;] [,iops_max_length=&lt;seconds&gt;] [,iops_rd=&lt;iops&gt;] [,iops_rd_max=&lt;iops&gt;] [,iops_rd_max_length=&lt;seconds&gt;] [,iops_wr=&lt;iops&gt;] [,iops_wr_max=&lt;iops&gt;] [,iops_wr_max_length=&lt;seconds&gt;] [,mbps=&lt;mbps&gt;] [,mbps_max=&lt;mbps&gt;] [,mbps_rd=&lt;mbps&gt;] [,mbps_rd_max=&lt;mbps&gt;] [,mbps_wr=&lt;mbps&gt;] [,mbps_wr_max=&lt;mbps&gt;] [,media=&lt;cdrom|disk&gt;] [,model=&lt;model&gt;] [,replicate=&lt;1|0&gt;] [,rerror=&lt;ignore|report|stop&gt;] [,secs=&lt;integer&gt;] [,serial=&lt;serial&gt;] [,shared=&lt;1|0&gt;] [,size=&lt;DiskSize&gt;] [,snapshot=&lt;1|0&gt;] [,ssd=&lt;1|0&gt;] [,trans=&lt;none|lba|auto&gt;] [,werror=&lt;enum&gt;] [,wwn=&lt;wwn&gt;]</span> 
</dt>
<dd>
<p>
Use volume as IDE hard disk or CD-ROM (n is 0 to 3). Use the special syntax STORAGE_ID:SIZE_IN_GiB to allocate a new volume. Use STORAGE_ID:0 and the <em>import-from</em> parameter to import from an existing volume.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--ipconfig[n]</span> <span class="monospaced">[gw=&lt;GatewayIPv4&gt;] [,gw6=&lt;GatewayIPv6&gt;] [,ip=&lt;IPv4Format/CIDR&gt;] [,ip6=&lt;IPv6Format/CIDR&gt;]</span> 
</dt>
<dd>
<p>
cloud-init: Specify IP addresses and gateways for the corresponding interface.
</p>
<div class="paragraph">
<p>IP addresses use CIDR notation, gateways are optional but need an IP of the same type specified.</p></div>
<div class="paragraph">
<p>The special string <em>dhcp</em> can be used for IP addresses to use DHCP, in which case no explicit
gateway should be provided.
For IPv6 the special string <em>auto</em> can be used to use stateless autoconfiguration. This requires
cloud-init 19.4 or newer.</p></div>
<div class="paragraph">
<p>If cloud-init is enabled and neither an IPv4 nor an IPv6 address is specified, it defaults to using
dhcp on IPv4.</p></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">--ivshmem</span> <span class="monospaced">size=&lt;integer&gt; [,name=&lt;string&gt;]</span> 
</dt>
<dd>
<p>
Inter-VM shared memory. Useful for direct communication between VMs, or to the host.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--keephugepages</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Use together with hugepages. If enabled, hugepages will not not be deleted after VM shutdown and can be used for subsequent starts.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--keyboard</span> <span class="monospaced">&lt;da | de | de-ch | en-gb | en-us | es | fi | fr | fr-be | fr-ca | fr-ch | hu | is | it | ja | lt | mk | nl | no | pl | pt | pt-br | sl | sv | tr&gt;</span> 
</dt>
<dd>
<p>
Keyboard layout for VNC server. This option is generally not required and is often better handled from within the guest OS.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--kvm</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Enable/disable KVM hardware virtualization.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--localtime</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Set the real time clock (RTC) to local time. This is enabled by default if the <span class="monospaced">ostype</span> indicates a Microsoft Windows OS.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--lock</span> <span class="monospaced">&lt;backup | clone | create | migrate | rollback | snapshot | snapshot-delete | suspended | suspending&gt;</span> 
</dt>
<dd>
<p>
Lock/unlock the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--machine</span> <span class="monospaced">[[type=]&lt;machine type&gt;] [,viommu=&lt;intel|virtio&gt;]</span> 
</dt>
<dd>
<p>
Specify the QEMU machine.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--memory</span> <span class="monospaced">[current=]&lt;integer&gt;</span> 
</dt>
<dd>
<p>
Memory properties.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--migrate_downtime</span> <span class="monospaced">&lt;number&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">0.1</span>)
</dt>
<dd>
<p>
Set maximum tolerated downtime (in seconds) for migrations. Should the migration not be able to converge in the very end, because too much newly dirtied RAM needs to be transferred, the limit will be increased automatically step-by-step until migration can converge.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--migrate_speed</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Set maximum speed (in MB/s) for migrations. Value 0 is no limit.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--name</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Set a name for the VM. Only used on the configuration web interface.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--nameserver</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
cloud-init: Sets DNS server IP address for a container. Create will automatically use the setting from the host if neither searchdomain nor nameserver are set.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--net[n]</span> <span class="monospaced">[model=]&lt;enum&gt; [,bridge=&lt;bridge&gt;] [,firewall=&lt;1|0&gt;] [,link_down=&lt;1|0&gt;] [,macaddr=&lt;XX:XX:XX:XX:XX:XX&gt;] [,mtu=&lt;integer&gt;] [,queues=&lt;integer&gt;] [,rate=&lt;number&gt;] [,tag=&lt;integer&gt;] [,trunks=&lt;vlanid[;vlanid...]&gt;] [,&lt;model&gt;=&lt;macaddr&gt;]</span> 
</dt>
<dd>
<p>
Specify network devices.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--numa</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Enable/disable NUMA.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--numa[n]</span> <span class="monospaced">cpus=&lt;id[-id];...&gt; [,hostnodes=&lt;id[-id];...&gt;] [,memory=&lt;number&gt;] [,policy=&lt;preferred|bind|interleave&gt;]</span> 
</dt>
<dd>
<p>
NUMA topology.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--onboot</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Specifies whether a VM will be started during system bootup.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--ostype</span> <span class="monospaced">&lt;l24 | l26 | other | solaris | w2k | w2k3 | w2k8 | win10 | win11 | win7 | win8 | wvista | wxp&gt;</span> 
</dt>
<dd>
<p>
Specify guest operating system.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--parallel[n]</span> <span class="monospaced">/dev/parport\d+|/dev/usb/lp\d+</span> 
</dt>
<dd>
<p>
Map host parallel devices (n is 0 to 2).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--protection</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Sets the protection flag of the VM. This will disable the remove VM and remove disk operations.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--reboot</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Allow reboot. If set to <em>0</em> the VM exit on reboot.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--revert</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Revert a pending change.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--rng0</span> <span class="monospaced">[source=]&lt;/dev/urandom|/dev/random|/dev/hwrng&gt; [,max_bytes=&lt;integer&gt;] [,period=&lt;integer&gt;]</span> 
</dt>
<dd>
<p>
Configure a VirtIO-based Random Number Generator.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--sata[n]</span> <span class="monospaced">[file=]&lt;volume&gt; [,aio=&lt;native|threads|io_uring&gt;] [,backup=&lt;1|0&gt;] [,bps=&lt;bps&gt;] [,bps_max_length=&lt;seconds&gt;] [,bps_rd=&lt;bps&gt;] [,bps_rd_max_length=&lt;seconds&gt;] [,bps_wr=&lt;bps&gt;] [,bps_wr_max_length=&lt;seconds&gt;] [,cache=&lt;enum&gt;] [,cyls=&lt;integer&gt;] [,detect_zeroes=&lt;1|0&gt;] [,discard=&lt;ignore|on&gt;] [,format=&lt;enum&gt;] [,heads=&lt;integer&gt;] [,import-from=&lt;source volume&gt;] [,iops=&lt;iops&gt;] [,iops_max=&lt;iops&gt;] [,iops_max_length=&lt;seconds&gt;] [,iops_rd=&lt;iops&gt;] [,iops_rd_max=&lt;iops&gt;] [,iops_rd_max_length=&lt;seconds&gt;] [,iops_wr=&lt;iops&gt;] [,iops_wr_max=&lt;iops&gt;] [,iops_wr_max_length=&lt;seconds&gt;] [,mbps=&lt;mbps&gt;] [,mbps_max=&lt;mbps&gt;] [,mbps_rd=&lt;mbps&gt;] [,mbps_rd_max=&lt;mbps&gt;] [,mbps_wr=&lt;mbps&gt;] [,mbps_wr_max=&lt;mbps&gt;] [,media=&lt;cdrom|disk&gt;] [,replicate=&lt;1|0&gt;] [,rerror=&lt;ignore|report|stop&gt;] [,secs=&lt;integer&gt;] [,serial=&lt;serial&gt;] [,shared=&lt;1|0&gt;] [,size=&lt;DiskSize&gt;] [,snapshot=&lt;1|0&gt;] [,ssd=&lt;1|0&gt;] [,trans=&lt;none|lba|auto&gt;] [,werror=&lt;enum&gt;] [,wwn=&lt;wwn&gt;]</span> 
</dt>
<dd>
<p>
Use volume as SATA hard disk or CD-ROM (n is 0 to 5). Use the special syntax STORAGE_ID:SIZE_IN_GiB to allocate a new volume. Use STORAGE_ID:0 and the <em>import-from</em> parameter to import from an existing volume.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--scsi[n]</span> <span class="monospaced">[file=]&lt;volume&gt; [,aio=&lt;native|threads|io_uring&gt;] [,backup=&lt;1|0&gt;] [,bps=&lt;bps&gt;] [,bps_max_length=&lt;seconds&gt;] [,bps_rd=&lt;bps&gt;] [,bps_rd_max_length=&lt;seconds&gt;] [,bps_wr=&lt;bps&gt;] [,bps_wr_max_length=&lt;seconds&gt;] [,cache=&lt;enum&gt;] [,cyls=&lt;integer&gt;] [,detect_zeroes=&lt;1|0&gt;] [,discard=&lt;ignore|on&gt;] [,format=&lt;enum&gt;] [,heads=&lt;integer&gt;] [,import-from=&lt;source volume&gt;] [,iops=&lt;iops&gt;] [,iops_max=&lt;iops&gt;] [,iops_max_length=&lt;seconds&gt;] [,iops_rd=&lt;iops&gt;] [,iops_rd_max=&lt;iops&gt;] [,iops_rd_max_length=&lt;seconds&gt;] [,iops_wr=&lt;iops&gt;] [,iops_wr_max=&lt;iops&gt;] [,iops_wr_max_length=&lt;seconds&gt;] [,iothread=&lt;1|0&gt;] [,mbps=&lt;mbps&gt;] [,mbps_max=&lt;mbps&gt;] [,mbps_rd=&lt;mbps&gt;] [,mbps_rd_max=&lt;mbps&gt;] [,mbps_wr=&lt;mbps&gt;] [,mbps_wr_max=&lt;mbps&gt;] [,media=&lt;cdrom|disk&gt;] [,product=&lt;product&gt;] [,queues=&lt;integer&gt;] [,replicate=&lt;1|0&gt;] [,rerror=&lt;ignore|report|stop&gt;] [,ro=&lt;1|0&gt;] [,scsiblock=&lt;1|0&gt;] [,secs=&lt;integer&gt;] [,serial=&lt;serial&gt;] [,shared=&lt;1|0&gt;] [,size=&lt;DiskSize&gt;] [,snapshot=&lt;1|0&gt;] [,ssd=&lt;1|0&gt;] [,trans=&lt;none|lba|auto&gt;] [,vendor=&lt;vendor&gt;] [,werror=&lt;enum&gt;] [,wwn=&lt;wwn&gt;]</span> 
</dt>
<dd>
<p>
Use volume as SCSI hard disk or CD-ROM (n is 0 to 30). Use the special syntax STORAGE_ID:SIZE_IN_GiB to allocate a new volume. Use STORAGE_ID:0 and the <em>import-from</em> parameter to import from an existing volume.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--scsihw</span> <span class="monospaced">&lt;lsi | lsi53c810 | megasas | pvscsi | virtio-scsi-pci | virtio-scsi-single&gt;</span> (<em>default =</em> <span class="monospaced">lsi</span>)
</dt>
<dd>
<p>
SCSI controller model
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--searchdomain</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
cloud-init: Sets DNS search domains for a container. Create will automatically use the setting from the host if neither searchdomain nor nameserver are set.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--serial[n]</span> <span class="monospaced">(/dev/.+|socket)</span> 
</dt>
<dd>
<p>
Create a serial device inside the VM (n is 0 to 3)
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--shares</span> <span class="monospaced">&lt;integer&gt; (0 - 50000)</span> (<em>default =</em> <span class="monospaced">1000</span>)
</dt>
<dd>
<p>
Amount of memory shares for auto-ballooning. The larger the number is, the more memory this VM gets. Number is relative to weights of all other running VMs. Using zero disables auto-ballooning. Auto-ballooning is done by pvestatd.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--skiplock</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Ignore locks - only root is allowed to use this option.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--smbios1</span> <span class="monospaced">[base64=&lt;1|0&gt;] [,family=&lt;Base64 encoded string&gt;] [,manufacturer=&lt;Base64 encoded string&gt;] [,product=&lt;Base64 encoded string&gt;] [,serial=&lt;Base64 encoded string&gt;] [,sku=&lt;Base64 encoded string&gt;] [,uuid=&lt;UUID&gt;] [,version=&lt;Base64 encoded string&gt;]</span> 
</dt>
<dd>
<p>
Specify SMBIOS type 1 fields.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--smp</span> <span class="monospaced">&lt;integer&gt; (1 - N)</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
The number of CPUs. Please use option -sockets instead.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--sockets</span> <span class="monospaced">&lt;integer&gt; (1 - N)</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
The number of CPU sockets.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--spice_enhancements</span> <span class="monospaced">[foldersharing=&lt;1|0&gt;] [,videostreaming=&lt;off|all|filter&gt;]</span> 
</dt>
<dd>
<p>
Configure additional enhancements for SPICE.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--sshkeys</span> <span class="monospaced">&lt;filepath&gt;</span> 
</dt>
<dd>
<p>
cloud-init: Setup public SSH keys (one key per line, OpenSSH format).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--startdate</span> <span class="monospaced">(now | YYYY-MM-DD | YYYY-MM-DDTHH:MM:SS)</span> (<em>default =</em> <span class="monospaced">now</span>)
</dt>
<dd>
<p>
Set the initial date of the real time clock. Valid format for date are:'now' or <em>2006-06-17T16:01:21</em> or <em>2006-06-17</em>.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--startup</span> `[[order=]\d+] [,up=\d+] [,down=\d+] ` 
</dt>
<dd>
<p>
Startup and shutdown behavior. Order is a non-negative number defining the general startup order. Shutdown in done with reverse ordering. Additionally you can set the <em>up</em> or <em>down</em> delay in seconds, which specifies a delay to wait before the next VM is started or stopped.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--tablet</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Enable/disable the USB tablet device.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--tags</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Tags of the VM. This is only meta information.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--tdf</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Enable/disable time drift fix.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--template</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Enable/disable Template.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--tpmstate0</span> <span class="monospaced">[file=]&lt;volume&gt; [,import-from=&lt;source volume&gt;] [,size=&lt;DiskSize&gt;] [,version=&lt;v1.2|v2.0&gt;]</span> 
</dt>
<dd>
<p>
Configure a Disk for storing TPM state. The format is fixed to <em>raw</em>. Use the special syntax STORAGE_ID:SIZE_IN_GiB to allocate a new volume. Note that SIZE_IN_GiB is ignored here and 4 MiB will be used instead. Use STORAGE_ID:0 and the <em>import-from</em> parameter to import from an existing volume.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--unused[n]</span> <span class="monospaced">[file=]&lt;volume&gt;</span> 
</dt>
<dd>
<p>
Reference to unused volumes. This is used internally, and should not be modified manually.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--usb[n]</span> <span class="monospaced">[[host=]&lt;HOSTUSBDEVICE|spice&gt;] [,mapping=&lt;mapping-id&gt;] [,usb3=&lt;1|0&gt;]</span> 
</dt>
<dd>
<p>
Configure an USB device (n is 0 to 4, for machine version &gt;= 7.1 and ostype l26 or windows &gt; 7, n can be up to 14).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--vcpus</span> <span class="monospaced">&lt;integer&gt; (1 - N)</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Number of hotplugged vcpus.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--vga</span> <span class="monospaced">[[type=]&lt;enum&gt;] [,clipboard=&lt;vnc&gt;] [,memory=&lt;integer&gt;]</span> 
</dt>
<dd>
<p>
Configure the VGA hardware.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--virtio[n]</span> <span class="monospaced">[file=]&lt;volume&gt; [,aio=&lt;native|threads|io_uring&gt;] [,backup=&lt;1|0&gt;] [,bps=&lt;bps&gt;] [,bps_max_length=&lt;seconds&gt;] [,bps_rd=&lt;bps&gt;] [,bps_rd_max_length=&lt;seconds&gt;] [,bps_wr=&lt;bps&gt;] [,bps_wr_max_length=&lt;seconds&gt;] [,cache=&lt;enum&gt;] [,cyls=&lt;integer&gt;] [,detect_zeroes=&lt;1|0&gt;] [,discard=&lt;ignore|on&gt;] [,format=&lt;enum&gt;] [,heads=&lt;integer&gt;] [,import-from=&lt;source volume&gt;] [,iops=&lt;iops&gt;] [,iops_max=&lt;iops&gt;] [,iops_max_length=&lt;seconds&gt;] [,iops_rd=&lt;iops&gt;] [,iops_rd_max=&lt;iops&gt;] [,iops_rd_max_length=&lt;seconds&gt;] [,iops_wr=&lt;iops&gt;] [,iops_wr_max=&lt;iops&gt;] [,iops_wr_max_length=&lt;seconds&gt;] [,iothread=&lt;1|0&gt;] [,mbps=&lt;mbps&gt;] [,mbps_max=&lt;mbps&gt;] [,mbps_rd=&lt;mbps&gt;] [,mbps_rd_max=&lt;mbps&gt;] [,mbps_wr=&lt;mbps&gt;] [,mbps_wr_max=&lt;mbps&gt;] [,media=&lt;cdrom|disk&gt;] [,replicate=&lt;1|0&gt;] [,rerror=&lt;ignore|report|stop&gt;] [,ro=&lt;1|0&gt;] [,secs=&lt;integer&gt;] [,serial=&lt;serial&gt;] [,shared=&lt;1|0&gt;] [,size=&lt;DiskSize&gt;] [,snapshot=&lt;1|0&gt;] [,trans=&lt;none|lba|auto&gt;] [,werror=&lt;enum&gt;]</span> 
</dt>
<dd>
<p>
Use volume as VIRTIO hard disk (n is 0 to 15). Use the special syntax STORAGE_ID:SIZE_IN_GiB to allocate a new volume. Use STORAGE_ID:0 and the <em>import-from</em> parameter to import from an existing volume.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--vmgenid</span> <span class="monospaced">&lt;UUID&gt;</span> (<em>default =</em> <span class="monospaced">1 (autogenerated)</span>)
</dt>
<dd>
<p>
Set VM Generation ID. Use <em>1</em> to autogenerate on create or update, pass <em>0</em> to disable explicitly.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--vmstatestorage</span> <span class="monospaced">&lt;storage ID&gt;</span> 
</dt>
<dd>
<p>
Default storage for VM state volumes/files.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--watchdog</span> <span class="monospaced">[[model=]&lt;i6300esb|ib700&gt;] [,action=&lt;enum&gt;]</span> 
</dt>
<dd>
<p>
Create a virtual hardware watchdog device.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm showcmd</strong> <span class="monospaced">&lt;vmid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Show command line which is used to start the VM (debug info).</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--pretty</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Puts each option on a new line to enhance human readability
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--snapshot</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Fetch config values from given snapshot.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm shutdown</strong> <span class="monospaced">&lt;vmid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Shutdown virtual machine. This is similar to pressing the power button on a
physical machine. This will send an ACPI event for the guest OS, which
should then proceed to a clean shutdown.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--forceStop</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Make sure the VM stops.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--keepActive</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Do not deactivate storage volumes.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--skiplock</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Ignore locks - only root is allowed to use this option.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--timeout</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> 
</dt>
<dd>
<p>
Wait maximal timeout seconds.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm snapshot</strong> <span class="monospaced">&lt;vmid&gt; &lt;snapname&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Snapshot a VM.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;snapname&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The name of the snapshot.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--description</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
A textual description or comment.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--vmstate</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Save the vmstate
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm start</strong> <span class="monospaced">&lt;vmid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Start virtual machine.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--force-cpu</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Override QEMU’s -cpu argument with the given string.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--machine</span> <span class="monospaced">[[type=]&lt;machine type&gt;] [,viommu=&lt;intel|virtio&gt;]</span> 
</dt>
<dd>
<p>
Specify the QEMU machine.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--migratedfrom</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The cluster node name.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--migration_network</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
CIDR of the (sub) network that is used for migration.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--migration_type</span> <span class="monospaced">&lt;insecure | secure&gt;</span> 
</dt>
<dd>
<p>
Migration traffic is encrypted using an SSH tunnel by default. On secure, completely private networks this can be disabled to increase performance.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--skiplock</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Ignore locks - only root is allowed to use this option.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--stateuri</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Some command save/restore state from this location.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--targetstorage</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Mapping from source to target storages. Providing only a single storage ID maps all source storages to that storage. Providing the special value <em>1</em> will map each source storage to itself.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--timeout</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">max(30, vm memory in GiB)</span>)
</dt>
<dd>
<p>
Wait maximal timeout seconds.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm status</strong> <span class="monospaced">&lt;vmid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Show VM status.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--verbose</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Verbose output format
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm stop</strong> <span class="monospaced">&lt;vmid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Stop virtual machine. The qemu process will exit immediately. This is akin
to pulling the power plug of a running computer and may damage the VM data.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--keepActive</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Do not deactivate storage volumes.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--migratedfrom</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The cluster node name.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--overrule-shutdown</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Try to abort active <em>qmshutdown</em> tasks before stopping.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--skiplock</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Ignore locks - only root is allowed to use this option.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--timeout</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> 
</dt>
<dd>
<p>
Wait maximal timeout seconds.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm suspend</strong> <span class="monospaced">&lt;vmid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Suspend virtual machine.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--skiplock</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Ignore locks - only root is allowed to use this option.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--statestorage</span> <span class="monospaced">&lt;storage ID&gt;</span> 
</dt>
<dd>
<p>
The storage for the VM state
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Requires option(s): <span class="monospaced">todisk</span></td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">--todisk</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
If set, suspends the VM to disk. Will be resumed on next VM start.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm template</strong> <span class="monospaced">&lt;vmid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Create a Template.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--disk</span> <span class="monospaced">&lt;efidisk0 | ide0 | ide1 | ide2 | ide3 | sata0 | sata1 | sata2 | sata3 | sata4 | sata5 | scsi0 | scsi1 | scsi10 | scsi11 | scsi12 | scsi13 | scsi14 | scsi15 | scsi16 | scsi17 | scsi18 | scsi19 | scsi2 | scsi20 | scsi21 | scsi22 | scsi23 | scsi24 | scsi25 | scsi26 | scsi27 | scsi28 | scsi29 | scsi3 | scsi30 | scsi4 | scsi5 | scsi6 | scsi7 | scsi8 | scsi9 | tpmstate0 | virtio0 | virtio1 | virtio10 | virtio11 | virtio12 | virtio13 | virtio14 | virtio15 | virtio2 | virtio3 | virtio4 | virtio5 | virtio6 | virtio7 | virtio8 | virtio9&gt;</span> 
</dt>
<dd>
<p>
If you want to convert only 1 disk to base image.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm terminal</strong> <span class="monospaced">&lt;vmid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Open a terminal using a serial device (The VM need to have a serial device
configured, for example <em>serial0: socket</em>)</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--escape</span> <span class="monospaced">&lt;string&gt;</span> (<em>default =</em> <span class="monospaced">^O</span>)
</dt>
<dd>
<p>
Escape character.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--iface</span> <span class="monospaced">&lt;serial0 | serial1 | serial2 | serial3&gt;</span> 
</dt>
<dd>
<p>
Select the serial device. By default we simply use the first suitable device.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm unlink</strong></p></div>
<div class="paragraph">
<p>An alias for <em>qm disk unlink</em>.</p></div>
<div class="paragraph">
<p><strong>qm unlock</strong> <span class="monospaced">&lt;vmid&gt;</span></p></div>
<div class="paragraph">
<p>Unlock the VM.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm vncproxy</strong> <span class="monospaced">&lt;vmid&gt;</span></p></div>
<div class="paragraph">
<p>Proxy VM VNC traffic to stdin/stdout</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>qm wait</strong> <span class="monospaced">&lt;vmid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Wait until the VM is stopped.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--timeout</span> <span class="monospaced">&lt;integer&gt; (1 - N)</span> 
</dt>
<dd>
<p>
Timeout in seconds. Default is to wait forever.
</p>
</dd>
</dl></div>
</div>
<div class="sect2">
<h3 id="_strong_qmrestore_strong_restore_qemuserver_span_class_monospaced_vzdump_span_backups">
<span>22.9. <strong>qmrestore</strong> - Restore QemuServer <span class="monospaced">vzdump</span> Backups</span>
 <a class="headerlink" href="#_strong_qmrestore_strong_restore_qemuserver_span_class_monospaced_vzdump_span_backups" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p><strong>qmrestore</strong> <span class="monospaced">help</span></p></div>
<div class="paragraph">
<p><strong>qmrestore</strong> <span class="monospaced">&lt;archive&gt; &lt;vmid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Restore QemuServer vzdump backups.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;archive&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The backup file. You can pass <em>-</em> to read from standard input.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--bwlimit</span> <span class="monospaced">&lt;number&gt; (0 - N)</span> 
</dt>
<dd>
<p>
Override I/O bandwidth limit (in KiB/s).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--force</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Allow to overwrite existing VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--live-restore</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Start the VM immediately from the backup and restore in background. PBS only.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--pool</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Add the VM to the specified pool.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--storage</span> <span class="monospaced">&lt;storage ID&gt;</span> 
</dt>
<dd>
<p>
Default storage.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--unique</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Assign a unique random ethernet address.
</p>
</dd>
</dl></div>
</div>
<div class="sect2">
<h3 id="_strong_pct_strong_proxmox_container_toolkit">
<span>22.10. <strong>pct</strong> - Proxmox Container Toolkit</span>
 <a class="headerlink" href="#_strong_pct_strong_proxmox_container_toolkit" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p><strong>pct</strong> <span class="monospaced">&lt;COMMAND&gt; [ARGS] [OPTIONS]</span></p></div>
<div class="paragraph">
<p><strong>pct clone</strong> <span class="monospaced">&lt;vmid&gt; &lt;newid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Create a container clone/copy</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;newid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
VMID for the clone.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--bwlimit</span> <span class="monospaced">&lt;number&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">clone limit from datacenter or storage config</span>)
</dt>
<dd>
<p>
Override I/O bandwidth limit (in KiB/s).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--description</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Description for the new CT.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--full</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Create a full copy of all disks. This is always done when you clone a normal CT. For CT templates, we try to create a linked clone by default.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--hostname</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Set a hostname for the new CT.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--pool</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Add the new CT to the specified pool.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--snapname</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The name of the snapshot.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--storage</span> <span class="monospaced">&lt;storage ID&gt;</span> 
</dt>
<dd>
<p>
Target storage for full clone.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--target</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Target node. Only allowed if the original VM is on shared storage.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pct config</strong> <span class="monospaced">&lt;vmid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Get container configuration.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--current</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Get current values (instead of pending values).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--snapshot</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Fetch config values from given snapshot.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pct console</strong> <span class="monospaced">&lt;vmid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Launch a console for the specified container.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--escape</span> <span class="monospaced">\^?[a-z]</span> (<em>default =</em> <span class="monospaced">^a</span>)
</dt>
<dd>
<p>
Escape sequence prefix. For example to use &lt;Ctrl+b q&gt; as the escape sequence pass <em>^b</em>.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pct cpusets</strong></p></div>
<div class="paragraph">
<p>Print the list of assigned CPU sets.</p></div>
<div class="paragraph">
<p><strong>pct create</strong> <span class="monospaced">&lt;vmid&gt; &lt;ostemplate&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Create or restore a container.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;ostemplate&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The OS template or backup file.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--arch</span> <span class="monospaced">&lt;amd64 | arm64 | armhf | i386 | riscv32 | riscv64&gt;</span> (<em>default =</em> <span class="monospaced">amd64</span>)
</dt>
<dd>
<p>
OS architecture type.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--bwlimit</span> <span class="monospaced">&lt;number&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">restore limit from datacenter or storage config</span>)
</dt>
<dd>
<p>
Override I/O bandwidth limit (in KiB/s).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--cmode</span> <span class="monospaced">&lt;console | shell | tty&gt;</span> (<em>default =</em> <span class="monospaced">tty</span>)
</dt>
<dd>
<p>
Console mode. By default, the console command tries to open a connection to one of the available tty devices. By setting cmode to <em>console</em> it tries to attach to /dev/console instead. If you set cmode to <em>shell</em>, it simply invokes a shell inside the container (no login).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--console</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Attach a console device (/dev/console) to the container.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--cores</span> <span class="monospaced">&lt;integer&gt; (1 - 8192)</span> 
</dt>
<dd>
<p>
The number of cores assigned to the container. A container can use all available cores by default.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--cpulimit</span> <span class="monospaced">&lt;number&gt; (0 - 8192)</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Limit of CPU usage.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">If the computer has 2 CPUs, it has a total of <em>2</em> CPU time. Value <em>0</em> indicates no CPU limit.</td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">--cpuunits</span> <span class="monospaced">&lt;integer&gt; (0 - 500000)</span> (<em>default =</em> <span class="monospaced">cgroup v1: 1024, cgroup v2: 100</span>)
</dt>
<dd>
<p>
CPU weight for a container, will be clamped to [1, 10000] in cgroup v2.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--debug</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Try to be more verbose. For now this only enables debug log-level on start.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--description</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Description for the Container. Shown in the web-interface CT’s summary. This is saved as comment inside the configuration file.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--dev[n]</span> <span class="monospaced">[[path=]&lt;Path&gt;] [,gid=&lt;integer&gt;] [,mode=&lt;Octal access mode&gt;] [,uid=&lt;integer&gt;]</span> 
</dt>
<dd>
<p>
Device to pass through to the container
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--features</span> <span class="monospaced">[force_rw_sys=&lt;1|0&gt;] [,fuse=&lt;1|0&gt;] [,keyctl=&lt;1|0&gt;] [,mknod=&lt;1|0&gt;] [,mount=&lt;fstype;fstype;...&gt;] [,nesting=&lt;1|0&gt;]</span> 
</dt>
<dd>
<p>
Allow containers access to advanced features.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--force</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Allow to overwrite existing container.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--hookscript</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Script that will be exectued during various steps in the containers lifetime.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--hostname</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Set a host name for the container.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--ignore-unpack-errors</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Ignore errors when extracting the template.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--lock</span> <span class="monospaced">&lt;backup | create | destroyed | disk | fstrim | migrate | mounted | rollback | snapshot | snapshot-delete&gt;</span> 
</dt>
<dd>
<p>
Lock/unlock the container.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--memory</span> <span class="monospaced">&lt;integer&gt; (16 - N)</span> (<em>default =</em> <span class="monospaced">512</span>)
</dt>
<dd>
<p>
Amount of RAM for the container in MB.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--mp[n]</span> <span class="monospaced">[volume=]&lt;volume&gt; ,mp=&lt;Path&gt; [,acl=&lt;1|0&gt;] [,backup=&lt;1|0&gt;] [,mountoptions=&lt;opt[;opt...]&gt;] [,quota=&lt;1|0&gt;] [,replicate=&lt;1|0&gt;] [,ro=&lt;1|0&gt;] [,shared=&lt;1|0&gt;] [,size=&lt;DiskSize&gt;]</span> 
</dt>
<dd>
<p>
Use volume as container mount point. Use the special syntax STORAGE_ID:SIZE_IN_GiB to allocate a new volume.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--nameserver</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Sets DNS server IP address for a container. Create will automatically use the setting from the host if you neither set searchdomain nor nameserver.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--net[n]</span> <span class="monospaced">name=&lt;string&gt; [,bridge=&lt;bridge&gt;] [,firewall=&lt;1|0&gt;] [,gw=&lt;GatewayIPv4&gt;] [,gw6=&lt;GatewayIPv6&gt;] [,hwaddr=&lt;XX:XX:XX:XX:XX:XX&gt;] [,ip=&lt;(IPv4/CIDR|dhcp|manual)&gt;] [,ip6=&lt;(IPv6/CIDR|auto|dhcp|manual)&gt;] [,link_down=&lt;1|0&gt;] [,mtu=&lt;integer&gt;] [,rate=&lt;mbps&gt;] [,tag=&lt;integer&gt;] [,trunks=&lt;vlanid[;vlanid...]&gt;] [,type=&lt;veth&gt;]</span> 
</dt>
<dd>
<p>
Specifies network interfaces for the container.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--onboot</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Specifies whether a container will be started during system bootup.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--ostype</span> <span class="monospaced">&lt;alpine | archlinux | centos | debian | devuan | fedora | gentoo | nixos | opensuse | ubuntu | unmanaged&gt;</span> 
</dt>
<dd>
<p>
OS type. This is used to setup configuration inside the container, and corresponds to lxc setup scripts in /usr/share/lxc/config/&lt;ostype&gt;.common.conf. Value <em>unmanaged</em> can be used to skip and OS specific setup.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--password</span> <span class="monospaced">&lt;password&gt;</span> 
</dt>
<dd>
<p>
Sets root password inside container.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--pool</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Add the VM to the specified pool.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--protection</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Sets the protection flag of the container. This will prevent the CT or CT’s disk remove/update operation.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--restore</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Mark this as restore task.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--rootfs</span> <span class="monospaced">[volume=]&lt;volume&gt; [,acl=&lt;1|0&gt;] [,mountoptions=&lt;opt[;opt...]&gt;] [,quota=&lt;1|0&gt;] [,replicate=&lt;1|0&gt;] [,ro=&lt;1|0&gt;] [,shared=&lt;1|0&gt;] [,size=&lt;DiskSize&gt;]</span> 
</dt>
<dd>
<p>
Use volume as container root.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--searchdomain</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Sets DNS search domains for a container. Create will automatically use the setting from the host if you neither set searchdomain nor nameserver.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--ssh-public-keys</span> <span class="monospaced">&lt;filepath&gt;</span> 
</dt>
<dd>
<p>
Setup public SSH keys (one key per line, OpenSSH format).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--start</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Start the CT after its creation finished successfully.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--startup</span> `[[order=]\d+] [,up=\d+] [,down=\d+] ` 
</dt>
<dd>
<p>
Startup and shutdown behavior. Order is a non-negative number defining the general startup order. Shutdown in done with reverse ordering. Additionally you can set the <em>up</em> or <em>down</em> delay in seconds, which specifies a delay to wait before the next VM is started or stopped.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--storage</span> <span class="monospaced">&lt;storage ID&gt;</span> (<em>default =</em> <span class="monospaced">local</span>)
</dt>
<dd>
<p>
Default Storage.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--swap</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">512</span>)
</dt>
<dd>
<p>
Amount of SWAP for the container in MB.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--tags</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Tags of the Container. This is only meta information.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--template</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Enable/disable Template.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--timezone</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Time zone to use in the container. If option isn’t set, then nothing will be done. Can be set to <em>host</em> to match the host time zone, or an arbitrary time zone option from /usr/share/zoneinfo/zone.tab
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--tty</span> <span class="monospaced">&lt;integer&gt; (0 - 6)</span> (<em>default =</em> <span class="monospaced">2</span>)
</dt>
<dd>
<p>
Specify the number of tty available to the container
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--unique</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Assign a unique random ethernet address.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Requires option(s): <span class="monospaced">restore</span></td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">--unprivileged</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Makes the container run as unprivileged user. (Should not be modified manually.)
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--unused[n]</span> <span class="monospaced">[volume=]&lt;volume&gt;</span> 
</dt>
<dd>
<p>
Reference to unused volumes. This is used internally, and should not be modified manually.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pct delsnapshot</strong> <span class="monospaced">&lt;vmid&gt; &lt;snapname&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Delete a LXC snapshot.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;snapname&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The name of the snapshot.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--force</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
For removal from config file, even if removing disk snapshots fails.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pct destroy</strong> <span class="monospaced">&lt;vmid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Destroy the container (also delete all uses files).</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--destroy-unreferenced-disks</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
If set, destroy additionally all disks with the VMID from all enabled storages which are not referenced in the config.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--force</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Force destroy, even if running.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--purge</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Remove container from all related configurations. For example, backup jobs, replication jobs or HA. Related ACLs and Firewall entries will <strong>always</strong> be removed.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pct df</strong> <span class="monospaced">&lt;vmid&gt;</span></p></div>
<div class="paragraph">
<p>Get the container’s current disk usage.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pct enter</strong> <span class="monospaced">&lt;vmid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Launch a shell for the specified container.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--keep-env</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Keep the current environment. This option will disabled by default with PVE 9. If you rely on a preserved environment, please use this option to be future-proof.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pct exec</strong> <span class="monospaced">&lt;vmid&gt; [&lt;extra-args&gt;]</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Launch a command inside the specified container.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;extra-args&gt;</span>: <span class="monospaced">&lt;array&gt;</span> 
</dt>
<dd>
<p>
Extra arguments as array
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--keep-env</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Keep the current environment. This option will disabled by default with PVE 9. If you rely on a preserved environment, please use this option to be future-proof.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pct fsck</strong> <span class="monospaced">&lt;vmid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Run a filesystem check (fsck) on a container volume.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--device</span> <span class="monospaced">&lt;mp0 | mp1 | mp10 | mp100 | mp101 | mp102 | mp103 | mp104 | mp105 | mp106 | mp107 | mp108 | mp109 | mp11 | mp110 | mp111 | mp112 | mp113 | mp114 | mp115 | mp116 | mp117 | mp118 | mp119 | mp12 | mp120 | mp121 | mp122 | mp123 | mp124 | mp125 | mp126 | mp127 | mp128 | mp129 | mp13 | mp130 | mp131 | mp132 | mp133 | mp134 | mp135 | mp136 | mp137 | mp138 | mp139 | mp14 | mp140 | mp141 | mp142 | mp143 | mp144 | mp145 | mp146 | mp147 | mp148 | mp149 | mp15 | mp150 | mp151 | mp152 | mp153 | mp154 | mp155 | mp156 | mp157 | mp158 | mp159 | mp16 | mp160 | mp161 | mp162 | mp163 | mp164 | mp165 | mp166 | mp167 | mp168 | mp169 | mp17 | mp170 | mp171 | mp172 | mp173 | mp174 | mp175 | mp176 | mp177 | mp178 | mp179 | mp18 | mp180 | mp181 | mp182 | mp183 | mp184 | mp185 | mp186 | mp187 | mp188 | mp189 | mp19 | mp190 | mp191 | mp192 | mp193 | mp194 | mp195 | mp196 | mp197 | mp198 | mp199 | mp2 | mp20 | mp200 | mp201 | mp202 | mp203 | mp204 | mp205 | mp206 | mp207 | mp208 | mp209 | mp21 | mp210 | mp211 | mp212 | mp213 | mp214 | mp215 | mp216 | mp217 | mp218 | mp219 | mp22 | mp220 | mp221 | mp222 | mp223 | mp224 | mp225 | mp226 | mp227 | mp228 | mp229 | mp23 | mp230 | mp231 | mp232 | mp233 | mp234 | mp235 | mp236 | mp237 | mp238 | mp239 | mp24 | mp240 | mp241 | mp242 | mp243 | mp244 | mp245 | mp246 | mp247 | mp248 | mp249 | mp25 | mp250 | mp251 | mp252 | mp253 | mp254 | mp255 | mp26 | mp27 | mp28 | mp29 | mp3 | mp30 | mp31 | mp32 | mp33 | mp34 | mp35 | mp36 | mp37 | mp38 | mp39 | mp4 | mp40 | mp41 | mp42 | mp43 | mp44 | mp45 | mp46 | mp47 | mp48 | mp49 | mp5 | mp50 | mp51 | mp52 | mp53 | mp54 | mp55 | mp56 | mp57 | mp58 | mp59 | mp6 | mp60 | mp61 | mp62 | mp63 | mp64 | mp65 | mp66 | mp67 | mp68 | mp69 | mp7 | mp70 | mp71 | mp72 | mp73 | mp74 | mp75 | mp76 | mp77 | mp78 | mp79 | mp8 | mp80 | mp81 | mp82 | mp83 | mp84 | mp85 | mp86 | mp87 | mp88 | mp89 | mp9 | mp90 | mp91 | mp92 | mp93 | mp94 | mp95 | mp96 | mp97 | mp98 | mp99 | rootfs&gt;</span> 
</dt>
<dd>
<p>
A volume on which to run the filesystem check
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--force</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Force checking, even if the filesystem seems clean
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pct fstrim</strong> <span class="monospaced">&lt;vmid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Run fstrim on a chosen CT and its mountpoints, except bind or read-only
mountpoints.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--ignore-mountpoints</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Skip all mountpoints, only do fstrim on the container root.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pct help</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Get help about specified command.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--extra-args</span> <span class="monospaced">&lt;array&gt;</span> 
</dt>
<dd>
<p>
Shows help for a specific command
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--verbose</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Verbose output format.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pct list</strong></p></div>
<div class="paragraph">
<p>LXC container index (per node).</p></div>
<div class="paragraph">
<p><strong>pct listsnapshot</strong> <span class="monospaced">&lt;vmid&gt;</span></p></div>
<div class="paragraph">
<p>List all snapshots.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pct migrate</strong> <span class="monospaced">&lt;vmid&gt; &lt;target&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Migrate the container to another node. Creates a new migration task.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;target&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Target node.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--bwlimit</span> <span class="monospaced">&lt;number&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">migrate limit from datacenter or storage config</span>)
</dt>
<dd>
<p>
Override I/O bandwidth limit (in KiB/s).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--online</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Use online/live migration.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--restart</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Use restart migration
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--target-storage</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Mapping from source to target storages. Providing only a single storage ID maps all source storages to that storage. Providing the special value <em>1</em> will map each source storage to itself.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--timeout</span> <span class="monospaced">&lt;integer&gt;</span> (<em>default =</em> <span class="monospaced">180</span>)
</dt>
<dd>
<p>
Timeout in seconds for shutdown for restart migration
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pct mount</strong> <span class="monospaced">&lt;vmid&gt;</span></p></div>
<div class="paragraph">
<p>Mount the container’s filesystem on the host. This will hold a lock on the
container and is meant for emergency maintenance only as it will prevent
further operations on the container other than start and stop.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pct move-volume</strong> <span class="monospaced">&lt;vmid&gt; &lt;volume&gt; [&lt;storage&gt;] [&lt;target-vmid&gt;] [&lt;target-volume&gt;]</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Move a rootfs-/mp-volume to a different storage or to a different
container.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;volume&gt;</span>: <span class="monospaced">&lt;mp0 | mp1 | mp10 | mp100 | mp101 | mp102 | mp103 | mp104 | mp105 | mp106 | mp107 | mp108 | mp109 | mp11 | mp110 | mp111 | mp112 | mp113 | mp114 | mp115 | mp116 | mp117 | mp118 | mp119 | mp12 | mp120 | mp121 | mp122 | mp123 | mp124 | mp125 | mp126 | mp127 | mp128 | mp129 | mp13 | mp130 | mp131 | mp132 | mp133 | mp134 | mp135 | mp136 | mp137 | mp138 | mp139 | mp14 | mp140 | mp141 | mp142 | mp143 | mp144 | mp145 | mp146 | mp147 | mp148 | mp149 | mp15 | mp150 | mp151 | mp152 | mp153 | mp154 | mp155 | mp156 | mp157 | mp158 | mp159 | mp16 | mp160 | mp161 | mp162 | mp163 | mp164 | mp165 | mp166 | mp167 | mp168 | mp169 | mp17 | mp170 | mp171 | mp172 | mp173 | mp174 | mp175 | mp176 | mp177 | mp178 | mp179 | mp18 | mp180 | mp181 | mp182 | mp183 | mp184 | mp185 | mp186 | mp187 | mp188 | mp189 | mp19 | mp190 | mp191 | mp192 | mp193 | mp194 | mp195 | mp196 | mp197 | mp198 | mp199 | mp2 | mp20 | mp200 | mp201 | mp202 | mp203 | mp204 | mp205 | mp206 | mp207 | mp208 | mp209 | mp21 | mp210 | mp211 | mp212 | mp213 | mp214 | mp215 | mp216 | mp217 | mp218 | mp219 | mp22 | mp220 | mp221 | mp222 | mp223 | mp224 | mp225 | mp226 | mp227 | mp228 | mp229 | mp23 | mp230 | mp231 | mp232 | mp233 | mp234 | mp235 | mp236 | mp237 | mp238 | mp239 | mp24 | mp240 | mp241 | mp242 | mp243 | mp244 | mp245 | mp246 | mp247 | mp248 | mp249 | mp25 | mp250 | mp251 | mp252 | mp253 | mp254 | mp255 | mp26 | mp27 | mp28 | mp29 | mp3 | mp30 | mp31 | mp32 | mp33 | mp34 | mp35 | mp36 | mp37 | mp38 | mp39 | mp4 | mp40 | mp41 | mp42 | mp43 | mp44 | mp45 | mp46 | mp47 | mp48 | mp49 | mp5 | mp50 | mp51 | mp52 | mp53 | mp54 | mp55 | mp56 | mp57 | mp58 | mp59 | mp6 | mp60 | mp61 | mp62 | mp63 | mp64 | mp65 | mp66 | mp67 | mp68 | mp69 | mp7 | mp70 | mp71 | mp72 | mp73 | mp74 | mp75 | mp76 | mp77 | mp78 | mp79 | mp8 | mp80 | mp81 | mp82 | mp83 | mp84 | mp85 | mp86 | mp87 | mp88 | mp89 | mp9 | mp90 | mp91 | mp92 | mp93 | mp94 | mp95 | mp96 | mp97 | mp98 | mp99 | rootfs | unused0 | unused1 | unused10 | unused100 | unused101 | unused102 | unused103 | unused104 | unused105 | unused106 | unused107 | unused108 | unused109 | unused11 | unused110 | unused111 | unused112 | unused113 | unused114 | unused115 | unused116 | unused117 | unused118 | unused119 | unused12 | unused120 | unused121 | unused122 | unused123 | unused124 | unused125 | unused126 | unused127 | unused128 | unused129 | unused13 | unused130 | unused131 | unused132 | unused133 | unused134 | unused135 | unused136 | unused137 | unused138 | unused139 | unused14 | unused140 | unused141 | unused142 | unused143 | unused144 | unused145 | unused146 | unused147 | unused148 | unused149 | unused15 | unused150 | unused151 | unused152 | unused153 | unused154 | unused155 | unused156 | unused157 | unused158 | unused159 | unused16 | unused160 | unused161 | unused162 | unused163 | unused164 | unused165 | unused166 | unused167 | unused168 | unused169 | unused17 | unused170 | unused171 | unused172 | unused173 | unused174 | unused175 | unused176 | unused177 | unused178 | unused179 | unused18 | unused180 | unused181 | unused182 | unused183 | unused184 | unused185 | unused186 | unused187 | unused188 | unused189 | unused19 | unused190 | unused191 | unused192 | unused193 | unused194 | unused195 | unused196 | unused197 | unused198 | unused199 | unused2 | unused20 | unused200 | unused201 | unused202 | unused203 | unused204 | unused205 | unused206 | unused207 | unused208 | unused209 | unused21 | unused210 | unused211 | unused212 | unused213 | unused214 | unused215 | unused216 | unused217 | unused218 | unused219 | unused22 | unused220 | unused221 | unused222 | unused223 | unused224 | unused225 | unused226 | unused227 | unused228 | unused229 | unused23 | unused230 | unused231 | unused232 | unused233 | unused234 | unused235 | unused236 | unused237 | unused238 | unused239 | unused24 | unused240 | unused241 | unused242 | unused243 | unused244 | unused245 | unused246 | unused247 | unused248 | unused249 | unused25 | unused250 | unused251 | unused252 | unused253 | unused254 | unused255 | unused26 | unused27 | unused28 | unused29 | unused3 | unused30 | unused31 | unused32 | unused33 | unused34 | unused35 | unused36 | unused37 | unused38 | unused39 | unused4 | unused40 | unused41 | unused42 | unused43 | unused44 | unused45 | unused46 | unused47 | unused48 | unused49 | unused5 | unused50 | unused51 | unused52 | unused53 | unused54 | unused55 | unused56 | unused57 | unused58 | unused59 | unused6 | unused60 | unused61 | unused62 | unused63 | unused64 | unused65 | unused66 | unused67 | unused68 | unused69 | unused7 | unused70 | unused71 | unused72 | unused73 | unused74 | unused75 | unused76 | unused77 | unused78 | unused79 | unused8 | unused80 | unused81 | unused82 | unused83 | unused84 | unused85 | unused86 | unused87 | unused88 | unused89 | unused9 | unused90 | unused91 | unused92 | unused93 | unused94 | unused95 | unused96 | unused97 | unused98 | unused99&gt;</span> 
</dt>
<dd>
<p>
Volume which will be moved.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;storage&gt;</span>: <span class="monospaced">&lt;storage ID&gt;</span> 
</dt>
<dd>
<p>
Target Storage.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;target-vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;target-volume&gt;</span>: <span class="monospaced">&lt;mp0 | mp1 | mp10 | mp100 | mp101 | mp102 | mp103 | mp104 | mp105 | mp106 | mp107 | mp108 | mp109 | mp11 | mp110 | mp111 | mp112 | mp113 | mp114 | mp115 | mp116 | mp117 | mp118 | mp119 | mp12 | mp120 | mp121 | mp122 | mp123 | mp124 | mp125 | mp126 | mp127 | mp128 | mp129 | mp13 | mp130 | mp131 | mp132 | mp133 | mp134 | mp135 | mp136 | mp137 | mp138 | mp139 | mp14 | mp140 | mp141 | mp142 | mp143 | mp144 | mp145 | mp146 | mp147 | mp148 | mp149 | mp15 | mp150 | mp151 | mp152 | mp153 | mp154 | mp155 | mp156 | mp157 | mp158 | mp159 | mp16 | mp160 | mp161 | mp162 | mp163 | mp164 | mp165 | mp166 | mp167 | mp168 | mp169 | mp17 | mp170 | mp171 | mp172 | mp173 | mp174 | mp175 | mp176 | mp177 | mp178 | mp179 | mp18 | mp180 | mp181 | mp182 | mp183 | mp184 | mp185 | mp186 | mp187 | mp188 | mp189 | mp19 | mp190 | mp191 | mp192 | mp193 | mp194 | mp195 | mp196 | mp197 | mp198 | mp199 | mp2 | mp20 | mp200 | mp201 | mp202 | mp203 | mp204 | mp205 | mp206 | mp207 | mp208 | mp209 | mp21 | mp210 | mp211 | mp212 | mp213 | mp214 | mp215 | mp216 | mp217 | mp218 | mp219 | mp22 | mp220 | mp221 | mp222 | mp223 | mp224 | mp225 | mp226 | mp227 | mp228 | mp229 | mp23 | mp230 | mp231 | mp232 | mp233 | mp234 | mp235 | mp236 | mp237 | mp238 | mp239 | mp24 | mp240 | mp241 | mp242 | mp243 | mp244 | mp245 | mp246 | mp247 | mp248 | mp249 | mp25 | mp250 | mp251 | mp252 | mp253 | mp254 | mp255 | mp26 | mp27 | mp28 | mp29 | mp3 | mp30 | mp31 | mp32 | mp33 | mp34 | mp35 | mp36 | mp37 | mp38 | mp39 | mp4 | mp40 | mp41 | mp42 | mp43 | mp44 | mp45 | mp46 | mp47 | mp48 | mp49 | mp5 | mp50 | mp51 | mp52 | mp53 | mp54 | mp55 | mp56 | mp57 | mp58 | mp59 | mp6 | mp60 | mp61 | mp62 | mp63 | mp64 | mp65 | mp66 | mp67 | mp68 | mp69 | mp7 | mp70 | mp71 | mp72 | mp73 | mp74 | mp75 | mp76 | mp77 | mp78 | mp79 | mp8 | mp80 | mp81 | mp82 | mp83 | mp84 | mp85 | mp86 | mp87 | mp88 | mp89 | mp9 | mp90 | mp91 | mp92 | mp93 | mp94 | mp95 | mp96 | mp97 | mp98 | mp99 | rootfs | unused0 | unused1 | unused10 | unused100 | unused101 | unused102 | unused103 | unused104 | unused105 | unused106 | unused107 | unused108 | unused109 | unused11 | unused110 | unused111 | unused112 | unused113 | unused114 | unused115 | unused116 | unused117 | unused118 | unused119 | unused12 | unused120 | unused121 | unused122 | unused123 | unused124 | unused125 | unused126 | unused127 | unused128 | unused129 | unused13 | unused130 | unused131 | unused132 | unused133 | unused134 | unused135 | unused136 | unused137 | unused138 | unused139 | unused14 | unused140 | unused141 | unused142 | unused143 | unused144 | unused145 | unused146 | unused147 | unused148 | unused149 | unused15 | unused150 | unused151 | unused152 | unused153 | unused154 | unused155 | unused156 | unused157 | unused158 | unused159 | unused16 | unused160 | unused161 | unused162 | unused163 | unused164 | unused165 | unused166 | unused167 | unused168 | unused169 | unused17 | unused170 | unused171 | unused172 | unused173 | unused174 | unused175 | unused176 | unused177 | unused178 | unused179 | unused18 | unused180 | unused181 | unused182 | unused183 | unused184 | unused185 | unused186 | unused187 | unused188 | unused189 | unused19 | unused190 | unused191 | unused192 | unused193 | unused194 | unused195 | unused196 | unused197 | unused198 | unused199 | unused2 | unused20 | unused200 | unused201 | unused202 | unused203 | unused204 | unused205 | unused206 | unused207 | unused208 | unused209 | unused21 | unused210 | unused211 | unused212 | unused213 | unused214 | unused215 | unused216 | unused217 | unused218 | unused219 | unused22 | unused220 | unused221 | unused222 | unused223 | unused224 | unused225 | unused226 | unused227 | unused228 | unused229 | unused23 | unused230 | unused231 | unused232 | unused233 | unused234 | unused235 | unused236 | unused237 | unused238 | unused239 | unused24 | unused240 | unused241 | unused242 | unused243 | unused244 | unused245 | unused246 | unused247 | unused248 | unused249 | unused25 | unused250 | unused251 | unused252 | unused253 | unused254 | unused255 | unused26 | unused27 | unused28 | unused29 | unused3 | unused30 | unused31 | unused32 | unused33 | unused34 | unused35 | unused36 | unused37 | unused38 | unused39 | unused4 | unused40 | unused41 | unused42 | unused43 | unused44 | unused45 | unused46 | unused47 | unused48 | unused49 | unused5 | unused50 | unused51 | unused52 | unused53 | unused54 | unused55 | unused56 | unused57 | unused58 | unused59 | unused6 | unused60 | unused61 | unused62 | unused63 | unused64 | unused65 | unused66 | unused67 | unused68 | unused69 | unused7 | unused70 | unused71 | unused72 | unused73 | unused74 | unused75 | unused76 | unused77 | unused78 | unused79 | unused8 | unused80 | unused81 | unused82 | unused83 | unused84 | unused85 | unused86 | unused87 | unused88 | unused89 | unused9 | unused90 | unused91 | unused92 | unused93 | unused94 | unused95 | unused96 | unused97 | unused98 | unused99&gt;</span> 
</dt>
<dd>
<p>
The config key the volume will be moved to. Default is the source volume key.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--bwlimit</span> <span class="monospaced">&lt;number&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">clone limit from datacenter or storage config</span>)
</dt>
<dd>
<p>
Override I/O bandwidth limit (in KiB/s).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--delete</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Delete the original volume after successful copy. By default the original is kept as an unused volume entry.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--digest</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Prevent changes if current configuration file has different SHA1 " .
                    "digest. This can be used to prevent concurrent modifications.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--target-digest</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Prevent changes if current configuration file of the target " .
                    "container has a different SHA1 digest. This can be used to prevent " .
                    "concurrent modifications.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pct move_volume</strong></p></div>
<div class="paragraph">
<p>An alias for <em>pct move-volume</em>.</p></div>
<div class="paragraph">
<p><strong>pct pending</strong> <span class="monospaced">&lt;vmid&gt;</span></p></div>
<div class="paragraph">
<p>Get container configuration, including pending changes.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pct pull</strong> <span class="monospaced">&lt;vmid&gt; &lt;path&gt; &lt;destination&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Copy a file from the container to the local system.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;path&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Path to a file inside the container to pull.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;destination&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Destination
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--group</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Owner group name or id.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--perms</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
File permissions to use (octal by default, prefix with <em>0x</em> for hexadecimal).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--user</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Owner user name or id.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pct push</strong> <span class="monospaced">&lt;vmid&gt; &lt;file&gt; &lt;destination&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Copy a local file to the container.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;file&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Path to a local file.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;destination&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Destination inside the container to write to.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--group</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Owner group name or id. When using a name it must exist inside the container.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--perms</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
File permissions to use (octal by default, prefix with <em>0x</em> for hexadecimal).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--user</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Owner user name or id. When using a name it must exist inside the container.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pct reboot</strong> <span class="monospaced">&lt;vmid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Reboot the container by shutting it down, and starting it again. Applies
pending changes.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--timeout</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> 
</dt>
<dd>
<p>
Wait maximal timeout seconds for the shutdown.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pct remote-migrate</strong> <span class="monospaced">&lt;vmid&gt; [&lt;target-vmid&gt;] &lt;target-endpoint&gt; --target-bridge &lt;string&gt; --target-storage &lt;string&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Migrate container to a remote cluster. Creates a new migration task.
EXPERIMENTAL feature!</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;target-vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;target-endpoint&gt;</span>: <span class="monospaced">apitoken=&lt;PVEAPIToken=user@realm!token=SECRET&gt; ,host=&lt;ADDRESS&gt; [,fingerprint=&lt;FINGERPRINT&gt;] [,port=&lt;PORT&gt;]</span> 
</dt>
<dd>
<p>
Remote target endpoint
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--bwlimit</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">migrate limit from datacenter or storage config</span>)
</dt>
<dd>
<p>
Override I/O bandwidth limit (in KiB/s).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--delete</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Delete the original CT and related data after successful migration. By default the original CT is kept on the source cluster in a stopped state.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--online</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Use online/live migration.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--restart</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Use restart migration
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--target-bridge</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Mapping from source to target bridges. Providing only a single bridge ID maps all source bridges to that bridge. Providing the special value <em>1</em> will map each source bridge to itself.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--target-storage</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Mapping from source to target storages. Providing only a single storage ID maps all source storages to that storage. Providing the special value <em>1</em> will map each source storage to itself.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--timeout</span> <span class="monospaced">&lt;integer&gt;</span> (<em>default =</em> <span class="monospaced">180</span>)
</dt>
<dd>
<p>
Timeout in seconds for shutdown for restart migration
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pct rescan</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Rescan all storages and update disk sizes and unused disk images.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--dryrun</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Do not actually write changes out to conifg.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--vmid</span> <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pct resize</strong> <span class="monospaced">&lt;vmid&gt; &lt;disk&gt; &lt;size&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Resize a container mount point.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;disk&gt;</span>: <span class="monospaced">&lt;mp0 | mp1 | mp10 | mp100 | mp101 | mp102 | mp103 | mp104 | mp105 | mp106 | mp107 | mp108 | mp109 | mp11 | mp110 | mp111 | mp112 | mp113 | mp114 | mp115 | mp116 | mp117 | mp118 | mp119 | mp12 | mp120 | mp121 | mp122 | mp123 | mp124 | mp125 | mp126 | mp127 | mp128 | mp129 | mp13 | mp130 | mp131 | mp132 | mp133 | mp134 | mp135 | mp136 | mp137 | mp138 | mp139 | mp14 | mp140 | mp141 | mp142 | mp143 | mp144 | mp145 | mp146 | mp147 | mp148 | mp149 | mp15 | mp150 | mp151 | mp152 | mp153 | mp154 | mp155 | mp156 | mp157 | mp158 | mp159 | mp16 | mp160 | mp161 | mp162 | mp163 | mp164 | mp165 | mp166 | mp167 | mp168 | mp169 | mp17 | mp170 | mp171 | mp172 | mp173 | mp174 | mp175 | mp176 | mp177 | mp178 | mp179 | mp18 | mp180 | mp181 | mp182 | mp183 | mp184 | mp185 | mp186 | mp187 | mp188 | mp189 | mp19 | mp190 | mp191 | mp192 | mp193 | mp194 | mp195 | mp196 | mp197 | mp198 | mp199 | mp2 | mp20 | mp200 | mp201 | mp202 | mp203 | mp204 | mp205 | mp206 | mp207 | mp208 | mp209 | mp21 | mp210 | mp211 | mp212 | mp213 | mp214 | mp215 | mp216 | mp217 | mp218 | mp219 | mp22 | mp220 | mp221 | mp222 | mp223 | mp224 | mp225 | mp226 | mp227 | mp228 | mp229 | mp23 | mp230 | mp231 | mp232 | mp233 | mp234 | mp235 | mp236 | mp237 | mp238 | mp239 | mp24 | mp240 | mp241 | mp242 | mp243 | mp244 | mp245 | mp246 | mp247 | mp248 | mp249 | mp25 | mp250 | mp251 | mp252 | mp253 | mp254 | mp255 | mp26 | mp27 | mp28 | mp29 | mp3 | mp30 | mp31 | mp32 | mp33 | mp34 | mp35 | mp36 | mp37 | mp38 | mp39 | mp4 | mp40 | mp41 | mp42 | mp43 | mp44 | mp45 | mp46 | mp47 | mp48 | mp49 | mp5 | mp50 | mp51 | mp52 | mp53 | mp54 | mp55 | mp56 | mp57 | mp58 | mp59 | mp6 | mp60 | mp61 | mp62 | mp63 | mp64 | mp65 | mp66 | mp67 | mp68 | mp69 | mp7 | mp70 | mp71 | mp72 | mp73 | mp74 | mp75 | mp76 | mp77 | mp78 | mp79 | mp8 | mp80 | mp81 | mp82 | mp83 | mp84 | mp85 | mp86 | mp87 | mp88 | mp89 | mp9 | mp90 | mp91 | mp92 | mp93 | mp94 | mp95 | mp96 | mp97 | mp98 | mp99 | rootfs&gt;</span> 
</dt>
<dd>
<p>
The disk you want to resize.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;size&gt;</span>: <span class="monospaced">\+?\d+(\.\d+)?[KMGT]?</span> 
</dt>
<dd>
<p>
The new size. With the <em>+</em> sign the value is added to the actual size of the volume and without it, the value is taken as an absolute one. Shrinking disk size is not supported.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--digest</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Prevent changes if current configuration file has different SHA1 digest. This can be used to prevent concurrent modifications.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pct restore</strong> <span class="monospaced">&lt;vmid&gt; &lt;ostemplate&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Create or restore a container.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;ostemplate&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The OS template or backup file.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--arch</span> <span class="monospaced">&lt;amd64 | arm64 | armhf | i386 | riscv32 | riscv64&gt;</span> (<em>default =</em> <span class="monospaced">amd64</span>)
</dt>
<dd>
<p>
OS architecture type.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--bwlimit</span> <span class="monospaced">&lt;number&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">restore limit from datacenter or storage config</span>)
</dt>
<dd>
<p>
Override I/O bandwidth limit (in KiB/s).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--cmode</span> <span class="monospaced">&lt;console | shell | tty&gt;</span> (<em>default =</em> <span class="monospaced">tty</span>)
</dt>
<dd>
<p>
Console mode. By default, the console command tries to open a connection to one of the available tty devices. By setting cmode to <em>console</em> it tries to attach to /dev/console instead. If you set cmode to <em>shell</em>, it simply invokes a shell inside the container (no login).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--console</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Attach a console device (/dev/console) to the container.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--cores</span> <span class="monospaced">&lt;integer&gt; (1 - 8192)</span> 
</dt>
<dd>
<p>
The number of cores assigned to the container. A container can use all available cores by default.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--cpulimit</span> <span class="monospaced">&lt;number&gt; (0 - 8192)</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Limit of CPU usage.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">If the computer has 2 CPUs, it has a total of <em>2</em> CPU time. Value <em>0</em> indicates no CPU limit.</td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">--cpuunits</span> <span class="monospaced">&lt;integer&gt; (0 - 500000)</span> (<em>default =</em> <span class="monospaced">cgroup v1: 1024, cgroup v2: 100</span>)
</dt>
<dd>
<p>
CPU weight for a container, will be clamped to [1, 10000] in cgroup v2.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--debug</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Try to be more verbose. For now this only enables debug log-level on start.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--description</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Description for the Container. Shown in the web-interface CT’s summary. This is saved as comment inside the configuration file.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--dev[n]</span> <span class="monospaced">[[path=]&lt;Path&gt;] [,gid=&lt;integer&gt;] [,mode=&lt;Octal access mode&gt;] [,uid=&lt;integer&gt;]</span> 
</dt>
<dd>
<p>
Device to pass through to the container
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--features</span> <span class="monospaced">[force_rw_sys=&lt;1|0&gt;] [,fuse=&lt;1|0&gt;] [,keyctl=&lt;1|0&gt;] [,mknod=&lt;1|0&gt;] [,mount=&lt;fstype;fstype;...&gt;] [,nesting=&lt;1|0&gt;]</span> 
</dt>
<dd>
<p>
Allow containers access to advanced features.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--force</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Allow to overwrite existing container.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--hookscript</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Script that will be exectued during various steps in the containers lifetime.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--hostname</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Set a host name for the container.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--ignore-unpack-errors</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Ignore errors when extracting the template.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--lock</span> <span class="monospaced">&lt;backup | create | destroyed | disk | fstrim | migrate | mounted | rollback | snapshot | snapshot-delete&gt;</span> 
</dt>
<dd>
<p>
Lock/unlock the container.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--memory</span> <span class="monospaced">&lt;integer&gt; (16 - N)</span> (<em>default =</em> <span class="monospaced">512</span>)
</dt>
<dd>
<p>
Amount of RAM for the container in MB.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--mp[n]</span> <span class="monospaced">[volume=]&lt;volume&gt; ,mp=&lt;Path&gt; [,acl=&lt;1|0&gt;] [,backup=&lt;1|0&gt;] [,mountoptions=&lt;opt[;opt...]&gt;] [,quota=&lt;1|0&gt;] [,replicate=&lt;1|0&gt;] [,ro=&lt;1|0&gt;] [,shared=&lt;1|0&gt;] [,size=&lt;DiskSize&gt;]</span> 
</dt>
<dd>
<p>
Use volume as container mount point. Use the special syntax STORAGE_ID:SIZE_IN_GiB to allocate a new volume.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--nameserver</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Sets DNS server IP address for a container. Create will automatically use the setting from the host if you neither set searchdomain nor nameserver.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--net[n]</span> <span class="monospaced">name=&lt;string&gt; [,bridge=&lt;bridge&gt;] [,firewall=&lt;1|0&gt;] [,gw=&lt;GatewayIPv4&gt;] [,gw6=&lt;GatewayIPv6&gt;] [,hwaddr=&lt;XX:XX:XX:XX:XX:XX&gt;] [,ip=&lt;(IPv4/CIDR|dhcp|manual)&gt;] [,ip6=&lt;(IPv6/CIDR|auto|dhcp|manual)&gt;] [,link_down=&lt;1|0&gt;] [,mtu=&lt;integer&gt;] [,rate=&lt;mbps&gt;] [,tag=&lt;integer&gt;] [,trunks=&lt;vlanid[;vlanid...]&gt;] [,type=&lt;veth&gt;]</span> 
</dt>
<dd>
<p>
Specifies network interfaces for the container.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--onboot</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Specifies whether a container will be started during system bootup.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--ostype</span> <span class="monospaced">&lt;alpine | archlinux | centos | debian | devuan | fedora | gentoo | nixos | opensuse | ubuntu | unmanaged&gt;</span> 
</dt>
<dd>
<p>
OS type. This is used to setup configuration inside the container, and corresponds to lxc setup scripts in /usr/share/lxc/config/&lt;ostype&gt;.common.conf. Value <em>unmanaged</em> can be used to skip and OS specific setup.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--password</span> <span class="monospaced">&lt;password&gt;</span> 
</dt>
<dd>
<p>
Sets root password inside container.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--pool</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Add the VM to the specified pool.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--protection</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Sets the protection flag of the container. This will prevent the CT or CT’s disk remove/update operation.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--rootfs</span> <span class="monospaced">[volume=]&lt;volume&gt; [,acl=&lt;1|0&gt;] [,mountoptions=&lt;opt[;opt...]&gt;] [,quota=&lt;1|0&gt;] [,replicate=&lt;1|0&gt;] [,ro=&lt;1|0&gt;] [,shared=&lt;1|0&gt;] [,size=&lt;DiskSize&gt;]</span> 
</dt>
<dd>
<p>
Use volume as container root.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--searchdomain</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Sets DNS search domains for a container. Create will automatically use the setting from the host if you neither set searchdomain nor nameserver.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--ssh-public-keys</span> <span class="monospaced">&lt;filepath&gt;</span> 
</dt>
<dd>
<p>
Setup public SSH keys (one key per line, OpenSSH format).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--start</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Start the CT after its creation finished successfully.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--startup</span> `[[order=]\d+] [,up=\d+] [,down=\d+] ` 
</dt>
<dd>
<p>
Startup and shutdown behavior. Order is a non-negative number defining the general startup order. Shutdown in done with reverse ordering. Additionally you can set the <em>up</em> or <em>down</em> delay in seconds, which specifies a delay to wait before the next VM is started or stopped.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--storage</span> <span class="monospaced">&lt;storage ID&gt;</span> (<em>default =</em> <span class="monospaced">local</span>)
</dt>
<dd>
<p>
Default Storage.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--swap</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">512</span>)
</dt>
<dd>
<p>
Amount of SWAP for the container in MB.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--tags</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Tags of the Container. This is only meta information.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--template</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Enable/disable Template.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--timezone</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Time zone to use in the container. If option isn’t set, then nothing will be done. Can be set to <em>host</em> to match the host time zone, or an arbitrary time zone option from /usr/share/zoneinfo/zone.tab
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--tty</span> <span class="monospaced">&lt;integer&gt; (0 - 6)</span> (<em>default =</em> <span class="monospaced">2</span>)
</dt>
<dd>
<p>
Specify the number of tty available to the container
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--unique</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Assign a unique random ethernet address.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Requires option(s): <span class="monospaced">restore</span></td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">--unprivileged</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Makes the container run as unprivileged user. (Should not be modified manually.)
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--unused[n]</span> <span class="monospaced">[volume=]&lt;volume&gt;</span> 
</dt>
<dd>
<p>
Reference to unused volumes. This is used internally, and should not be modified manually.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pct resume</strong> <span class="monospaced">&lt;vmid&gt;</span></p></div>
<div class="paragraph">
<p>Resume the container.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pct rollback</strong> <span class="monospaced">&lt;vmid&gt; &lt;snapname&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Rollback LXC state to specified snapshot.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;snapname&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The name of the snapshot.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--start</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Whether the container should get started after rolling back successfully
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pct set</strong> <span class="monospaced">&lt;vmid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Set container options.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--arch</span> <span class="monospaced">&lt;amd64 | arm64 | armhf | i386 | riscv32 | riscv64&gt;</span> (<em>default =</em> <span class="monospaced">amd64</span>)
</dt>
<dd>
<p>
OS architecture type.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--cmode</span> <span class="monospaced">&lt;console | shell | tty&gt;</span> (<em>default =</em> <span class="monospaced">tty</span>)
</dt>
<dd>
<p>
Console mode. By default, the console command tries to open a connection to one of the available tty devices. By setting cmode to <em>console</em> it tries to attach to /dev/console instead. If you set cmode to <em>shell</em>, it simply invokes a shell inside the container (no login).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--console</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Attach a console device (/dev/console) to the container.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--cores</span> <span class="monospaced">&lt;integer&gt; (1 - 8192)</span> 
</dt>
<dd>
<p>
The number of cores assigned to the container. A container can use all available cores by default.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--cpulimit</span> <span class="monospaced">&lt;number&gt; (0 - 8192)</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Limit of CPU usage.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">If the computer has 2 CPUs, it has a total of <em>2</em> CPU time. Value <em>0</em> indicates no CPU limit.</td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">--cpuunits</span> <span class="monospaced">&lt;integer&gt; (0 - 500000)</span> (<em>default =</em> <span class="monospaced">cgroup v1: 1024, cgroup v2: 100</span>)
</dt>
<dd>
<p>
CPU weight for a container, will be clamped to [1, 10000] in cgroup v2.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--debug</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Try to be more verbose. For now this only enables debug log-level on start.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--delete</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
A list of settings you want to delete.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--description</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Description for the Container. Shown in the web-interface CT’s summary. This is saved as comment inside the configuration file.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--dev[n]</span> <span class="monospaced">[[path=]&lt;Path&gt;] [,gid=&lt;integer&gt;] [,mode=&lt;Octal access mode&gt;] [,uid=&lt;integer&gt;]</span> 
</dt>
<dd>
<p>
Device to pass through to the container
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--digest</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Prevent changes if current configuration file has different SHA1 digest. This can be used to prevent concurrent modifications.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--features</span> <span class="monospaced">[force_rw_sys=&lt;1|0&gt;] [,fuse=&lt;1|0&gt;] [,keyctl=&lt;1|0&gt;] [,mknod=&lt;1|0&gt;] [,mount=&lt;fstype;fstype;...&gt;] [,nesting=&lt;1|0&gt;]</span> 
</dt>
<dd>
<p>
Allow containers access to advanced features.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--hookscript</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Script that will be exectued during various steps in the containers lifetime.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--hostname</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Set a host name for the container.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--lock</span> <span class="monospaced">&lt;backup | create | destroyed | disk | fstrim | migrate | mounted | rollback | snapshot | snapshot-delete&gt;</span> 
</dt>
<dd>
<p>
Lock/unlock the container.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--memory</span> <span class="monospaced">&lt;integer&gt; (16 - N)</span> (<em>default =</em> <span class="monospaced">512</span>)
</dt>
<dd>
<p>
Amount of RAM for the container in MB.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--mp[n]</span> <span class="monospaced">[volume=]&lt;volume&gt; ,mp=&lt;Path&gt; [,acl=&lt;1|0&gt;] [,backup=&lt;1|0&gt;] [,mountoptions=&lt;opt[;opt...]&gt;] [,quota=&lt;1|0&gt;] [,replicate=&lt;1|0&gt;] [,ro=&lt;1|0&gt;] [,shared=&lt;1|0&gt;] [,size=&lt;DiskSize&gt;]</span> 
</dt>
<dd>
<p>
Use volume as container mount point. Use the special syntax STORAGE_ID:SIZE_IN_GiB to allocate a new volume.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--nameserver</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Sets DNS server IP address for a container. Create will automatically use the setting from the host if you neither set searchdomain nor nameserver.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--net[n]</span> <span class="monospaced">name=&lt;string&gt; [,bridge=&lt;bridge&gt;] [,firewall=&lt;1|0&gt;] [,gw=&lt;GatewayIPv4&gt;] [,gw6=&lt;GatewayIPv6&gt;] [,hwaddr=&lt;XX:XX:XX:XX:XX:XX&gt;] [,ip=&lt;(IPv4/CIDR|dhcp|manual)&gt;] [,ip6=&lt;(IPv6/CIDR|auto|dhcp|manual)&gt;] [,link_down=&lt;1|0&gt;] [,mtu=&lt;integer&gt;] [,rate=&lt;mbps&gt;] [,tag=&lt;integer&gt;] [,trunks=&lt;vlanid[;vlanid...]&gt;] [,type=&lt;veth&gt;]</span> 
</dt>
<dd>
<p>
Specifies network interfaces for the container.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--onboot</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Specifies whether a container will be started during system bootup.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--ostype</span> <span class="monospaced">&lt;alpine | archlinux | centos | debian | devuan | fedora | gentoo | nixos | opensuse | ubuntu | unmanaged&gt;</span> 
</dt>
<dd>
<p>
OS type. This is used to setup configuration inside the container, and corresponds to lxc setup scripts in /usr/share/lxc/config/&lt;ostype&gt;.common.conf. Value <em>unmanaged</em> can be used to skip and OS specific setup.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--protection</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Sets the protection flag of the container. This will prevent the CT or CT’s disk remove/update operation.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--revert</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Revert a pending change.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--rootfs</span> <span class="monospaced">[volume=]&lt;volume&gt; [,acl=&lt;1|0&gt;] [,mountoptions=&lt;opt[;opt...]&gt;] [,quota=&lt;1|0&gt;] [,replicate=&lt;1|0&gt;] [,ro=&lt;1|0&gt;] [,shared=&lt;1|0&gt;] [,size=&lt;DiskSize&gt;]</span> 
</dt>
<dd>
<p>
Use volume as container root.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--searchdomain</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Sets DNS search domains for a container. Create will automatically use the setting from the host if you neither set searchdomain nor nameserver.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--startup</span> `[[order=]\d+] [,up=\d+] [,down=\d+] ` 
</dt>
<dd>
<p>
Startup and shutdown behavior. Order is a non-negative number defining the general startup order. Shutdown in done with reverse ordering. Additionally you can set the <em>up</em> or <em>down</em> delay in seconds, which specifies a delay to wait before the next VM is started or stopped.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--swap</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">512</span>)
</dt>
<dd>
<p>
Amount of SWAP for the container in MB.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--tags</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Tags of the Container. This is only meta information.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--template</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Enable/disable Template.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--timezone</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Time zone to use in the container. If option isn’t set, then nothing will be done. Can be set to <em>host</em> to match the host time zone, or an arbitrary time zone option from /usr/share/zoneinfo/zone.tab
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--tty</span> <span class="monospaced">&lt;integer&gt; (0 - 6)</span> (<em>default =</em> <span class="monospaced">2</span>)
</dt>
<dd>
<p>
Specify the number of tty available to the container
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--unprivileged</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Makes the container run as unprivileged user. (Should not be modified manually.)
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--unused[n]</span> <span class="monospaced">[volume=]&lt;volume&gt;</span> 
</dt>
<dd>
<p>
Reference to unused volumes. This is used internally, and should not be modified manually.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pct shutdown</strong> <span class="monospaced">&lt;vmid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Shutdown the container. This will trigger a clean shutdown of the
container, see lxc-stop(1) for details.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--forceStop</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Make sure the Container stops.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--timeout</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">60</span>)
</dt>
<dd>
<p>
Wait maximal timeout seconds.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pct snapshot</strong> <span class="monospaced">&lt;vmid&gt; &lt;snapname&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Snapshot a container.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;snapname&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The name of the snapshot.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--description</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
A textual description or comment.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pct start</strong> <span class="monospaced">&lt;vmid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Start the container.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--debug</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
If set, enables very verbose debug log-level on start.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--skiplock</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Ignore locks - only root is allowed to use this option.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pct status</strong> <span class="monospaced">&lt;vmid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Show CT status.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--verbose</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Verbose output format
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pct stop</strong> <span class="monospaced">&lt;vmid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Stop the container. This will abruptly stop all processes running in the
container.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--overrule-shutdown</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Try to abort active <em>vzshutdown</em> tasks before stopping.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--skiplock</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Ignore locks - only root is allowed to use this option.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pct suspend</strong> <span class="monospaced">&lt;vmid&gt;</span></p></div>
<div class="paragraph">
<p>Suspend the container. This is experimental.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pct template</strong> <span class="monospaced">&lt;vmid&gt;</span></p></div>
<div class="paragraph">
<p>Create a Template.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pct unlock</strong> <span class="monospaced">&lt;vmid&gt;</span></p></div>
<div class="paragraph">
<p>Unlock the VM.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pct unmount</strong> <span class="monospaced">&lt;vmid&gt;</span></p></div>
<div class="paragraph">
<p>Unmount the container’s filesystem.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
</dl></div>
</div>
<div class="sect2">
<h3 id="_strong_pveam_strong_proxmox_ve_appliance_manager">
<span>22.11. <strong>pveam</strong> -  Proxmox VE Appliance Manager</span>
 <a class="headerlink" href="#_strong_pveam_strong_proxmox_ve_appliance_manager" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p><strong>pveam</strong> <span class="monospaced">&lt;COMMAND&gt; [ARGS] [OPTIONS]</span></p></div>
<div class="paragraph">
<p><strong>pveam available</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>List available templates.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--section</span> <span class="monospaced">&lt;mail | system | turnkeylinux&gt;</span> 
</dt>
<dd>
<p>
Restrict list to specified section.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveam download</strong> <span class="monospaced">&lt;storage&gt; &lt;template&gt;</span></p></div>
<div class="paragraph">
<p>Download appliance templates.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;storage&gt;</span>: <span class="monospaced">&lt;storage ID&gt;</span> 
</dt>
<dd>
<p>
The storage where the template will be stored
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;template&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The template which will downloaded
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveam help</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Get help about specified command.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--extra-args</span> <span class="monospaced">&lt;array&gt;</span> 
</dt>
<dd>
<p>
Shows help for a specific command
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--verbose</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Verbose output format.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveam list</strong> <span class="monospaced">&lt;storage&gt;</span></p></div>
<div class="paragraph">
<p>Get list of all templates on storage</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;storage&gt;</span>: <span class="monospaced">&lt;storage ID&gt;</span> 
</dt>
<dd>
<p>
Only list templates on specified storage
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveam remove</strong> <span class="monospaced">&lt;template_path&gt;</span></p></div>
<div class="paragraph">
<p>Remove a template.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;template_path&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The template to remove.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveam update</strong></p></div>
<div class="paragraph">
<p>Update Container Template Database.</p></div>
</div>
<div class="sect2">
<h3 id="_strong_pvecm_strong_proxmox_ve_cluster_manager">
<span>22.12. <strong>pvecm</strong> - Proxmox VE Cluster Manager</span>
 <a class="headerlink" href="#_strong_pvecm_strong_proxmox_ve_cluster_manager" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p><strong>pvecm</strong> <span class="monospaced">&lt;COMMAND&gt; [ARGS] [OPTIONS]</span></p></div>
<div class="paragraph">
<p><strong>pvecm add</strong> <span class="monospaced">&lt;hostname&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Adds the current node to an existing cluster.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;hostname&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Hostname (or IP) of an existing cluster member.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--fingerprint</span> <span class="monospaced">([A-Fa-f0-9]{2}:){31}[A-Fa-f0-9]{2}</span> 
</dt>
<dd>
<p>
Certificate SHA 256 fingerprint.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--force</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Do not throw error if node already exists.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--link[n]</span> <span class="monospaced">[address=]&lt;IP&gt; [,priority=&lt;integer&gt;]</span> 
</dt>
<dd>
<p>
Address and priority information of a single corosync link. (up to 8 links supported; link0..link7)
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--nodeid</span> <span class="monospaced">&lt;integer&gt; (1 - N)</span> 
</dt>
<dd>
<p>
Node id for this node.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--use_ssh</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Always use SSH to join, even if peer may do it over API.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--votes</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> 
</dt>
<dd>
<p>
Number of votes for this node
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvecm addnode</strong> <span class="monospaced">&lt;node&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Adds a node to the cluster configuration. This call is for internal use.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;node&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The cluster node name.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--apiversion</span> <span class="monospaced">&lt;integer&gt;</span> 
</dt>
<dd>
<p>
The JOIN_API_VERSION of the new node.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--force</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Do not throw error if node already exists.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--link[n]</span> <span class="monospaced">[address=]&lt;IP&gt; [,priority=&lt;integer&gt;]</span> 
</dt>
<dd>
<p>
Address and priority information of a single corosync link. (up to 8 links supported; link0..link7)
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--new_node_ip</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
IP Address of node to add. Used as fallback if no links are given.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--nodeid</span> <span class="monospaced">&lt;integer&gt; (1 - N)</span> 
</dt>
<dd>
<p>
Node id for this node.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--votes</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> 
</dt>
<dd>
<p>
Number of votes for this node
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvecm apiver</strong></p></div>
<div class="paragraph">
<p>Return the version of the cluster join API available on this node.</p></div>
<div class="paragraph">
<p><strong>pvecm create</strong> <span class="monospaced">&lt;clustername&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Generate new cluster configuration. If no links given, default to local IP
address as link0.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;clustername&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The name of the cluster.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--link[n]</span> <span class="monospaced">[address=]&lt;IP&gt; [,priority=&lt;integer&gt;]</span> 
</dt>
<dd>
<p>
Address and priority information of a single corosync link. (up to 8 links supported; link0..link7)
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--nodeid</span> <span class="monospaced">&lt;integer&gt; (1 - N)</span> 
</dt>
<dd>
<p>
Node id for this node.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--votes</span> <span class="monospaced">&lt;integer&gt; (1 - N)</span> 
</dt>
<dd>
<p>
Number of votes for this node.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvecm delnode</strong> <span class="monospaced">&lt;node&gt;</span></p></div>
<div class="paragraph">
<p>Removes a node from the cluster configuration.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;node&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The cluster node name.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvecm expected</strong> <span class="monospaced">&lt;expected&gt;</span></p></div>
<div class="paragraph">
<p>Tells corosync a new value of expected votes.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;expected&gt;</span>: <span class="monospaced">&lt;integer&gt; (1 - N)</span> 
</dt>
<dd>
<p>
Expected votes
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvecm help</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Get help about specified command.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--extra-args</span> <span class="monospaced">&lt;array&gt;</span> 
</dt>
<dd>
<p>
Shows help for a specific command
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--verbose</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Verbose output format.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvecm keygen</strong> <span class="monospaced">&lt;filename&gt;</span></p></div>
<div class="paragraph">
<p>Generate new cryptographic key for corosync.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;filename&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Output file name
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvecm mtunnel</strong> <span class="monospaced">[&lt;extra-args&gt;]</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Used by VM/CT migration - do not use manually.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;extra-args&gt;</span>: <span class="monospaced">&lt;array&gt;</span> 
</dt>
<dd>
<p>
Extra arguments as array
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--get_migration_ip</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
return the migration IP, if configured
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--migration_network</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
the migration network used to detect the local migration IP
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--run-command</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Run a command with a tcp socket as standard input. The IP address and port are printed via this command’s stdandard output first, each on a separate line.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvecm nodes</strong></p></div>
<div class="paragraph">
<p>Displays the local view of the cluster nodes.</p></div>
<div class="paragraph">
<p><strong>pvecm qdevice remove</strong></p></div>
<div class="paragraph">
<p>Remove a configured QDevice</p></div>
<div class="paragraph">
<p><strong>pvecm qdevice setup</strong> <span class="monospaced">&lt;address&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Setup the use of a QDevice</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;address&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Specifies the network address of an external corosync QDevice
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--force</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Do not throw error on possible dangerous operations.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--network</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The network which should be used to connect to the external qdevice
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvecm status</strong></p></div>
<div class="paragraph">
<p>Displays the local view of the cluster status.</p></div>
<div class="paragraph">
<p><strong>pvecm updatecerts</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Update node certificates (and generate all needed files/directories).</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--force</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Force generation of new SSL certificate.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--silent</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Ignore errors (i.e. when cluster has no quorum).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--unmerge-known-hosts</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Unmerge legacy SSH known hosts.
</p>
</dd>
</dl></div>
</div>
<div class="sect2">
<h3 id="_strong_pvesr_strong_proxmox_ve_storage_replication">
<span>22.13. <strong>pvesr</strong> - Proxmox VE Storage Replication</span>
 <a class="headerlink" href="#_strong_pvesr_strong_proxmox_ve_storage_replication" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p><strong>pvesr</strong> <span class="monospaced">&lt;COMMAND&gt; [ARGS] [OPTIONS]</span></p></div>
<div class="paragraph">
<p><strong>pvesr create-local-job</strong> <span class="monospaced">&lt;id&gt; &lt;target&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Create a new replication job</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;id&gt;</span>: <span class="monospaced">[1-9][0-9]{2,8}-\d{1,9}</span> 
</dt>
<dd>
<p>
Replication Job ID. The ID is composed of a Guest ID and a job number, separated by a hyphen, i.e. <em>&lt;GUEST&gt;-&lt;JOBNUM&gt;</em>.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;target&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Target node.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--comment</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Description.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--disable</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Flag to disable/deactivate the entry.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--rate</span> <span class="monospaced">&lt;number&gt; (1 - N)</span> 
</dt>
<dd>
<p>
Rate limit in mbps (megabytes per second) as floating point number.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--remove_job</span> <span class="monospaced">&lt;full | local&gt;</span> 
</dt>
<dd>
<p>
Mark the replication job for removal. The job will remove all local replication snapshots. When set to <em>full</em>, it also tries to remove replicated volumes on the target. The job then removes itself from the configuration file.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--schedule</span> <span class="monospaced">&lt;string&gt;</span> (<em>default =</em> <span class="monospaced">*/15</span>)
</dt>
<dd>
<p>
Storage replication schedule. The format is a subset of <span class="monospaced">systemd</span> calendar events.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--source</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
For internal use, to detect if the guest was stolen.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesr delete</strong> <span class="monospaced">&lt;id&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Mark replication job for removal.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;id&gt;</span>: <span class="monospaced">[1-9][0-9]{2,8}-\d{1,9}</span> 
</dt>
<dd>
<p>
Replication Job ID. The ID is composed of a Guest ID and a job number, separated by a hyphen, i.e. <em>&lt;GUEST&gt;-&lt;JOBNUM&gt;</em>.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--force</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Will remove the jobconfig entry, but will not cleanup.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--keep</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Keep replicated data at target (do not remove).
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesr disable</strong> <span class="monospaced">&lt;id&gt;</span></p></div>
<div class="paragraph">
<p>Disable a replication job.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;id&gt;</span>: <span class="monospaced">[1-9][0-9]{2,8}-\d{1,9}</span> 
</dt>
<dd>
<p>
Replication Job ID. The ID is composed of a Guest ID and a job number, separated by a hyphen, i.e. <em>&lt;GUEST&gt;-&lt;JOBNUM&gt;</em>.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesr enable</strong> <span class="monospaced">&lt;id&gt;</span></p></div>
<div class="paragraph">
<p>Enable a replication job.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;id&gt;</span>: <span class="monospaced">[1-9][0-9]{2,8}-\d{1,9}</span> 
</dt>
<dd>
<p>
Replication Job ID. The ID is composed of a Guest ID and a job number, separated by a hyphen, i.e. <em>&lt;GUEST&gt;-&lt;JOBNUM&gt;</em>.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesr finalize-local-job</strong> <span class="monospaced">&lt;id&gt; [&lt;extra-args&gt;]</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Finalize a replication job. This removes all replications snapshots with
timestamps different than &lt;last_sync&gt;.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;id&gt;</span>: <span class="monospaced">[1-9][0-9]{2,8}-\d{1,9}</span> 
</dt>
<dd>
<p>
Replication Job ID. The ID is composed of a Guest ID and a job number, separated by a hyphen, i.e. <em>&lt;GUEST&gt;-&lt;JOBNUM&gt;</em>.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;extra-args&gt;</span>: <span class="monospaced">&lt;array&gt;</span> 
</dt>
<dd>
<p>
The list of volume IDs to consider.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--last_sync</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> 
</dt>
<dd>
<p>
Time (UNIX epoch) of last successful sync. If not specified, all replication snapshots gets removed.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesr help</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Get help about specified command.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--extra-args</span> <span class="monospaced">&lt;array&gt;</span> 
</dt>
<dd>
<p>
Shows help for a specific command
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--verbose</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Verbose output format.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesr list</strong></p></div>
<div class="paragraph">
<p>List replication jobs.</p></div>
<div class="paragraph">
<p><strong>pvesr prepare-local-job</strong> <span class="monospaced">&lt;id&gt; [&lt;extra-args&gt;]</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Prepare for starting a replication job. This is called on the target node
before replication starts. This call is for internal use, and return a JSON
object on stdout. The method first test if VM &lt;vmid&gt; reside on the local
node. If so, stop immediately. After that the method scans all volume IDs
for snapshots, and removes all replications snapshots with timestamps
different than &lt;last_sync&gt;. It also removes any unused volumes. Returns a
hash with boolean markers for all volumes with existing replication
snapshots.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;id&gt;</span>: <span class="monospaced">[1-9][0-9]{2,8}-\d{1,9}</span> 
</dt>
<dd>
<p>
Replication Job ID. The ID is composed of a Guest ID and a job number, separated by a hyphen, i.e. <em>&lt;GUEST&gt;-&lt;JOBNUM&gt;</em>.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;extra-args&gt;</span>: <span class="monospaced">&lt;array&gt;</span> 
</dt>
<dd>
<p>
The list of volume IDs to consider.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--force</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Allow to remove all existion volumes (empty volume list).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--last_sync</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> 
</dt>
<dd>
<p>
Time (UNIX epoch) of last successful sync. If not specified, all replication snapshots get removed.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--parent_snapname</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The name of the snapshot.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--scan</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
List of storage IDs to scan for stale volumes.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesr read</strong> <span class="monospaced">&lt;id&gt;</span></p></div>
<div class="paragraph">
<p>Read replication job configuration.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;id&gt;</span>: <span class="monospaced">[1-9][0-9]{2,8}-\d{1,9}</span> 
</dt>
<dd>
<p>
Replication Job ID. The ID is composed of a Guest ID and a job number, separated by a hyphen, i.e. <em>&lt;GUEST&gt;-&lt;JOBNUM&gt;</em>.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesr run</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>This method is called by the systemd-timer and executes all (or a specific)
sync jobs.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--id</span> <span class="monospaced">[1-9][0-9]{2,8}-\d{1,9}</span> 
</dt>
<dd>
<p>
Replication Job ID. The ID is composed of a Guest ID and a job number, separated by a hyphen, i.e. <em>&lt;GUEST&gt;-&lt;JOBNUM&gt;</em>.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--mail</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Send an email notification in case of a failure.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--verbose</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Print more verbose logs to stdout.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesr schedule-now</strong> <span class="monospaced">&lt;id&gt;</span></p></div>
<div class="paragraph">
<p>Schedule replication job to start as soon as possible.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;id&gt;</span>: <span class="monospaced">[1-9][0-9]{2,8}-\d{1,9}</span> 
</dt>
<dd>
<p>
Replication Job ID. The ID is composed of a Guest ID and a job number, separated by a hyphen, i.e. <em>&lt;GUEST&gt;-&lt;JOBNUM&gt;</em>.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesr set-state</strong> <span class="monospaced">&lt;vmid&gt; &lt;state&gt;</span></p></div>
<div class="paragraph">
<p>Set the job replication state on migration. This call is for internal use.
It will accept the job state as ja JSON obj.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
The (unique) ID of the VM.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;state&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Job state as JSON decoded string.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesr status</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>List status of all replication jobs on this node.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--guest</span> <span class="monospaced">&lt;integer&gt; (100 - 999999999)</span> 
</dt>
<dd>
<p>
Only list replication jobs for this guest.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvesr update</strong> <span class="monospaced">&lt;id&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Update replication job configuration.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;id&gt;</span>: <span class="monospaced">[1-9][0-9]{2,8}-\d{1,9}</span> 
</dt>
<dd>
<p>
Replication Job ID. The ID is composed of a Guest ID and a job number, separated by a hyphen, i.e. <em>&lt;GUEST&gt;-&lt;JOBNUM&gt;</em>.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--comment</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Description.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--delete</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
A list of settings you want to delete.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--digest</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Prevent changes if current configuration file has a different digest. This can be used to prevent concurrent modifications.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--disable</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Flag to disable/deactivate the entry.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--rate</span> <span class="monospaced">&lt;number&gt; (1 - N)</span> 
</dt>
<dd>
<p>
Rate limit in mbps (megabytes per second) as floating point number.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--remove_job</span> <span class="monospaced">&lt;full | local&gt;</span> 
</dt>
<dd>
<p>
Mark the replication job for removal. The job will remove all local replication snapshots. When set to <em>full</em>, it also tries to remove replicated volumes on the target. The job then removes itself from the configuration file.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--schedule</span> <span class="monospaced">&lt;string&gt;</span> (<em>default =</em> <span class="monospaced">*/15</span>)
</dt>
<dd>
<p>
Storage replication schedule. The format is a subset of <span class="monospaced">systemd</span> calendar events.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--source</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
For internal use, to detect if the guest was stolen.
</p>
</dd>
</dl></div>
</div>
<div class="sect2">
<h3 id="_strong_pveum_strong_proxmox_ve_user_manager">
<span>22.14. <strong>pveum</strong> - Proxmox VE User Manager</span>
 <a class="headerlink" href="#_strong_pveum_strong_proxmox_ve_user_manager" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p><strong>pveum</strong> <span class="monospaced">&lt;COMMAND&gt; [ARGS] [OPTIONS]</span></p></div>
<div class="paragraph">
<p><strong>pveum acl delete</strong> <span class="monospaced">&lt;path&gt; --roles &lt;string&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Update Access Control List (add or remove permissions).</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;path&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Access control path
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--groups</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
List of groups.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--propagate</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Allow to propagate (inherit) permissions.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--roles</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
List of roles.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--tokens</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
List of API tokens.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--users</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
List of users.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveum acl list</strong> <span class="monospaced">[FORMAT_OPTIONS]</span></p></div>
<div class="paragraph">
<p>Get Access Control List (ACLs).</p></div>
<div class="paragraph">
<p><strong>pveum acl modify</strong> <span class="monospaced">&lt;path&gt; --roles &lt;string&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Update Access Control List (add or remove permissions).</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;path&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Access control path
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--groups</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
List of groups.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--propagate</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Allow to propagate (inherit) permissions.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--roles</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
List of roles.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--tokens</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
List of API tokens.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--users</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
List of users.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveum acldel</strong></p></div>
<div class="paragraph">
<p>An alias for <em>pveum acl delete</em>.</p></div>
<div class="paragraph">
<p><strong>pveum aclmod</strong></p></div>
<div class="paragraph">
<p>An alias for <em>pveum acl modify</em>.</p></div>
<div class="paragraph">
<p><strong>pveum group add</strong> <span class="monospaced">&lt;groupid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Create new group.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;groupid&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
no description available
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--comment</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
no description available
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveum group delete</strong> <span class="monospaced">&lt;groupid&gt;</span></p></div>
<div class="paragraph">
<p>Delete group.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;groupid&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
no description available
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveum group list</strong> <span class="monospaced">[FORMAT_OPTIONS]</span></p></div>
<div class="paragraph">
<p>Group index.</p></div>
<div class="paragraph">
<p><strong>pveum group modify</strong> <span class="monospaced">&lt;groupid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Update group data.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;groupid&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
no description available
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--comment</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
no description available
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveum groupadd</strong></p></div>
<div class="paragraph">
<p>An alias for <em>pveum group add</em>.</p></div>
<div class="paragraph">
<p><strong>pveum groupdel</strong></p></div>
<div class="paragraph">
<p>An alias for <em>pveum group delete</em>.</p></div>
<div class="paragraph">
<p><strong>pveum groupmod</strong></p></div>
<div class="paragraph">
<p>An alias for <em>pveum group modify</em>.</p></div>
<div class="paragraph">
<p><strong>pveum help</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Get help about specified command.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--extra-args</span> <span class="monospaced">&lt;array&gt;</span> 
</dt>
<dd>
<p>
Shows help for a specific command
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--verbose</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Verbose output format.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveum passwd</strong> <span class="monospaced">&lt;userid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Change user password.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;userid&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Full User ID, in the <span class="monospaced">name@realm</span> format.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--confirmation-password</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The current password of the user performing the change.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveum pool add</strong> <span class="monospaced">&lt;poolid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Create new pool.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;poolid&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
no description available
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--comment</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
no description available
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveum pool delete</strong> <span class="monospaced">&lt;poolid&gt;</span></p></div>
<div class="paragraph">
<p>Delete pool.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;poolid&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
no description available
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveum pool list</strong> <span class="monospaced">[OPTIONS]</span> <span class="monospaced">[FORMAT_OPTIONS]</span></p></div>
<div class="paragraph">
<p>List pools or get pool configuration.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--poolid</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
no description available
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--type</span> <span class="monospaced">&lt;lxc | qemu | storage&gt;</span> 
</dt>
<dd>
<p>
no description available
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Requires option(s): <span class="monospaced">poolid</span></td>
</tr></tbody></table>
</div>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveum pool modify</strong> <span class="monospaced">&lt;poolid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Update pool.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;poolid&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
no description available
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--allow-move</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Allow adding a guest even if already in another pool. The guest will be removed from its current pool and added to this one.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--comment</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
no description available
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--delete</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Remove the passed VMIDs and/or storage IDs instead of adding them.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--storage</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
List of storage IDs to add or remove from this pool.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--vms</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
List of guest VMIDs to add or remove from this pool.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveum realm add</strong> <span class="monospaced">&lt;realm&gt; --type &lt;string&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Add an authentication server.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;realm&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Authentication domain ID
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--acr-values</span> <span class="monospaced">^[^\x00-\x1F\x7F &lt;&gt;#"]*$</span> 
</dt>
<dd>
<p>
Specifies the Authentication Context Class Reference values that theAuthorization Server is being requested to use for the Auth Request.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--autocreate</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Automatically create users if they do not exist.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--base_dn</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
LDAP base domain name
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--bind_dn</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
LDAP bind domain name
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--capath</span> <span class="monospaced">&lt;string&gt;</span> (<em>default =</em> <span class="monospaced">/etc/ssl/certs</span>)
</dt>
<dd>
<p>
Path to the CA certificate store
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--case-sensitive</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
username is case-sensitive
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--cert</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Path to the client certificate
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--certkey</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Path to the client certificate key
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--check-connection</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Check bind connection to the server.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--client-id</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
OpenID Client ID
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--client-key</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
OpenID Client Key
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--comment</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Description.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--default</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Use this as default realm
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--domain</span> <span class="monospaced">\S+</span> 
</dt>
<dd>
<p>
AD domain name
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--filter</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
LDAP filter for user sync.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--group_classes</span> <span class="monospaced">&lt;string&gt;</span> (<em>default =</em> <span class="monospaced">groupOfNames, group, univentionGroup, ipausergroup</span>)
</dt>
<dd>
<p>
The objectclasses for groups.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--group_dn</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
LDAP base domain name for group sync. If not set, the base_dn will be used.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--group_filter</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
LDAP filter for group sync.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--group_name_attr</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
LDAP attribute representing a groups name. If not set or found, the first value of the DN will be used as name.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--issuer-url</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
OpenID Issuer Url
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--mode</span> <span class="monospaced">&lt;ldap | ldap+starttls | ldaps&gt;</span> (<em>default =</em> <span class="monospaced">ldap</span>)
</dt>
<dd>
<p>
LDAP protocol mode.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--password</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
LDAP bind password. Will be stored in <em>/etc/pve/priv/realm/&lt;REALM&gt;.pw</em>.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--port</span> <span class="monospaced">&lt;integer&gt; (1 - 65535)</span> 
</dt>
<dd>
<p>
Server port.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--prompt</span> <span class="monospaced">(?:none|login|consent|select_account|\S+)</span> 
</dt>
<dd>
<p>
Specifies whether the Authorization Server prompts the End-User for reauthentication and consent.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--scopes</span> <span class="monospaced">&lt;string&gt;</span> (<em>default =</em> <span class="monospaced">email profile</span>)
</dt>
<dd>
<p>
Specifies the scopes (user details) that should be authorized and returned, for example <em>email</em> or <em>profile</em>.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--secure</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Use secure LDAPS protocol. DEPRECATED: use <em>mode</em> instead.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--server1</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Server IP address (or DNS name)
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--server2</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Fallback Server IP address (or DNS name)
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--sslversion</span> <span class="monospaced">&lt;tlsv1 | tlsv1_1 | tlsv1_2 | tlsv1_3&gt;</span> 
</dt>
<dd>
<p>
LDAPS TLS/SSL version. It’s not recommended to use version older than 1.2!
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--sync-defaults-options</span> <span class="monospaced">[enable-new=&lt;1|0&gt;] [,full=&lt;1|0&gt;] [,purge=&lt;1|0&gt;] [,remove-vanished=([acl];[properties];[entry])|none] [,scope=&lt;users|groups|both&gt;]</span> 
</dt>
<dd>
<p>
The default options for behavior of synchronizations.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--sync_attributes</span> <span class="monospaced">\w+=[^,]+(,\s*\w+=[^,]+)*</span> 
</dt>
<dd>
<p>
Comma separated list of key=value pairs for specifying which LDAP attributes map to which PVE user field. For example, to map the LDAP attribute <em>mail</em> to PVEs <em>email</em>, write  <em>email=mail</em>. By default, each PVE user field is represented  by an LDAP attribute of the same name.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--tfa</span> <span class="monospaced">type=&lt;TFATYPE&gt; [,digits=&lt;COUNT&gt;] [,id=&lt;ID&gt;] [,key=&lt;KEY&gt;] [,step=&lt;SECONDS&gt;] [,url=&lt;URL&gt;]</span> 
</dt>
<dd>
<p>
Use Two-factor authentication.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--type</span> <span class="monospaced">&lt;ad | ldap | openid | pam | pve&gt;</span> 
</dt>
<dd>
<p>
Realm type.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--user_attr</span> <span class="monospaced">\S{2,}</span> 
</dt>
<dd>
<p>
LDAP user attribute name
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--user_classes</span> <span class="monospaced">&lt;string&gt;</span> (<em>default =</em> <span class="monospaced">inetorgperson, posixaccount, person, user</span>)
</dt>
<dd>
<p>
The objectclasses for users.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--username-claim</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
OpenID claim used to generate the unique username.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--verify</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Verify the server’s SSL certificate
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveum realm delete</strong> <span class="monospaced">&lt;realm&gt;</span></p></div>
<div class="paragraph">
<p>Delete an authentication server.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;realm&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Authentication domain ID
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveum realm list</strong> <span class="monospaced">[FORMAT_OPTIONS]</span></p></div>
<div class="paragraph">
<p>Authentication domain index.</p></div>
<div class="paragraph">
<p><strong>pveum realm modify</strong> <span class="monospaced">&lt;realm&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Update authentication server settings.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;realm&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Authentication domain ID
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--acr-values</span> <span class="monospaced">^[^\x00-\x1F\x7F &lt;&gt;#"]*$</span> 
</dt>
<dd>
<p>
Specifies the Authentication Context Class Reference values that theAuthorization Server is being requested to use for the Auth Request.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--autocreate</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Automatically create users if they do not exist.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--base_dn</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
LDAP base domain name
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--bind_dn</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
LDAP bind domain name
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--capath</span> <span class="monospaced">&lt;string&gt;</span> (<em>default =</em> <span class="monospaced">/etc/ssl/certs</span>)
</dt>
<dd>
<p>
Path to the CA certificate store
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--case-sensitive</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
username is case-sensitive
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--cert</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Path to the client certificate
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--certkey</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Path to the client certificate key
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--check-connection</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Check bind connection to the server.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--client-id</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
OpenID Client ID
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--client-key</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
OpenID Client Key
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--comment</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Description.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--default</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Use this as default realm
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--delete</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
A list of settings you want to delete.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--digest</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Prevent changes if current configuration file has a different digest. This can be used to prevent concurrent modifications.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--domain</span> <span class="monospaced">\S+</span> 
</dt>
<dd>
<p>
AD domain name
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--filter</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
LDAP filter for user sync.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--group_classes</span> <span class="monospaced">&lt;string&gt;</span> (<em>default =</em> <span class="monospaced">groupOfNames, group, univentionGroup, ipausergroup</span>)
</dt>
<dd>
<p>
The objectclasses for groups.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--group_dn</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
LDAP base domain name for group sync. If not set, the base_dn will be used.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--group_filter</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
LDAP filter for group sync.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--group_name_attr</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
LDAP attribute representing a groups name. If not set or found, the first value of the DN will be used as name.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--issuer-url</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
OpenID Issuer Url
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--mode</span> <span class="monospaced">&lt;ldap | ldap+starttls | ldaps&gt;</span> (<em>default =</em> <span class="monospaced">ldap</span>)
</dt>
<dd>
<p>
LDAP protocol mode.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--password</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
LDAP bind password. Will be stored in <em>/etc/pve/priv/realm/&lt;REALM&gt;.pw</em>.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--port</span> <span class="monospaced">&lt;integer&gt; (1 - 65535)</span> 
</dt>
<dd>
<p>
Server port.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--prompt</span> <span class="monospaced">(?:none|login|consent|select_account|\S+)</span> 
</dt>
<dd>
<p>
Specifies whether the Authorization Server prompts the End-User for reauthentication and consent.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--scopes</span> <span class="monospaced">&lt;string&gt;</span> (<em>default =</em> <span class="monospaced">email profile</span>)
</dt>
<dd>
<p>
Specifies the scopes (user details) that should be authorized and returned, for example <em>email</em> or <em>profile</em>.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--secure</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Use secure LDAPS protocol. DEPRECATED: use <em>mode</em> instead.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--server1</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Server IP address (or DNS name)
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--server2</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Fallback Server IP address (or DNS name)
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--sslversion</span> <span class="monospaced">&lt;tlsv1 | tlsv1_1 | tlsv1_2 | tlsv1_3&gt;</span> 
</dt>
<dd>
<p>
LDAPS TLS/SSL version. It’s not recommended to use version older than 1.2!
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--sync-defaults-options</span> <span class="monospaced">[enable-new=&lt;1|0&gt;] [,full=&lt;1|0&gt;] [,purge=&lt;1|0&gt;] [,remove-vanished=([acl];[properties];[entry])|none] [,scope=&lt;users|groups|both&gt;]</span> 
</dt>
<dd>
<p>
The default options for behavior of synchronizations.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--sync_attributes</span> <span class="monospaced">\w+=[^,]+(,\s*\w+=[^,]+)*</span> 
</dt>
<dd>
<p>
Comma separated list of key=value pairs for specifying which LDAP attributes map to which PVE user field. For example, to map the LDAP attribute <em>mail</em> to PVEs <em>email</em>, write  <em>email=mail</em>. By default, each PVE user field is represented  by an LDAP attribute of the same name.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--tfa</span> <span class="monospaced">type=&lt;TFATYPE&gt; [,digits=&lt;COUNT&gt;] [,id=&lt;ID&gt;] [,key=&lt;KEY&gt;] [,step=&lt;SECONDS&gt;] [,url=&lt;URL&gt;]</span> 
</dt>
<dd>
<p>
Use Two-factor authentication.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--user_attr</span> <span class="monospaced">\S{2,}</span> 
</dt>
<dd>
<p>
LDAP user attribute name
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--user_classes</span> <span class="monospaced">&lt;string&gt;</span> (<em>default =</em> <span class="monospaced">inetorgperson, posixaccount, person, user</span>)
</dt>
<dd>
<p>
The objectclasses for users.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--verify</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Verify the server’s SSL certificate
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveum realm sync</strong> <span class="monospaced">&lt;realm&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Syncs users and/or groups from the configured LDAP to user.cfg. NOTE:
Synced groups will have the name <em>name-$realm</em>, so make sure those groups
do not exist to prevent overwriting.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;realm&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Authentication domain ID
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--dry-run</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
If set, does not write anything.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--enable-new</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Enable newly synced users immediately.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--full</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
DEPRECATED: use <em>remove-vanished</em> instead. If set, uses the LDAP Directory as source of truth, deleting users or groups not returned from the sync and removing all locally modified properties of synced users. If not set, only syncs information which is present in the synced data, and does not delete or modify anything else.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--purge</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
DEPRECATED: use <em>remove-vanished</em> instead. Remove ACLs for users or groups which were removed from the config during a sync.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--remove-vanished</span> <span class="monospaced">([acl];[properties];[entry])|none</span> (<em>default =</em> <span class="monospaced">none</span>)
</dt>
<dd>
<p>
A semicolon-seperated list of things to remove when they or the user vanishes during a sync. The following values are possible: <em>entry</em> removes the user/group when not returned from the sync. <em>properties</em> removes the set properties on existing user/group that do not appear in the source (even custom ones). <em>acl</em> removes acls when the user/group is not returned from the sync. Instead of a list it also can be <em>none</em> (the default).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--scope</span> <span class="monospaced">&lt;both | groups | users&gt;</span> 
</dt>
<dd>
<p>
Select what to sync.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveum role add</strong> <span class="monospaced">&lt;roleid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Create new role.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;roleid&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
no description available
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--privs</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
no description available
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveum role delete</strong> <span class="monospaced">&lt;roleid&gt;</span></p></div>
<div class="paragraph">
<p>Delete role.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;roleid&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
no description available
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveum role list</strong> <span class="monospaced">[FORMAT_OPTIONS]</span></p></div>
<div class="paragraph">
<p>Role index.</p></div>
<div class="paragraph">
<p><strong>pveum role modify</strong> <span class="monospaced">&lt;roleid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Update an existing role.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;roleid&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
no description available
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--append</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
no description available
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Requires option(s): <span class="monospaced">privs</span></td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">--privs</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
no description available
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveum roleadd</strong></p></div>
<div class="paragraph">
<p>An alias for <em>pveum role add</em>.</p></div>
<div class="paragraph">
<p><strong>pveum roledel</strong></p></div>
<div class="paragraph">
<p>An alias for <em>pveum role delete</em>.</p></div>
<div class="paragraph">
<p><strong>pveum rolemod</strong></p></div>
<div class="paragraph">
<p>An alias for <em>pveum role modify</em>.</p></div>
<div class="paragraph">
<p><strong>pveum ticket</strong> <span class="monospaced">&lt;username&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Create or verify authentication ticket.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;username&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
User name
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--new-format</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
This parameter is now ignored and assumed to be 1.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--otp</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
One-time password for Two-factor authentication.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--path</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Verify ticket, and check if user have access <em>privs</em> on <em>path</em>
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Requires option(s): <span class="monospaced">privs</span></td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">--privs</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Verify ticket, and check if user have access <em>privs</em> on <em>path</em>
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Requires option(s): <span class="monospaced">path</span></td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">--realm</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
You can optionally pass the realm using this parameter. Normally the realm is simply added to the username &lt;username&gt;@&lt;relam&gt;.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--tfa-challenge</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The signed TFA challenge string the user wants to respond to.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveum user add</strong> <span class="monospaced">&lt;userid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Create new user.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;userid&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Full User ID, in the <span class="monospaced">name@realm</span> format.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--comment</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
no description available
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--email</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
no description available
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--enable</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Enable the account (default). You can set this to <em>0</em> to disable the account
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--expire</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> 
</dt>
<dd>
<p>
Account expiration date (seconds since epoch). <em>0</em> means no expiration date.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--firstname</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
no description available
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--groups</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
no description available
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--keys</span> <span class="monospaced">[0-9a-zA-Z!=]{0,4096}</span> 
</dt>
<dd>
<p>
Keys for two factor auth (yubico).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--lastname</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
no description available
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--password</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Initial password.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveum user delete</strong> <span class="monospaced">&lt;userid&gt;</span></p></div>
<div class="paragraph">
<p>Delete user.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;userid&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Full User ID, in the <span class="monospaced">name@realm</span> format.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveum user list</strong> <span class="monospaced">[OPTIONS]</span> <span class="monospaced">[FORMAT_OPTIONS]</span></p></div>
<div class="paragraph">
<p>User index.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--enabled</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Optional filter for enable property.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--full</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Include group and token information.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveum user modify</strong> <span class="monospaced">&lt;userid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Update user configuration.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;userid&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Full User ID, in the <span class="monospaced">name@realm</span> format.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--append</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
no description available
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Requires option(s): <span class="monospaced">groups</span></td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">--comment</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
no description available
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--email</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
no description available
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--enable</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Enable the account (default). You can set this to <em>0</em> to disable the account
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--expire</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> 
</dt>
<dd>
<p>
Account expiration date (seconds since epoch). <em>0</em> means no expiration date.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--firstname</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
no description available
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--groups</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
no description available
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--keys</span> <span class="monospaced">[0-9a-zA-Z!=]{0,4096}</span> 
</dt>
<dd>
<p>
Keys for two factor auth (yubico).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--lastname</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
no description available
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveum user permissions</strong> <span class="monospaced">[&lt;userid&gt;]</span> <span class="monospaced">[OPTIONS]</span> <span class="monospaced">[FORMAT_OPTIONS]</span></p></div>
<div class="paragraph">
<p>Retrieve effective permissions of given user/token.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;userid&gt;</span>: <span class="monospaced">(?^:^(?^:[^\s:/]+)\@(?^:[A-Za-z][A-Za-z0-9\.\-_]+)(?:!(?^:[A-Za-z][A-Za-z0-9\.\-_]+))?$)</span> 
</dt>
<dd>
<p>
User ID or full API token ID
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--path</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Only dump this specific path, not the whole tree.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveum user tfa delete</strong> <span class="monospaced">&lt;userid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Delete TFA entries from a user.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;userid&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Full User ID, in the <span class="monospaced">name@realm</span> format.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--id</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The TFA ID, if none provided, all TFA entries will be deleted.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveum user tfa list</strong> <span class="monospaced">[&lt;userid&gt;]</span></p></div>
<div class="paragraph">
<p>List TFA entries.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;userid&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Full User ID, in the <span class="monospaced">name@realm</span> format.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveum user tfa unlock</strong> <span class="monospaced">&lt;userid&gt;</span></p></div>
<div class="paragraph">
<p>Unlock a user’s TFA authentication.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;userid&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Full User ID, in the <span class="monospaced">name@realm</span> format.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveum user token add</strong> <span class="monospaced">&lt;userid&gt; &lt;tokenid&gt;</span> <span class="monospaced">[OPTIONS]</span> <span class="monospaced">[FORMAT_OPTIONS]</span></p></div>
<div class="paragraph">
<p>Generate a new API token for a specific user. NOTE: returns API token
value, which needs to be stored as it cannot be retrieved afterwards!</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;userid&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Full User ID, in the <span class="monospaced">name@realm</span> format.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;tokenid&gt;</span>: <span class="monospaced">(?^:[A-Za-z][A-Za-z0-9\.\-_]+)</span> 
</dt>
<dd>
<p>
User-specific token identifier.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--comment</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
no description available
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--expire</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">same as user</span>)
</dt>
<dd>
<p>
API token expiration date (seconds since epoch). <em>0</em> means no expiration date.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--privsep</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Restrict API token privileges with separate ACLs (default), or give full privileges of corresponding user.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveum user token delete</strong> <span class="monospaced">&lt;userid&gt; &lt;tokenid&gt;</span> <span class="monospaced">[FORMAT_OPTIONS]</span></p></div>
<div class="paragraph">
<p>Remove API token for a specific user.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;userid&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Full User ID, in the <span class="monospaced">name@realm</span> format.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;tokenid&gt;</span>: <span class="monospaced">(?^:[A-Za-z][A-Za-z0-9\.\-_]+)</span> 
</dt>
<dd>
<p>
User-specific token identifier.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveum user token list</strong> <span class="monospaced">&lt;userid&gt;</span> <span class="monospaced">[FORMAT_OPTIONS]</span></p></div>
<div class="paragraph">
<p>Get user API tokens.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;userid&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Full User ID, in the <span class="monospaced">name@realm</span> format.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveum user token modify</strong> <span class="monospaced">&lt;userid&gt; &lt;tokenid&gt;</span> <span class="monospaced">[OPTIONS]</span> <span class="monospaced">[FORMAT_OPTIONS]</span></p></div>
<div class="paragraph">
<p>Update API token for a specific user.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;userid&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Full User ID, in the <span class="monospaced">name@realm</span> format.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;tokenid&gt;</span>: <span class="monospaced">(?^:[A-Za-z][A-Za-z0-9\.\-_]+)</span> 
</dt>
<dd>
<p>
User-specific token identifier.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--comment</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
no description available
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--expire</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">same as user</span>)
</dt>
<dd>
<p>
API token expiration date (seconds since epoch). <em>0</em> means no expiration date.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--privsep</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Restrict API token privileges with separate ACLs (default), or give full privileges of corresponding user.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveum user token permissions</strong> <span class="monospaced">&lt;userid&gt; &lt;tokenid&gt;</span> <span class="monospaced">[OPTIONS]</span> <span class="monospaced">[FORMAT_OPTIONS]</span></p></div>
<div class="paragraph">
<p>Retrieve effective permissions of given token.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;userid&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Full User ID, in the <span class="monospaced">name@realm</span> format.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;tokenid&gt;</span>: <span class="monospaced">(?^:[A-Za-z][A-Za-z0-9\.\-_]+)</span> 
</dt>
<dd>
<p>
User-specific token identifier.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--path</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Only dump this specific path, not the whole tree.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveum user token remove</strong></p></div>
<div class="paragraph">
<p>An alias for <em>pveum user token delete</em>.</p></div>
<div class="paragraph">
<p><strong>pveum useradd</strong></p></div>
<div class="paragraph">
<p>An alias for <em>pveum user add</em>.</p></div>
<div class="paragraph">
<p><strong>pveum userdel</strong></p></div>
<div class="paragraph">
<p>An alias for <em>pveum user delete</em>.</p></div>
<div class="paragraph">
<p><strong>pveum usermod</strong></p></div>
<div class="paragraph">
<p>An alias for <em>pveum user modify</em>.</p></div>
</div>
<div class="sect2">
<h3 id="_strong_vzdump_strong_backup_utility_for_vms_and_containers">
<span>22.15. <strong>vzdump</strong> - Backup Utility for VMs and Containers</span>
 <a class="headerlink" href="#_strong_vzdump_strong_backup_utility_for_vms_and_containers" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p><strong>vzdump</strong> <span class="monospaced">help</span></p></div>
<div class="paragraph">
<p><strong>vzdump</strong> <span class="monospaced">{&lt;vmid&gt;}</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Create backup.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;vmid&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The ID of the guest system you want to backup.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--all</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Backup all known guest systems on this host.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--bwlimit</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Limit I/O bandwidth (in KiB/s).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--compress</span> <span class="monospaced">&lt;0 | 1 | gzip | lzo | zstd&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Compress dump file.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--dumpdir</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Store resulting files to specified directory.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--exclude</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Exclude specified guest systems (assumes --all)
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--exclude-path</span> <span class="monospaced">&lt;array&gt;</span> 
</dt>
<dd>
<p>
Exclude certain files/directories (shell globs). Paths starting with <em>/</em> are anchored to the container’s root, other paths match relative to each subdirectory.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--fleecing</span> <span class="monospaced">[[enabled=]&lt;1|0&gt;] [,storage=&lt;storage ID&gt;]</span> 
</dt>
<dd>
<p>
Options for backup fleecing (VM only).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--ionice</span> <span class="monospaced">&lt;integer&gt; (0 - 8)</span> (<em>default =</em> <span class="monospaced">7</span>)
</dt>
<dd>
<p>
Set IO priority when using the BFQ scheduler. For snapshot and suspend mode backups of VMs, this only affects the compressor. A value of 8 means the idle priority is used, otherwise the best-effort priority is used with the specified value.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--job-id</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The ID of the backup job. If set, the <em>backup-job</em> metadata field of the backup notification will be set to this value. Only <a href="mailto:root@pam">root@pam</a> can set this parameter.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--lockwait</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">180</span>)
</dt>
<dd>
<p>
Maximal time to wait for the global lock (minutes).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--mailnotification</span> <span class="monospaced">&lt;always | failure&gt;</span> (<em>default =</em> <span class="monospaced">always</span>)
</dt>
<dd>
<p>
Deprecated: use notification targets/matchers instead. Specify when to send a notification mail
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--mailto</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Deprecated: Use notification targets/matchers instead. Comma-separated list of email addresses or users that should receive email notifications.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--maxfiles</span> <span class="monospaced">&lt;integer&gt; (1 - N)</span> 
</dt>
<dd>
<p>
Deprecated: use <em>prune-backups</em> instead. Maximal number of backup files per guest system.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--mode</span> <span class="monospaced">&lt;snapshot | stop | suspend&gt;</span> (<em>default =</em> <span class="monospaced">snapshot</span>)
</dt>
<dd>
<p>
Backup mode.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--node</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Only run if executed on this node.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--notes-template</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Template string for generating notes for the backup(s). It can contain variables which will be replaced by their values. Currently supported are {\{\cluster}}, {\{\guestname}}, {\{\node}}, and {\{\vmid}}, but more might be added in the future. Needs to be a single line, newline and backslash need to be escaped as <em>\n</em> and <em>\\</em> respectively.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Requires option(s): <span class="monospaced">storage</span></td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">--notification-mode</span> <span class="monospaced">&lt;auto | legacy-sendmail | notification-system&gt;</span> (<em>default =</em> <span class="monospaced">auto</span>)
</dt>
<dd>
<p>
Determine which notification system to use. If set to <em>legacy-sendmail</em>, vzdump will consider the mailto/mailnotification parameters and send emails to the specified address(es) via the <em>sendmail</em> command. If set to <em>notification-system</em>, a notification will be sent via PVE’s notification system, and the mailto and mailnotification will be ignored. If set to <em>auto</em> (default setting), an email will be sent if mailto is set, and the notification system will be used if not.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--notification-policy</span> <span class="monospaced">&lt;always | failure | never&gt;</span> (<em>default =</em> <span class="monospaced">always</span>)
</dt>
<dd>
<p>
Deprecated: Do not use
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--notification-target</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Deprecated: Do not use
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--pbs-change-detection-mode</span> <span class="monospaced">&lt;data | legacy | metadata&gt;</span> 
</dt>
<dd>
<p>
PBS mode used to detect file changes and switch encoding. NOTE: <span class="monospaced">data</span> and <span class="monospaced">metadata</span> modes are experimental. format for container backups.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--performance</span> <span class="monospaced">[max-workers=&lt;integer&gt;] [,pbs-entries-max=&lt;integer&gt;]</span> 
</dt>
<dd>
<p>
Other performance-related settings.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--pigz</span> <span class="monospaced">&lt;integer&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Use pigz instead of gzip when N&gt;0. N=1 uses half of cores, N&gt;1 uses N as thread count.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--pool</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Backup all known guest systems included in the specified pool.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--protected</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
If true, mark backup(s) as protected.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Requires option(s): <span class="monospaced">storage</span></td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">--prune-backups</span> <span class="monospaced">[keep-all=&lt;1|0&gt;] [,keep-daily=&lt;N&gt;] [,keep-hourly=&lt;N&gt;] [,keep-last=&lt;N&gt;] [,keep-monthly=&lt;N&gt;] [,keep-weekly=&lt;N&gt;] [,keep-yearly=&lt;N&gt;]</span> (<em>default =</em> <span class="monospaced">keep-all=1</span>)
</dt>
<dd>
<p>
Use these retention options instead of those from the storage configuration.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--quiet</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Be quiet.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--remove</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Prune older backups according to <em>prune-backups</em>.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--script</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Use specified hook script.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--stdexcludes</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Exclude temporary files and logs.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--stdout</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Write tar to stdout, not to a file.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--stop</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Stop running backup jobs on this host.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--stopwait</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">10</span>)
</dt>
<dd>
<p>
Maximal time to wait until a guest system is stopped (minutes).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--storage</span> <span class="monospaced">&lt;storage ID&gt;</span> 
</dt>
<dd>
<p>
Store resulting file to this storage.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--tmpdir</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Store temporary files to specified directory.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--zstd</span> <span class="monospaced">&lt;integer&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Zstd threads. N=0 uses half of the available cores, if N is set to a value bigger than 0, N is used as thread count.
</p>
</dd>
</dl></div>
</div>
<div class="sect2">
<h3 id="_strong_ha_manager_strong_proxmox_ve_ha_manager">
<span>22.16. <strong>ha-manager</strong> - Proxmox VE HA Manager</span>
 <a class="headerlink" href="#_strong_ha_manager_strong_proxmox_ve_ha_manager" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p><strong>ha-manager</strong> <span class="monospaced">&lt;COMMAND&gt; [ARGS] [OPTIONS]</span></p></div>
<div class="paragraph">
<p><strong>ha-manager add</strong> <span class="monospaced">&lt;sid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Create a new HA resource.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;sid&gt;</span>: <span class="monospaced">&lt;type&gt;:&lt;name&gt;</span> 
</dt>
<dd>
<p>
HA resource ID. This consists of a resource type followed by a resource specific name, separated with colon (example: vm:100 / ct:100). For virtual machines and containers, you can simply use the VM or CT id as a shortcut (example: 100).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--comment</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Description.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--group</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The HA group identifier.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--max_relocate</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Maximal number of service relocate tries when a service failes to start.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--max_restart</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Maximal number of tries to restart the service on a node after its start failed.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--state</span> <span class="monospaced">&lt;disabled | enabled | ignored | started | stopped&gt;</span> (<em>default =</em> <span class="monospaced">started</span>)
</dt>
<dd>
<p>
Requested resource state.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--type</span> <span class="monospaced">&lt;ct | vm&gt;</span> 
</dt>
<dd>
<p>
Resource type.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>ha-manager config</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>List HA resources.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--type</span> <span class="monospaced">&lt;ct | vm&gt;</span> 
</dt>
<dd>
<p>
Only list resources of specific type
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>ha-manager crm-command migrate</strong> <span class="monospaced">&lt;sid&gt; &lt;node&gt;</span></p></div>
<div class="paragraph">
<p>Request resource migration (online) to another node.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;sid&gt;</span>: <span class="monospaced">&lt;type&gt;:&lt;name&gt;</span> 
</dt>
<dd>
<p>
HA resource ID. This consists of a resource type followed by a resource specific name, separated with colon (example: vm:100 / ct:100). For virtual machines and containers, you can simply use the VM or CT id as a shortcut (example: 100).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;node&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Target node.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>ha-manager crm-command node-maintenance disable</strong> <span class="monospaced">&lt;node&gt;</span></p></div>
<div class="paragraph">
<p>Change the node-maintenance request state.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;node&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The cluster node name.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>ha-manager crm-command node-maintenance enable</strong> <span class="monospaced">&lt;node&gt;</span></p></div>
<div class="paragraph">
<p>Change the node-maintenance request state.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;node&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The cluster node name.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>ha-manager crm-command relocate</strong> <span class="monospaced">&lt;sid&gt; &lt;node&gt;</span></p></div>
<div class="paragraph">
<p>Request resource relocatzion to another node. This stops the service on the
old node, and restarts it on the target node.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;sid&gt;</span>: <span class="monospaced">&lt;type&gt;:&lt;name&gt;</span> 
</dt>
<dd>
<p>
HA resource ID. This consists of a resource type followed by a resource specific name, separated with colon (example: vm:100 / ct:100). For virtual machines and containers, you can simply use the VM or CT id as a shortcut (example: 100).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;node&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Target node.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>ha-manager crm-command stop</strong> <span class="monospaced">&lt;sid&gt; &lt;timeout&gt;</span></p></div>
<div class="paragraph">
<p>Request the service to be stopped.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;sid&gt;</span>: <span class="monospaced">&lt;type&gt;:&lt;name&gt;</span> 
</dt>
<dd>
<p>
HA resource ID. This consists of a resource type followed by a resource specific name, separated with colon (example: vm:100 / ct:100). For virtual machines and containers, you can simply use the VM or CT id as a shortcut (example: 100).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">&lt;timeout&gt;</span>: <span class="monospaced">&lt;integer&gt; (0 - N)</span> 
</dt>
<dd>
<p>
Timeout in seconds. If set to 0 a hard stop will be performed.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>ha-manager groupadd</strong> <span class="monospaced">&lt;group&gt; --nodes &lt;string&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Create a new HA group.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;group&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The HA group identifier.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--comment</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Description.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--nodes</span> <span class="monospaced">&lt;node&gt;[:&lt;pri&gt;]{,&lt;node&gt;[:&lt;pri&gt;]}*</span> 
</dt>
<dd>
<p>
List of cluster node names with optional priority.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--nofailback</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
The CRM tries to run services on the node with the highest priority. If a node with higher priority comes online, the CRM migrates the service to that node. Enabling nofailback prevents that behavior.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--restricted</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Resources bound to restricted groups may only run on nodes defined by the group.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--type</span> <span class="monospaced">&lt;group&gt;</span> 
</dt>
<dd>
<p>
Group type.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>ha-manager groupconfig</strong></p></div>
<div class="paragraph">
<p>Get HA groups.</p></div>
<div class="paragraph">
<p><strong>ha-manager groupremove</strong> <span class="monospaced">&lt;group&gt;</span></p></div>
<div class="paragraph">
<p>Delete ha group configuration.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;group&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The HA group identifier.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>ha-manager groupset</strong> <span class="monospaced">&lt;group&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Update ha group configuration.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;group&gt;</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The HA group identifier.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--comment</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Description.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--delete</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
A list of settings you want to delete.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--digest</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Prevent changes if current configuration file has a different digest. This can be used to prevent concurrent modifications.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--nodes</span> <span class="monospaced">&lt;node&gt;[:&lt;pri&gt;]{,&lt;node&gt;[:&lt;pri&gt;]}*</span> 
</dt>
<dd>
<p>
List of cluster node names with optional priority.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--nofailback</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
The CRM tries to run services on the node with the highest priority. If a node with higher priority comes online, the CRM migrates the service to that node. Enabling nofailback prevents that behavior.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--restricted</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Resources bound to restricted groups may only run on nodes defined by the group.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>ha-manager help</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Get help about specified command.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--extra-args</span> <span class="monospaced">&lt;array&gt;</span> 
</dt>
<dd>
<p>
Shows help for a specific command
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--verbose</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Verbose output format.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>ha-manager migrate</strong></p></div>
<div class="paragraph">
<p>An alias for <em>ha-manager crm-command migrate</em>.</p></div>
<div class="paragraph">
<p><strong>ha-manager relocate</strong></p></div>
<div class="paragraph">
<p>An alias for <em>ha-manager crm-command relocate</em>.</p></div>
<div class="paragraph">
<p><strong>ha-manager remove</strong> <span class="monospaced">&lt;sid&gt;</span></p></div>
<div class="paragraph">
<p>Delete resource configuration.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;sid&gt;</span>: <span class="monospaced">&lt;type&gt;:&lt;name&gt;</span> 
</dt>
<dd>
<p>
HA resource ID. This consists of a resource type followed by a resource specific name, separated with colon (example: vm:100 / ct:100). For virtual machines and containers, you can simply use the VM or CT id as a shortcut (example: 100).
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>ha-manager set</strong> <span class="monospaced">&lt;sid&gt;</span> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Update resource configuration.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">&lt;sid&gt;</span>: <span class="monospaced">&lt;type&gt;:&lt;name&gt;</span> 
</dt>
<dd>
<p>
HA resource ID. This consists of a resource type followed by a resource specific name, separated with colon (example: vm:100 / ct:100). For virtual machines and containers, you can simply use the VM or CT id as a shortcut (example: 100).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--comment</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Description.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--delete</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
A list of settings you want to delete.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--digest</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Prevent changes if current configuration file has a different digest. This can be used to prevent concurrent modifications.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--group</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
The HA group identifier.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--max_relocate</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Maximal number of service relocate tries when a service failes to start.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--max_restart</span> <span class="monospaced">&lt;integer&gt; (0 - N)</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Maximal number of tries to restart the service on a node after its start failed.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--state</span> <span class="monospaced">&lt;disabled | enabled | ignored | started | stopped&gt;</span> (<em>default =</em> <span class="monospaced">started</span>)
</dt>
<dd>
<p>
Requested resource state.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>ha-manager status</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Display HA manger status.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--verbose</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Verbose output. Include complete CRM and LRM status (JSON).
</p>
</dd>
</dl></div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_service_daemons">23. Appendix B: Service Daemons</h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="_strong_pve_firewall_strong_proxmox_ve_firewall_daemon">
<span>23.1. <strong>pve-firewall</strong> - Proxmox VE Firewall Daemon</span>
 <a class="headerlink" href="#_strong_pve_firewall_strong_proxmox_ve_firewall_daemon" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p><strong>pve-firewall</strong> <span class="monospaced">&lt;COMMAND&gt; [ARGS] [OPTIONS]</span></p></div>
<div class="paragraph">
<p><strong>pve-firewall compile</strong></p></div>
<div class="paragraph">
<p>Compile and print firewall rules. This is useful for testing.</p></div>
<div class="paragraph">
<p><strong>pve-firewall help</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Get help about specified command.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--extra-args</span> <span class="monospaced">&lt;array&gt;</span> 
</dt>
<dd>
<p>
Shows help for a specific command
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--verbose</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Verbose output format.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pve-firewall localnet</strong></p></div>
<div class="paragraph">
<p>Print information about local network.</p></div>
<div class="paragraph">
<p><strong>pve-firewall restart</strong></p></div>
<div class="paragraph">
<p>Restart the Proxmox VE firewall service.</p></div>
<div class="paragraph">
<p><strong>pve-firewall simulate</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Simulate firewall rules. This does not simulates the kernel <em>routing</em>
table, but simply assumes that routing from source zone to destination zone
is possible.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--dest</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Destination IP address.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--dport</span> <span class="monospaced">&lt;integer&gt;</span> 
</dt>
<dd>
<p>
Destination port.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--from</span> <span class="monospaced">(host|outside|vm\d+|ct\d+|([a-zA-Z][a-zA-Z0-9]{0,9})/(\S+))</span> (<em>default =</em> <span class="monospaced">outside</span>)
</dt>
<dd>
<p>
Source zone.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--protocol</span> <span class="monospaced">(tcp|udp)</span> (<em>default =</em> <span class="monospaced">tcp</span>)
</dt>
<dd>
<p>
Protocol.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--source</span> <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Source IP address.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--sport</span> <span class="monospaced">&lt;integer&gt;</span> 
</dt>
<dd>
<p>
Source port.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--to</span> <span class="monospaced">(host|outside|vm\d+|ct\d+|([a-zA-Z][a-zA-Z0-9]{0,9})/(\S+))</span> (<em>default =</em> <span class="monospaced">host</span>)
</dt>
<dd>
<p>
Destination zone.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--verbose</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Verbose output.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pve-firewall start</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Start the Proxmox VE firewall service.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--debug</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Debug mode - stay in foreground
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pve-firewall status</strong></p></div>
<div class="paragraph">
<p>Get firewall status.</p></div>
<div class="paragraph">
<p><strong>pve-firewall stop</strong></p></div>
<div class="paragraph">
<p>Stop the Proxmox VE firewall service. Note, stopping actively removes all
Proxmox VE related iptable rules rendering the host potentially
unprotected.</p></div>
</div>
<div class="sect2">
<h3 id="_strong_pvedaemon_strong_proxmox_ve_api_daemon">
<span>23.2. <strong>pvedaemon</strong> - Proxmox VE API Daemon</span>
 <a class="headerlink" href="#_strong_pvedaemon_strong_proxmox_ve_api_daemon" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p><strong>pvedaemon</strong> <span class="monospaced">&lt;COMMAND&gt; [ARGS] [OPTIONS]</span></p></div>
<div class="paragraph">
<p><strong>pvedaemon help</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Get help about specified command.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--extra-args</span> <span class="monospaced">&lt;array&gt;</span> 
</dt>
<dd>
<p>
Shows help for a specific command
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--verbose</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Verbose output format.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvedaemon restart</strong></p></div>
<div class="paragraph">
<p>Restart the daemon (or start if not running).</p></div>
<div class="paragraph">
<p><strong>pvedaemon start</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Start the daemon.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--debug</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Debug mode - stay in foreground
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvedaemon status</strong></p></div>
<div class="paragraph">
<p>Get daemon status.</p></div>
<div class="paragraph">
<p><strong>pvedaemon stop</strong></p></div>
<div class="paragraph">
<p>Stop the daemon.</p></div>
</div>
<div class="sect2">
<h3 id="_strong_pveproxy_strong_proxmox_ve_api_proxy_daemon">
<span>23.3. <strong>pveproxy</strong> - Proxmox VE API Proxy Daemon</span>
 <a class="headerlink" href="#_strong_pveproxy_strong_proxmox_ve_api_proxy_daemon" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p><strong>pveproxy</strong> <span class="monospaced">&lt;COMMAND&gt; [ARGS] [OPTIONS]</span></p></div>
<div class="paragraph">
<p><strong>pveproxy help</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Get help about specified command.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--extra-args</span> <span class="monospaced">&lt;array&gt;</span> 
</dt>
<dd>
<p>
Shows help for a specific command
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--verbose</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Verbose output format.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveproxy restart</strong></p></div>
<div class="paragraph">
<p>Restart the daemon (or start if not running).</p></div>
<div class="paragraph">
<p><strong>pveproxy start</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Start the daemon.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--debug</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Debug mode - stay in foreground
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pveproxy status</strong></p></div>
<div class="paragraph">
<p>Get daemon status.</p></div>
<div class="paragraph">
<p><strong>pveproxy stop</strong></p></div>
<div class="paragraph">
<p>Stop the daemon.</p></div>
</div>
<div class="sect2">
<h3 id="_strong_pvestatd_strong_proxmox_ve_status_daemon">
<span>23.4. <strong>pvestatd</strong> - Proxmox VE Status Daemon</span>
 <a class="headerlink" href="#_strong_pvestatd_strong_proxmox_ve_status_daemon" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p><strong>pvestatd</strong> <span class="monospaced">&lt;COMMAND&gt; [ARGS] [OPTIONS]</span></p></div>
<div class="paragraph">
<p><strong>pvestatd help</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Get help about specified command.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--extra-args</span> <span class="monospaced">&lt;array&gt;</span> 
</dt>
<dd>
<p>
Shows help for a specific command
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--verbose</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Verbose output format.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvestatd restart</strong></p></div>
<div class="paragraph">
<p>Restart the daemon (or start if not running).</p></div>
<div class="paragraph">
<p><strong>pvestatd start</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Start the daemon.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--debug</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Debug mode - stay in foreground
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvestatd status</strong></p></div>
<div class="paragraph">
<p>Get daemon status.</p></div>
<div class="paragraph">
<p><strong>pvestatd stop</strong></p></div>
<div class="paragraph">
<p>Stop the daemon.</p></div>
</div>
<div class="sect2">
<h3 id="_strong_spiceproxy_strong_spice_proxy_service">
<span>23.5. <strong>spiceproxy</strong> - SPICE Proxy Service</span>
 <a class="headerlink" href="#_strong_spiceproxy_strong_spice_proxy_service" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p><strong>spiceproxy</strong> <span class="monospaced">&lt;COMMAND&gt; [ARGS] [OPTIONS]</span></p></div>
<div class="paragraph">
<p><strong>spiceproxy help</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Get help about specified command.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--extra-args</span> <span class="monospaced">&lt;array&gt;</span> 
</dt>
<dd>
<p>
Shows help for a specific command
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--verbose</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Verbose output format.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>spiceproxy restart</strong></p></div>
<div class="paragraph">
<p>Restart the daemon (or start if not running).</p></div>
<div class="paragraph">
<p><strong>spiceproxy start</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Start the daemon.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--debug</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Debug mode - stay in foreground
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>spiceproxy status</strong></p></div>
<div class="paragraph">
<p>Get daemon status.</p></div>
<div class="paragraph">
<p><strong>spiceproxy stop</strong></p></div>
<div class="paragraph">
<p>Stop the daemon.</p></div>
</div>
<div class="sect2">
<h3 id="_strong_pmxcfs_strong_proxmox_cluster_file_system">
<span>23.6. <strong>pmxcfs</strong> - Proxmox Cluster File System</span>
 <a class="headerlink" href="#_strong_pmxcfs_strong_proxmox_cluster_file_system" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p><strong>pmxcfs</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Help Options:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">-h</span>, <span class="monospaced">--help</span>
</dt>
<dd>
<p>
Show help options
</p>
</dd>
</dl></div>
<div class="paragraph">
<p>Application Options:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">-d</span>, <span class="monospaced">--debug</span>
</dt>
<dd>
<p>
Turn on debug messages
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">-f</span>, <span class="monospaced">--foreground</span>
</dt>
<dd>
<p>
Do not daemonize server
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">-l</span>, <span class="monospaced">--local</span>
</dt>
<dd>
<p>
Force local mode (ignore corosync.conf, force quorum)
</p>
</dd>
</dl></div>
<div class="paragraph">
<p>This service is usually started and managed using systemd toolset. The
service is called <em>pve-cluster</em>.</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>systemctl start pve-cluster</pre>
</div></div>
<div class="literalblock">
<div class="content monospaced">
<pre>systemctl stop pve-cluster</pre>
</div></div>
<div class="literalblock">
<div class="content monospaced">
<pre>systemctl status pve-cluster</pre>
</div></div>
</div>
<div class="sect2">
<h3 id="_strong_pve_ha_crm_strong_cluster_resource_manager_daemon">
<span>23.7. <strong>pve-ha-crm</strong> - Cluster Resource Manager Daemon</span>
 <a class="headerlink" href="#_strong_pve_ha_crm_strong_cluster_resource_manager_daemon" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p><strong>pve-ha-crm</strong> <span class="monospaced">&lt;COMMAND&gt; [ARGS] [OPTIONS]</span></p></div>
<div class="paragraph">
<p><strong>pve-ha-crm help</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Get help about specified command.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--extra-args</span> <span class="monospaced">&lt;array&gt;</span> 
</dt>
<dd>
<p>
Shows help for a specific command
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--verbose</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Verbose output format.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pve-ha-crm start</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Start the daemon.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--debug</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Debug mode - stay in foreground
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pve-ha-crm status</strong></p></div>
<div class="paragraph">
<p>Get daemon status.</p></div>
<div class="paragraph">
<p><strong>pve-ha-crm stop</strong></p></div>
<div class="paragraph">
<p>Stop the daemon.</p></div>
</div>
<div class="sect2">
<h3 id="_strong_pve_ha_lrm_strong_local_resource_manager_daemon">
<span>23.8. <strong>pve-ha-lrm</strong> - Local Resource Manager Daemon</span>
 <a class="headerlink" href="#_strong_pve_ha_lrm_strong_local_resource_manager_daemon" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p><strong>pve-ha-lrm</strong> <span class="monospaced">&lt;COMMAND&gt; [ARGS] [OPTIONS]</span></p></div>
<div class="paragraph">
<p><strong>pve-ha-lrm help</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Get help about specified command.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--extra-args</span> <span class="monospaced">&lt;array&gt;</span> 
</dt>
<dd>
<p>
Shows help for a specific command
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--verbose</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Verbose output format.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pve-ha-lrm start</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Start the daemon.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--debug</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Debug mode - stay in foreground
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pve-ha-lrm status</strong></p></div>
<div class="paragraph">
<p>Get daemon status.</p></div>
<div class="paragraph">
<p><strong>pve-ha-lrm stop</strong></p></div>
<div class="paragraph">
<p>Stop the daemon.</p></div>
</div>
<div class="sect2">
<h3 id="_strong_pvescheduler_strong_proxmox_ve_scheduler_daemon">
<span>23.9. <strong>pvescheduler</strong> - Proxmox VE Scheduler Daemon</span>
 <a class="headerlink" href="#_strong_pvescheduler_strong_proxmox_ve_scheduler_daemon" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p><strong>pvescheduler</strong> <span class="monospaced">&lt;COMMAND&gt; [ARGS] [OPTIONS]</span></p></div>
<div class="paragraph">
<p><strong>pvescheduler help</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Get help about specified command.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--extra-args</span> <span class="monospaced">&lt;array&gt;</span> 
</dt>
<dd>
<p>
Shows help for a specific command
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">--verbose</span> <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Verbose output format.
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvescheduler restart</strong></p></div>
<div class="paragraph">
<p>Restart the daemon (or start if not running).</p></div>
<div class="paragraph">
<p><strong>pvescheduler start</strong> <span class="monospaced">[OPTIONS]</span></p></div>
<div class="paragraph">
<p>Start the daemon.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">--debug</span> <span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Debug mode - stay in foreground
</p>
</dd>
</dl></div>
<div class="paragraph">
<p><strong>pvescheduler status</strong></p></div>
<div class="paragraph">
<p>Get daemon status.</p></div>
<div class="paragraph">
<p><strong>pvescheduler stop</strong></p></div>
<div class="paragraph">
<p>Stop the daemon.</p></div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_configuration_files_2">24. Appendix C: Configuration Files</h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="datacenter_configuration_file">
<span>24.1. Datacenter Configuration</span>
 <a class="headerlink" href="#datacenter_configuration_file" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>The file <span class="monospaced">/etc/pve/datacenter.cfg</span> is a configuration file for
Proxmox VE. It contains cluster wide default values used by all nodes.</p></div>
<div class="sect3">
<h4 id="_file_format_3">24.1.1. File Format
 <a class="headerlink" href="#_file_format_3" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>The file uses a simple colon separated key/value format. Each line has
the following format:</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>OPTION: value</pre>
</div></div>
<div class="paragraph">
<p>Blank lines in the file are ignored, and lines starting with a <span class="monospaced">#</span>
character are treated as comments and are also ignored.</p></div>
</div>
<div class="sect3">
<h4 id="_options">24.1.2. Options
 <a class="headerlink" href="#_options" title="Permalink to this heading"></a>
</h4>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">bwlimit</span>: <span class="monospaced">[clone=&lt;LIMIT&gt;] [,default=&lt;LIMIT&gt;] [,migration=&lt;LIMIT&gt;] [,move=&lt;LIMIT&gt;] [,restore=&lt;LIMIT&gt;]</span> 
</dt>
<dd>
<p>
Set I/O bandwidth limit for various operations (in KiB/s).
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">clone</span>=<span class="monospaced">&lt;LIMIT&gt;</span> 
</dt>
<dd>
<p>
bandwidth limit in KiB/s for cloning disks
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">default</span>=<span class="monospaced">&lt;LIMIT&gt;</span> 
</dt>
<dd>
<p>
default bandwidth limit in KiB/s
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">migration</span>=<span class="monospaced">&lt;LIMIT&gt;</span> 
</dt>
<dd>
<p>
bandwidth limit in KiB/s for migrating guests (including moving local disks)
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">move</span>=<span class="monospaced">&lt;LIMIT&gt;</span> 
</dt>
<dd>
<p>
bandwidth limit in KiB/s for moving disks
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">restore</span>=<span class="monospaced">&lt;LIMIT&gt;</span> 
</dt>
<dd>
<p>
bandwidth limit in KiB/s for restoring guests from backups
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">console</span>: <span class="monospaced">&lt;applet | html5 | vv | xtermjs&gt;</span> 
</dt>
<dd>
<p>
Select the default Console viewer. You can either use the builtin java applet (VNC; deprecated and maps to html5), an external virt-viewer comtatible application (SPICE), an HTML5 based vnc viewer (noVNC), or an HTML5 based console client (xtermjs). If the selected viewer is not available (e.g. SPICE not activated for the VM), the fallback is noVNC.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">crs</span>: <span class="monospaced">[ha=&lt;basic|static&gt;] [,ha-rebalance-on-start=&lt;1|0&gt;]</span> 
</dt>
<dd>
<p>
Cluster resource scheduling settings.
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">ha</span>=<span class="monospaced">&lt;basic | static&gt;</span> (<em>default =</em> <span class="monospaced">basic</span>)
</dt>
<dd>
<p>
Configures how the HA manager should select nodes to start or recover services. With <em>basic</em>, only the number of services is used, with <em>static</em>, static CPU and memory configuration of services is considered.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">ha-rebalance-on-start</span>=<span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Set to use CRS for selecting a suited node when a HA services request-state changes from stop to start.
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">description</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Datacenter description. Shown in the web-interface datacenter notes panel. This is saved as comment inside the configuration file.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">email_from</span>: <span class="monospaced">&lt;string&gt;</span> 
</dt>
<dd>
<p>
Specify email address to send notification from (default is root@$hostname)
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">fencing</span>: <span class="monospaced">&lt;both | hardware | watchdog&gt;</span> (<em>default =</em> <span class="monospaced">watchdog</span>)
</dt>
<dd>
<p>
Set the fencing mode of the HA cluster. Hardware mode needs a valid configuration of fence devices in /etc/pve/ha/fence.cfg. With both all two modes are used.
</p>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Warning" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAMVUlEQVRogdWZeXDVVZbHP7/f27JB
wtJIiCERRFlbx5FuHRrRBgtBsRIwCCOrFmGmiDBjYVNlQlgiQjU6IjI4xLJxGf5QGp0Cbaftsu3R
hu6aYXqgLZoWEsjyyDP7S972e7/l3vnj5cW3Ji9M/zOn6lRS997fvd/vueece+59ipSS/89iv5mP
ZEQQQsS23RQARVEAUFUVRVFQog0ZyogJSClld3c327Ztw7IsLMuKto90KgBsNhtVVVXMnj2bvLw8
7Ha7HBEJKWXGKoSQXV1dcsuWLfLSpUsyKkKIIdWyrLTqdrvlU089Jc+cOSM9Ho8Mh8NSCCEzxTRi
8JWVlbKtre0vAt6yLGmapmxtbZXr1q2TZ86ckW1tbSMikTH4GzduyKqqqkHwsSA1TZOhUChOg8Hg
oBqGIQ3DSAk+VleuXDliEspwviullD09PbzwwgscO3Yszt91XScQCNDZ2YlhGIPfuFwu7Pb48FJV
lZycnEG/z8/PRwiBqqqDYzweD88//zyrV69m7ty5jBs3DofDMWRgD0lASik9Hg979uzh2LFjcYEa
CoVob2/n2q9+hbZ585BGSCWltbVMr61NtSYbN25k7dq1zJkzZ1gSaqrGWPD79+9PAh8MBuns7OTi
e+8R2rwZCYOaqTTt3cuf6+qSwAMcP36co0eP8vXXX9Pd3Y1hGMg0lk65A1HwdXV1vPHGG0mW7+jo
4A8/+xn2BAAAI0riQOnu3cyork7Zt3btWtavX89dd92VdieSCEgppdvt5sCBAxw5ciSuLxQK0dTU
RMOHHyJ37hwh1NSiAKU7djDzxRdT9q9YsYJt27YxY8aMlCTiCEgp5eXLl6mvr+fVV1+NmygYDNLW
1sY3J09ipLEYQDPw2sBfCTwMLARKgKwhiNxWU8OsXbvi2qLYnnzySSorK1PvRGyqvH79uty6dWtS
Lg8EAvLKlSvyo1275M8hSV8H+eMBvRPkP9Vslp0tV6W3u12+c7hOTgT5NyAXgtwE8gTIUyn0D9XV
0jAMqet6klZUVMjPP/88KcWqUcs3NjZy+PBhDh06FGeFqNtcOnECY8+euICVwDWgFtj/m1/wD+8f
5xtg6uy5ZI+5BSkkU6fexgu7tvLWhd/x1tXL5P/905wAggnzSODavn388Sc/SblDL7/8Mq+99hqX
L1+OC2zb7t27uX79+u7Dhw+ndJvW1lYaP/qI0EDKS1z0XeDjC7/l9llz+d6EWxjv0imdcgeFRcVo
fR46vm1Cx8ndP5hP3qjR/GDuX/GbC5/iberl1qirxGjv73+PYRjc8tBDcVhGjRpFRUUF27dvp7i4
mPz8fFwuV2QHVq1aldLyDQ0NXDt5En91NRIQCRqNHld2HsLUCfu6GD9mNIoVQggDy9RQpYLTbkdV
7Vh6CH9vJ/fcfz/vALsAN2AlzHv1wAEu1tTExUI0Hl555RXq6upobm4mEAhECEyePDkOvK7rtLS0
cOP0aby1tUnAo2oNkFAUSTjoRfN+ix7wgjRBShACVVFQRGSkHvTypwtnmXHPQhrbb3D0k1McAdpi
DBLVhp/+lIvV1UlV7oQJE8jNzcXtduPz+ZIPMtM0aWhooPH99+mtrU1ymVRqUxUQFlJYSBFGCivG
OSyEaSCFAGlhd+Vy7w/n43BmM3POXWx69hl8AyMTDXTl4EEu7t2bCBGAQCCAruupCdjtdrp3705r
+ViN22YhsUwdyzRBCpACyzIRMnpngCxXNjYlMk4L+ggoYXxp5pbAlX37cDgcSQSEEAxmoVgQg/9n
AP67bVdAUcFmQ3W6IgtIiWUKLCGwLGvQj69e+m+EsBB6iGB/N/3e9oyNlIgRYm5kiR2JH0L6MkFY
Ei0UQKAycdJUbIodT3Mjpu6n3xfCptoRlomQCqZpRkhLCykMhLBQSV1HJbalKnvSF3MpNO0OSIEw
DWyuPCaVzCRv7ER8Ph+mtJM35lZsdjvuxktYhobDkYOCgqJEDWLLaAfSVc1p78SJO2AAvUBggLUN
KIx2Kgqjv3crY2wOiqaqKKoKihKJCctCC/npbW+jq7sbS7UjkGCZCCkQQkdJsV6mRWHGBH4HLHpx
F3LcGDq6u7nw2S85++V/4gb0kB9hFoAZwuZwgc2BqjpQVRWbzYXd4cDlysHn7WJi0W2EfH24XA6E
jKySqhTPtDRPIjB4XUtoPw3sKV/O+KJipJT4V63mq3/7Obu21/LN/5xl9l/PIys7G2fOaBS7E5vN
QrU7UW0KNlUFp5P8MeO5/8HFhPo7CXr7MMIaQW8neSkMlqnEEYj1s8QJbwXaWpopmDgJh93OqHGF
LFy9jjunTeHNo2uAvUy54/uMuaUYV24BUpVYpoVEwaY4UVUbitNJ9ugxOBw2tEAPfX29QCjuVB9O
EmNBTdeRGEj3A56OdkwtgDDCCKFjz8ql6N55/G3VcRouX+SXH72Ht6udcCg4cJiJSPIn8oCloIKU
KAhURUUIiVCjx128pjtrEiVtFkrMNNOBA+s34XG3YIRDSMtAkSbOrDzuvOchHnmiklEFY3n76G7O
f/nv9Ht7EMJCSjGoljCxjOBAnaSjhYPowevDZrx04FMSGLwnJEzmAOYCn33wAUYoiGXoSMsEoeNw
ZTO2aDrzFpWxck0VJVOn4fd2E/T7oneNyOFlGkjLwNTD6FoILRQEa/hDcyhJIuB0OlFVNeVE04B3
XjnC+d9+hRYKYJlhpDBBGDidDiaUfp/xk24nJ3c0eaPzsKsgzEhtZJkGIuzDNDRCAR99fb309XYh
NAYPsnQ6IgJSSgoKCihavDilJZ4Aajf8HVf/fAnLCCNNAyktECaqIsgbV8So0WMHrn2RGLBMHREO
YBg6eiiIFgzQ7/US8vmYoEDuENa/u6oqcwJRPysoKOC+N9+kcNGiJGvkAQuADY+twn3tCuFg/4CV
DbB0VKHhzHbhdGVhs6kgDKTuR9f86KEAWtBHd0c73p52pB5Av5KewJxnnmHeoUOEw+EkjHEEYi8M
EHlFKyws5IfHjlG4cGHSxJOA1cATPy7nT388j+73YoYDCEuP1DhSRrKOlFhmmLAWQNMCBHw9tHtu
8K2nBcsI8uGxf8HoiRBINNTtjz7KQ/X1xL6iRDHGvubZAfr7+1Nuz+TJk6G+nv/atInmX/86rq8A
WAlUrahk/Fio/8XHjM4fi93pHCgRFCzLQtd1tFAQLeCjs9NNb4cHxdL44J8PozfD7XyXRqNy97p1
PPz220gp0XU9zriKogw+SaqqGtmB48ePU15enkRASklRURH31tcz6cEHk1JrAfAUcEcPLLjvMU69
/694uzsI+X3093TS5blBl8eNt72VHk8Tfd9eR+vv4MQ/HkZpiJwttgTL3/Hoo2nBNzY2smjRIhYs
WEBBQQEulyvyLuT3+2VTUxM7d+7k5MmTKXNuS0sL555+mtYvv0zqE0A7kavhqFL4oAlCCWOygGIi
qTiHyMnuTBgz/bHHKDt9GillnN8D+P1+ysrKWLJkCdOmTWPWrFmUlpZGCAghpN/vp7m5mZqaGl5/
/XUKCwtJFLfbzX+sX8+Nr75K6rtZiXr49KVLKf/4Y4QQ6LqeBP6RRx6hvLycadOmMXPmTEpKSsjN
zf3uZS6WRHV1NadOnUpaTEpJa2sr5zZupOkmSaQqk2csWcLyTz4BQNO0uL7GxkbWr1+fErzNZlMG
w1lVVSUvL4+SkhL27dvH8uXL8Xg8ceABiouLue+ttyieN2+IK2Z6TRx/5+LFacE3NDRQWVmZFjwk
nAOxJF566SW2bNnChQsXkmKipKSEB959l9vmzcvo1pZOpz/8ME98+mla8Bs2bGDZsmVpwUOa5/VE
d0oXE319fXxWXs43Z88m9Q0n03/0I1YPJIRE8OfOnWPHjh2UlZUNCT4tgUQSO3fupKqqigceeGCw
P/a7VM8emUg0VcbK+fPn2bp1KytWrBgW/JAEEknU1NTw7LPPMn/+/GFBZQI8lZw/f57nnnuO5cuX
M2XKlGHBD0sgkURdXR2PP/44FRUVNwV+qP7Tp09z8ODBjNxmRAQSSRw4cIBly5axdOnSvwh4RVH4
4osv2L9/P2VlZRlbfkQEEkls376djo4OQqHE83bk4nK5EEKwZs0aSktLRwR+RATgOxItLS1cu3aN
vr6+wTfK/4tEfzeeMmUKkydPzhg8jJAAREgEg0H6+vrQNC2pFL8ZUVWVrKws8vPzycnJQVXVjH/s
/F/lgJiyQFHragAAAABJRU5ErkJggg==">
</td>
<td class="content"><em>hardware</em> and <em>both</em> are EXPERIMENTAL &amp; WIP</td>
</tr></tbody></table>
</div>
</dd>
<dt class="hdlist1">
<span class="monospaced">ha</span>: <span class="monospaced">shutdown_policy=&lt;enum&gt;</span> 
</dt>
<dd>
<p>
Cluster wide HA settings.
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">shutdown_policy</span>=<span class="monospaced">&lt;conditional | failover | freeze | migrate&gt;</span> (<em>default =</em> <span class="monospaced">conditional</span>)
</dt>
<dd>
<p>
Describes the policy for handling HA services on poweroff or reboot of a node. Freeze will always freeze services which are still located on the node on shutdown, those services won’t be recovered by the HA manager. Failover will not mark the services as frozen and thus the services will get recovered to other nodes, if the shutdown node does not come up again quickly (&lt; 1min). <em>conditional</em> chooses automatically depending on the type of shutdown, i.e., on a reboot the service will be frozen but on a poweroff the service will stay as is, and thus get recovered after about 2 minutes. Migrate will try to move all running services to another node when a reboot or shutdown was triggered. The poweroff process will only continue once no running services are located on the node anymore. If the node comes up again, the service will be moved back to the previously powered-off node, at least if no other migration, reloaction or recovery took place.
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">http_proxy</span>: <span class="monospaced">http://.*</span> 
</dt>
<dd>
<p>
Specify external http proxy which is used for downloads (example: <em>http://username:password@host:port/</em>)
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">keyboard</span>: <span class="monospaced">&lt;da | de | de-ch | en-gb | en-us | es | fi | fr | fr-be | fr-ca | fr-ch | hu | is | it | ja | lt | mk | nl | no | pl | pt | pt-br | sl | sv | tr&gt;</span> 
</dt>
<dd>
<p>
Default keybord layout for vnc server.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">language</span>: <span class="monospaced">&lt;ar | ca | da | de | en | es | eu | fa | fr | he | hr | it | ja | ka | kr | nb | nl | nn | pl | pt_BR | ru | sl | sv | tr | ukr | zh_CN | zh_TW&gt;</span> 
</dt>
<dd>
<p>
Default GUI language.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">mac_prefix</span>: <span class="monospaced">&lt;string&gt;</span> (<em>default =</em> <span class="monospaced">BC:24:11</span>)
</dt>
<dd>
<p>
Prefix for the auto-generated MAC addresses of virtual guests. The default <span class="monospaced">BC:24:11</span> is the Organizationally Unique Identifier (OUI) assigned by the IEEE to Proxmox Server Solutions GmbH for a MAC Address Block Large (MA-L). You’re allowed to use this in local networks, i.e., those not directly reachable by the public (e.g., in a LAN or NAT/Masquerading).
</p>
</dd>
</dl></div>
<div class="paragraph">
<p>Note that when you run multiple cluster that (partially) share the networks of their virtual guests, it’s highly recommended that you extend the default MAC prefix, or generate a custom (valid) one, to reduce the chance of MAC collisions. For example, add a separate extra hexadecimal to the Proxmox OUI for each cluster, like <span class="monospaced">BC:24:11:0</span> for the first, <span class="monospaced">BC:24:11:1</span> for the second, and so on.
 Alternatively, you can also separate the networks of the guests logically, e.g., by using VLANs.</p></div>
<div class="paragraph">
<p>+
For publicly accessible guests it’s recommended that you get your own <a href="https://standards.ieee.org/products-programs/regauth/">OUI from the IEEE</a> registered or coordinate with your, or your hosting providers, network admins.</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">max_workers</span>: <span class="monospaced">&lt;integer&gt; (1 - N)</span> 
</dt>
<dd>
<p>
Defines how many workers (per node) are maximal started  on actions like <em>stopall VMs</em> or task from the ha-manager.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">migration</span>: <span class="monospaced">[type=]&lt;secure|insecure&gt; [,network=&lt;CIDR&gt;]</span> 
</dt>
<dd>
<p>
For cluster wide migration settings.
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">network</span>=<span class="monospaced">&lt;CIDR&gt;</span> 
</dt>
<dd>
<p>
CIDR of the (sub) network that is used for migration.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">type</span>=<span class="monospaced">&lt;insecure | secure&gt;</span> (<em>default =</em> <span class="monospaced">secure</span>)
</dt>
<dd>
<p>
Migration traffic is encrypted using an SSH tunnel by default. On secure, completely private networks this can be disabled to increase performance.
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">migration_unsecure</span>: <span class="monospaced">&lt;boolean&gt;</span> 
</dt>
<dd>
<p>
Migration is secure using SSH tunnel by default. For secure private networks you can disable it to speed up migration. Deprecated, use the <em>migration</em> property instead!
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">next-id</span>: <span class="monospaced">[lower=&lt;integer&gt;] [,upper=&lt;integer&gt;]</span> 
</dt>
<dd>
<p>
Control the range for the free VMID auto-selection pool.
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">lower</span>=<span class="monospaced">&lt;integer&gt;</span> (<em>default =</em> <span class="monospaced">100</span>)
</dt>
<dd>
<p>
Lower, inclusive boundary for free next-id API range.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">upper</span>=<span class="monospaced">&lt;integer&gt;</span> (<em>default =</em> <span class="monospaced">1000000</span>)
</dt>
<dd>
<p>
Upper, exclusive boundary for free next-id API range.
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">notify</span>: <span class="monospaced">[fencing=&lt;always|never&gt;] [,package-updates=&lt;auto|always|never&gt;] [,replication=&lt;always|never&gt;] [,target-fencing=&lt;TARGET&gt;] [,target-package-updates=&lt;TARGET&gt;] [,target-replication=&lt;TARGET&gt;]</span> 
</dt>
<dd>
<p>
Cluster-wide notification settings.
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">fencing</span>=<span class="monospaced">&lt;always | never&gt;</span> 
</dt>
<dd>
<p>
UNUSED - Use datacenter notification settings instead.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">package-updates</span>=<span class="monospaced">&lt;always | auto | never&gt;</span> (<em>default =</em> <span class="monospaced">auto</span>)
</dt>
<dd>
<p>
DEPRECATED: Use datacenter notification settings instead.
Control how often the daily update job should send out notifications:
</p>
<div class="ulist"><ul>
<li>
<p>
<em>auto</em> daily for systems with a valid subscription, as those are assumed to be  production-ready and thus should know about pending updates.
</p>
</li>
<li>
<p>
<em>always</em> every update, if there are new pending updates.
</p>
</li>
<li>
<p>
<em>never</em> never send a notification for new pending updates.
</p>
</li>
</ul></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">replication</span>=<span class="monospaced">&lt;always | never&gt;</span> 
</dt>
<dd>
<p>
UNUSED - Use datacenter notification settings instead.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">target-fencing</span>=<span class="monospaced">&lt;TARGET&gt;</span> 
</dt>
<dd>
<p>
UNUSED - Use datacenter notification settings instead.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">target-package-updates</span>=<span class="monospaced">&lt;TARGET&gt;</span> 
</dt>
<dd>
<p>
UNUSED - Use datacenter notification settings instead.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">target-replication</span>=<span class="monospaced">&lt;TARGET&gt;</span> 
</dt>
<dd>
<p>
UNUSED - Use datacenter notification settings instead.
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">registered-tags</span>: <span class="monospaced">&lt;tag&gt;[;&lt;tag&gt;...]</span> 
</dt>
<dd>
<p>
A list of tags that require a <span class="monospaced">Sys.Modify</span> on <em>/</em> to set and delete. Tags set here that are also in <em>user-tag-access</em> also require <span class="monospaced">Sys.Modify</span>.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">tag-style</span>: <span class="monospaced">[case-sensitive=&lt;1|0&gt;] [,color-map=&lt;tag&gt;:&lt;hex-color&gt;[:&lt;hex-color-for-text&gt;][;&lt;tag&gt;=...]] [,ordering=&lt;config|alphabetical&gt;] [,shape=&lt;enum&gt;]</span> 
</dt>
<dd>
<p>
Tag style options.
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">case-sensitive</span>=<span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">0</span>)
</dt>
<dd>
<p>
Controls if filtering for unique tags on update should check case-sensitive.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">color-map</span>=<span class="monospaced">&lt;tag&gt;:&lt;hex-color&gt;[:&lt;hex-color-for-text&gt;][;&lt;tag&gt;=...]</span> 
</dt>
<dd>
<p>
Manual color mapping for tags (semicolon separated).
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">ordering</span>=<span class="monospaced">&lt;alphabetical | config&gt;</span> (<em>default =</em> <span class="monospaced">alphabetical</span>)
</dt>
<dd>
<p>
Controls the sorting of the tags in the web-interface and the API update.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">shape</span>=<span class="monospaced">&lt;circle | dense | full | none&gt;</span> (<em>default =</em> <span class="monospaced">circle</span>)
</dt>
<dd>
<p>
Tag shape for the web ui tree. <em>full</em> draws the full tag. <em>circle</em> draws only a circle with the background color. <em>dense</em> only draws a small rectancle (useful when many tags are assigned to each guest).<em>none</em> disables showing the tags.
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">u2f</span>: <span class="monospaced">[appid=&lt;APPID&gt;] [,origin=&lt;URL&gt;]</span> 
</dt>
<dd>
<p>
u2f
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">appid</span>=<span class="monospaced">&lt;APPID&gt;</span> 
</dt>
<dd>
<p>
U2F AppId URL override. Defaults to the origin.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">origin</span>=<span class="monospaced">&lt;URL&gt;</span> 
</dt>
<dd>
<p>
U2F Origin override. Mostly useful for single nodes with a single URL.
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">user-tag-access</span>: <span class="monospaced">[user-allow=&lt;enum&gt;] [,user-allow-list=&lt;tag&gt;[;&lt;tag&gt;...]]</span> 
</dt>
<dd>
<p>
Privilege options for user-settable tags
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">user-allow</span>=<span class="monospaced">&lt;existing | free | list | none&gt;</span> (<em>default =</em> <span class="monospaced">free</span>)
</dt>
<dd>
<p>
Controls which tags can be set or deleted on resources a user controls (such as guests). Users with the <span class="monospaced">Sys.Modify</span> privilege on <span class="monospaced">/</span> are alwaysunrestricted.
</p>
<div class="ulist"><ul>
<li>
<p>
<em>none</em> no tags are usable.
</p>
</li>
<li>
<p>
<em>list</em> tags from <em>user-allow-list</em> are usable.
</p>
</li>
<li>
<p>
<em>existing</em> like list, but already existing tags of resources are also usable.
</p>
</li>
<li>
<p>
<em>free</em> no tag restrictions.
</p>
</li>
</ul></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">user-allow-list</span>=<span class="monospaced">&lt;tag&gt;[;&lt;tag&gt;...]</span> 
</dt>
<dd>
<p>
List of tags users are allowed to set and delete (semicolon separated) for <em>user-allow</em> values <em>list</em> and <em>existing</em>.
</p>
</dd>
</dl></div>
</dd>
<dt class="hdlist1">
<span class="monospaced">webauthn</span>: <span class="monospaced">[allow-subdomains=&lt;1|0&gt;] [,id=&lt;DOMAINNAME&gt;] [,origin=&lt;URL&gt;] [,rp=&lt;RELYING_PARTY&gt;]</span> 
</dt>
<dd>
<p>
webauthn configuration
</p>
<div class="dlist"><dl>
<dt class="hdlist1">
<span class="monospaced">allow-subdomains</span>=<span class="monospaced">&lt;boolean&gt;</span> (<em>default =</em> <span class="monospaced">1</span>)
</dt>
<dd>
<p>
Whether to allow the origin to be a subdomain, rather than the exact URL.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">id</span>=<span class="monospaced">&lt;DOMAINNAME&gt;</span> 
</dt>
<dd>
<p>
Relying party ID. Must be the domain name without protocol, port or location. Changing this <strong>will</strong> break existing credentials.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">origin</span>=<span class="monospaced">&lt;URL&gt;</span> 
</dt>
<dd>
<p>
Site origin. Must be a <span class="monospaced">https://</span> URL (or <span class="monospaced">http://localhost</span>). Should contain the address users type in their browsers to access the web interface. Changing this <strong>may</strong> break existing credentials.
</p>
</dd>
<dt class="hdlist1">
<span class="monospaced">rp</span>=<span class="monospaced">&lt;RELYING_PARTY&gt;</span> 
</dt>
<dd>
<p>
Relying party name. Any text identifier. Changing this <strong>may</strong> break existing credentials.
</p>
</dd>
</dl></div>
</dd>
</dl></div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_calendar_events">25. Appendix D: Calendar Events</h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="chapter_calendar_events">
<span>25.1. Schedule Format</span>
 <a class="headerlink" href="#chapter_calendar_events" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Proxmox VE has a very flexible scheduling configuration. It is based on the systemd
time calendar event format.<span class="footnote" data-note="see <span class=&quot;monospaced&quot;>man 7 systemd.time</span> for more information">[<a id="_footnoteref_58" href="#_footnote_58" title="View footnote" class="footnote">58</a>]</span>
Calendar events may be used to refer to one or more points in time in a
single expression.</p></div>
<div class="paragraph">
<p>Such a calendar event uses the following format:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>[WEEKDAY] [[YEARS-]MONTHS-DAYS] [HOURS:MINUTES[:SECONDS]]</pre>
</div></div>
<div class="paragraph">
<p>This format allows you to configure a set of days on which the job should run.
You can also set one or more start times. It tells the replication scheduler
the moments in time when a job should start.
With this information we, can create a job which runs every workday at 10
PM: <span class="monospaced">'mon,tue,wed,thu,fri 22'</span> which could be abbreviated to: <span class="monospaced">'mon..fri
22'</span>, most reasonable schedules can be written quite intuitive this way.</p></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">Hours are formatted in 24-hour format.</td>
</tr></tbody></table>
</div>
<div class="paragraph">
<p>To allow a convenient and shorter configuration, one or more repeat times per
guest can be set. They indicate that replications are done on the start-time(s)
itself and the start-time(s) plus all multiples of the repetition value. If
you want to start replication at 8 AM and repeat it every 15 minutes until
9 AM you would use: <span class="monospaced">'8:00/15'</span></p></div>
<div class="paragraph">
<p>Here you see that if no hour separation (<span class="monospaced">:</span>), is used the value gets
interpreted as minute. If such a separation is used, the value on the left
denotes the hour(s), and the value on the right denotes the minute(s).
Further, you can use <span class="monospaced">*</span> to match all possible values.</p></div>
<div class="paragraph">
<p>To get additional ideas look at
<a href="#pvesr_schedule_format_examples">more Examples below</a>.</p></div>
</div>
<div class="sect2">
<h3 id="_detailed_specification">
<span>25.2. Detailed Specification</span>
 <a class="headerlink" href="#_detailed_specification" title="Permalink to this heading"></a>
</h3>
<div class="dlist"><dl>
<dt class="hdlist1">
weekdays
</dt>
<dd>
<p>
Days are specified with an abbreviated English version: <span class="monospaced">sun, mon,
tue, wed, thu, fri and sat</span>. You may use multiple days as a comma-separated
list. A range of days can also be set by specifying the start and end day
separated by “..”, for example <span class="monospaced">mon..fri</span>. These formats can be mixed.
If omitted <span class="monospaced">'*'</span> is assumed.
</p>
</dd>
<dt class="hdlist1">
time-format
</dt>
<dd>
<p>
A time format consists of hours and minutes interval lists.
Hours and minutes are separated by <span class="monospaced">':'</span>. Both hour and minute can be list
and ranges of values, using the same format as days.
First are hours, then minutes. Hours can be omitted if not needed. In this
case <span class="monospaced">'*'</span> is assumed for the value of hours.
The valid range for values is <span class="monospaced">0-23</span> for hours and <span class="monospaced">0-59</span> for minutes.
</p>
</dd>
</dl></div>
<div class="sect3">
<h4 id="pvesr_schedule_format_examples">25.2.1. Examples:
 <a class="headerlink" href="#pvesr_schedule_format_examples" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>There are some special values that have a specific meaning:</p></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<caption class="title">Table 19. Special Values</caption>
<colgroup><col style="width:50%;">
<col style="width:50%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Value                                  </th>
<th class="tableblock halign-left valign-top">Syntax</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><span class="monospaced">minutely</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><span class="monospaced">*-*-* *:*:00</span></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><span class="monospaced">hourly</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><span class="monospaced">*-*-* *:00:00</span></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><span class="monospaced">daily</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><span class="monospaced">*-*-* 00:00:00</span></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><span class="monospaced">weekly</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><span class="monospaced">mon *-*-* 00:00:00</span></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><span class="monospaced">monthly</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><span class="monospaced">*-*-01 00:00:00</span></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><span class="monospaced">yearly</span> or <span class="monospaced">annually</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><span class="monospaced">*-01-01 00:00:00</span></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><span class="monospaced">quarterly</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><span class="monospaced">*-01,04,07,10-01 00:00:00</span></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><span class="monospaced">semiannually</span> or <span class="monospaced">semi-annually</span></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><span class="monospaced">*-01,07-01 00:00:00</span></p></td>
</tr>
</tbody>
</table>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<caption class="title">Table 20. Schedule Examples</caption>
<colgroup><col style="width:33%;">
<col style="width:33%;">
<col style="width:33%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Schedule String        </th>
<th class="tableblock halign-left valign-top">Alternative            </th>
<th class="tableblock halign-left valign-top">Meaning</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">mon,tue,wed,thu,fri</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">mon..fri</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Every working day at 0:00</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">sat,sun</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">sat..sun</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Only on weekends at 0:00</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">mon,wed,fri</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">— </p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Only on Monday, Wednesday and Friday at 0:00</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">12:05</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">12:05</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Every day at 12:05 PM</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">*/5</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0/5</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Every five minutes</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">mon..wed 30/10</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">mon,tue,wed 30/10</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Monday, Tuesday, Wednesday 30, 40 and 50 minutes after every full hour</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">mon..fri 8..17,22:0/15</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">— </p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Every working day every 15 minutes between 8 AM and 6 PM and between 10 PM and 11 PM</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">fri 12..13:5/20</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">fri 12,13:5/20</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Friday at 12:05, 12:25, 12:45, 13:05, 13:25 and 13:45</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">12,14,16,18,20,22:5</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">12/2:5</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Every day starting at 12:05 until 22:05, every 2 hours</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">*</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">*/1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Every minute (minimum interval)</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">*-05</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">— </p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">On the 5th day of every Month</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">Sat *-1..7 15:00</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">— </p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">First Saturday each Month at 15:00</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">2015-10-21</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">— </p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">21st October 2015 at 00:00</p></td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_qemu_vcpu_list">26. Appendix E: QEMU vCPU List</h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="chapter_qm_vcpu_list">
<span>26.1. Introduction</span>
 <a class="headerlink" href="#chapter_qm_vcpu_list" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>This is a list of AMD and Intel x86-64/amd64 CPU types as defined in QEMU,
going back to 2007.</p></div>
</div>
<div class="sect2">
<h3 id="_intel_cpu_types">
<span>26.2. Intel CPU Types</span>
 <a class="headerlink" href="#_intel_cpu_types" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/List_of_Intel_Xeon_processors">Intel processors</a></p></div>
<div class="ulist"><ul>
<li>
<p>
<em>Nahelem</em> : <a href="https://en.wikipedia.org/wiki/Nehalem_(microarchitecture)">1st generation of the Intel Core processor</a>
</p>
</li>
<li>
<p>
<em>Nahelem-IBRS (v2)</em> : add Spectre v1 protection (<em>+spec-ctrl</em>)
</p>
</li>
<li>
<p>
<em>Westmere</em> : <a href="https://en.wikipedia.org/wiki/Westmere_(microarchitecture)">1st generation of the Intel Core processor (Xeon E7-)</a>
</p>
</li>
<li>
<p>
<em>Westmere-IBRS (v2)</em> : add Spectre v1 protection (<em>+spec-ctrl</em>)
</p>
</li>
<li>
<p>
<em>SandyBridge</em> : <a href="https://en.wikipedia.org/wiki/Sandy_Bridge">2nd generation of the Intel Core processor</a>
</p>
</li>
<li>
<p>
<em>SandyBridge-IBRS (v2)</em> : add Spectre v1 protection (<em>+spec-ctrl</em>)
</p>
</li>
<li>
<p>
<em>IvyBridge</em> : <a href="https://en.wikipedia.org/wiki/Ivy_Bridge_(microarchitecture)">3rd generation of the Intel Core processor</a>
</p>
</li>
<li>
<p>
<em>IvyBridge-IBRS (v2)</em>: add Spectre v1 protection (<em>+spec-ctrl</em>)
</p>
</li>
<li>
<p>
<em>Haswell</em> : <a href="https://en.wikipedia.org/wiki/Haswell_(microarchitecture)">4th generation of the Intel Core processor</a>
</p>
</li>
<li>
<p>
<em>Haswell-noTSX (v2)</em> : disable TSX (<em>-hle</em>, <em>-rtm</em>)
</p>
</li>
<li>
<p>
<em>Haswell-IBRS (v3)</em> : re-add TSX, add Spectre v1 protection (<em>+hle</em>, <em>+rtm</em>,
<em>+spec-ctrl</em>)
</p>
</li>
<li>
<p>
<em>Haswell-noTSX-IBRS (v4)</em> : disable TSX (<em>-hle</em>, <em>-rtm</em>)
</p>
</li>
<li>
<p>
<em>Broadwell</em>: <a href="https://en.wikipedia.org/wiki/Broadwell_(microarchitecture)">5th generation of the Intel Core processor</a>
</p>
</li>
<li>
<p>
<em>Skylake</em>: <a href="https://en.wikipedia.org/wiki/Skylake_(microarchitecture)">1st generation Xeon Scalable server processors</a>
</p>
</li>
<li>
<p>
<em>Skylake-IBRS (v2)</em> : add Spectre v1 protection, disable CLFLUSHOPT
(<em>+spec-ctrl</em>, <em>-clflushopt</em>)
</p>
</li>
<li>
<p>
<em>Skylake-noTSX-IBRS (v3)</em> : disable TSX (<em>-hle</em>, <em>-rtm</em>)
</p>
</li>
<li>
<p>
<em>Skylake-v4</em>: add EPT switching (<em>+vmx-eptp-switching</em>)
</p>
</li>
<li>
<p>
<em>Cascadelake</em>: <a href="https://en.wikipedia.org/wiki/Cascade_Lake_(microprocessor)">2nd generation Xeon Scalable processor</a>
</p>
</li>
<li>
<p>
<em>Cascadelake-v2</em> : add arch_capabilities msr (<em>+arch-capabilities</em>,
<em>+rdctl-no</em>, <em>+ibrs-all</em>, <em>+skip-l1dfl-vmentry</em>, <em>+mds-no</em>)
</p>
</li>
<li>
<p>
<em>Cascadelake-v3</em> : disable TSX (<em>-hle</em>, <em>-rtm</em>)
</p>
</li>
<li>
<p>
<em>Cascadelake-v4</em> : add EPT switching (<em>+vmx-eptp-switching</em>)
</p>
</li>
<li>
<p>
<em>Cascadelake-v5</em> : add XSAVES (<em>+xsaves</em>, <em>+vmx-xsaves</em>)
</p>
</li>
<li>
<p>
<em>Cooperlake</em> : <a href="https://en.wikipedia.org/wiki/Cooper_Lake_(microprocessor)">3rd generation Xeon Scalable processors for 4 &amp; 8 sockets servers</a>
</p>
</li>
<li>
<p>
<em>Cooperlake-v2</em> : add XSAVES (<em>+xsaves</em>, <em>+vmx-xsaves</em>)
</p>
</li>
<li>
<p>
<em>Icelake</em>: <a href="https://en.wikipedia.org/wiki/Ice_Lake_(microprocessor)">3rd generation Xeon Scalable server processors</a>
</p>
</li>
<li>
<p>
<em>Icelake-v2</em> : disable TSX (<em>-hle</em>, <em>-rtm</em>)
</p>
</li>
<li>
<p>
<em>Icelake-v3</em> : add arch_capabilities msr (<em>+arch-capabilities</em>, <em>+rdctl-no</em>,
<em>+ibrs-all</em>, <em>+skip-l1dfl-vmentry</em>, <em>+mds-no</em>, <em>+pschange-mc-no</em>, <em>+taa-no</em>)
</p>
</li>
<li>
<p>
<em>Icelake-v4</em> : add missing flags (<em>+sha-ni</em>, <em>+avx512ifma</em>, <em>+rdpid</em>, <em>+fsrm</em>,
<em>+vmx-rdseed-exit</em>, <em>+vmx-pml</em>, <em>+vmx-eptp-switching</em>)
</p>
</li>
<li>
<p>
<em>Icelake-v5</em> : add XSAVES (<em>+xsaves</em>, <em>+vmx-xsaves</em>)
</p>
</li>
<li>
<p>
<em>Icelake-v6</em> : add "5-level EPT" (<em>+vmx-page-walk-5</em>)
</p>
</li>
<li>
<p>
<em>SapphireRapids</em> : <a href="https://en.wikipedia.org/wiki/Sapphire_Rapids">4th generation Xeon Scalable server processors</a>
</p>
</li>
</ul></div>
</div>
<div class="sect2">
<h3 id="_amd_cpu_types">
<span>26.3. AMD CPU Types</span>
 <a class="headerlink" href="#_amd_cpu_types" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/List_of_AMD_processors">AMD processors</a></p></div>
<div class="ulist"><ul>
<li>
<p>
<em>Opteron_G3</em> : <a href="https://en.wikipedia.org/wiki/AMD_10h">K10</a>
</p>
</li>
<li>
<p>
<em>Opteron_G4</em> : <a href="https://en.wikipedia.org/wiki/Bulldozer_(microarchitecture)">Bulldozer</a>
</p>
</li>
<li>
<p>
<em>Opteron_G5</em> :  <a href="https://en.wikipedia.org/wiki/Piledriver_(microarchitecture)">Piledriver</a>
</p>
</li>
<li>
<p>
<em>EPYC</em> : <a href="https://en.wikipedia.org/wiki/Zen_(first_generation)">1st generation of Zen processors</a>
</p>
</li>
<li>
<p>
<em>EPYC-IBPB (v2)</em> : add Spectre v1 protection (<em>+ibpb</em>)
</p>
</li>
<li>
<p>
<em>EPYC-v3</em> : add missing flags (<em>+perfctr-core</em>, <em>+clzero</em>, <em>+xsaveerptr</em>,
<em>+xsaves</em>)
</p>
</li>
<li>
<p>
<em>EPYC-Rome</em> : <a href="https://en.wikipedia.org/wiki/Zen_2">2nd generation of Zen processors</a>
</p>
</li>
<li>
<p>
<em>EPYC-Rome-v2</em> : add Spectre v2, v4 protection (<em>+ibrs</em>, <em>+amd-ssbd</em>)
</p>
</li>
<li>
<p>
<em>EPYC-Milan</em> : <a href="https://en.wikipedia.org/wiki/Zen_3">3rd generation of Zen processors</a>
</p>
</li>
<li>
<p>
<em>EPYC-Milan-v2</em> : add missing flags (<em>+vaes</em>, <em>+vpclmulqdq</em>,
<em>+stibp-always-on</em>, <em>+amd-psfd</em>, <em>+no-nested-data-bp</em>,
<em>+lfence-always-serializing</em>, <em>+null-sel-clr-base</em>)
</p>
</li>
</ul></div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_firewall_macro_definitions">27. Appendix F: Firewall Macro Definitions</h2>
<div class="sectionbody">
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>Amanda</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Amanda Backup
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">udp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">10080</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">10080</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>Auth</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Auth (identd) traffic
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">113</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>BGP</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Border Gateway Protocol traffic
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">179</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>BitTorrent</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
BitTorrent traffic for BitTorrent 3.1 and earlier
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">6881:6889</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">udp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">6881</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>BitTorrent32</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
BitTorrent traffic for BitTorrent 3.2 and later
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">6881:6999</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">udp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">6881</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>CVS</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Concurrent Versions System pserver traffic
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2401</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>Ceph</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Ceph Storage Cluster traffic (Ceph Monitors, OSD &amp; MDS Daemons)
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">6789</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3300</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">6800:7300</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>Citrix</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Citrix/ICA traffic (ICA, ICA Browser, CGP)
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1494</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">udp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1604</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2598</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>DAAP</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Digital Audio Access Protocol traffic (iTunes, Rythmbox daemons)
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3689</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">udp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3689</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>DCC</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Distributed Checksum Clearinghouse spam filtering mechanism
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">6277</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>DHCPfwd</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Forwarded DHCP traffic
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">udp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">67:68</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">67:68</p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>DHCPv6</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
DHCPv6 traffic
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">udp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">546:547</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">546:547</p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>DNS</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Domain Name System traffic (upd and tcp)
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">udp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">53</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">53</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>Distcc</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Distributed Compiler service
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3632</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>FTP</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
File Transfer Protocol
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">21</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>Finger</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Finger protocol (RFC 742)
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">79</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>GNUnet</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
GNUnet secure peer-to-peer networking traffic
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2086</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">udp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2086</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1080</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">udp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1080</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>GRE</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Generic Routing Encapsulation tunneling protocol
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">47</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>Git</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Git distributed revision control traffic
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">9418</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>HKP</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
OpenPGP HTTP key server protocol traffic
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">11371</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>HTTP</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Hypertext Transfer Protocol (WWW)
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">80</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>HTTPS</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Hypertext Transfer Protocol (WWW) over SSL
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">443</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>ICPV2</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Internet Cache Protocol V2 (Squid) traffic
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">udp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3130</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>ICQ</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
AOL Instant Messenger traffic
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">5190</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>IMAP</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Internet Message Access Protocol
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">143</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>IMAPS</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Internet Message Access Protocol over SSL
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">993</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>IPIP</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
IPIP capsulation traffic
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">94</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>IPsec</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
IPsec traffic
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">udp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">500</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">500</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">50</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>IPsecah</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
IPsec authentication (AH) traffic
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">udp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">500</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">500</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">51</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>IPsecnat</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
IPsec traffic and Nat-Traversal
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">udp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">500</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">udp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">4500</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">50</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>IRC</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Internet Relay Chat traffic
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">6667</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>Jetdirect</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
HP Jetdirect printing
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">9100</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>L2TP</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Layer 2 Tunneling Protocol traffic
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">udp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1701</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>LDAP</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Lightweight Directory Access Protocol traffic
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">389</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>LDAPS</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Secure Lightweight Directory Access Protocol traffic
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">636</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>MDNS</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Multicast DNS
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">udp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">5353</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>MSNP</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Microsoft Notification Protocol
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1863</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>MSSQL</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Microsoft SQL Server
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1433</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>Mail</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Mail traffic (SMTP, SMTPS, Submission)
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">25</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">465</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">587</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>Munin</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Munin networked resource monitoring traffic
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">4949</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>MySQL</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
MySQL server
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3306</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>NNTP</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
NNTP traffic (Usenet).
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">119</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>NNTPS</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Encrypted NNTP traffic (Usenet)
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">563</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>NTP</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Network Time Protocol (ntpd)
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">udp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">123</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>NeighborDiscovery</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
IPv6 neighbor solicitation, neighbor and router advertisement
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">icmpv6</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">router-solicitation</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">icmpv6</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">router-advertisement</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">icmpv6</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">neighbor-solicitation</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">icmpv6</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">neighbor-advertisement</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>OSPF</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
OSPF multicast traffic
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">89</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>OpenVPN</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
OpenVPN traffic
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">udp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1194</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>PCA</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Symantec PCAnywere (tm)
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">udp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">5632</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">5631</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>PMG</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Proxmox Mail Gateway web interface
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">8006</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>POP3</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
POP3 traffic
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">110</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>POP3S</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Encrypted POP3 traffic
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">995</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>PPtP</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Point-to-Point Tunneling Protocol
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">47</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1723</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>Ping</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
ICMP echo request
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">icmp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">echo-request</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>PostgreSQL</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
PostgreSQL server
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">5432</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>Printer</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Line Printer protocol printing
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">515</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>RDP</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Microsoft Remote Desktop Protocol traffic
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3389</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>RIP</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Routing Information Protocol (bidirectional)
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">udp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">520</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>RNDC</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
BIND remote management protocol
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">953</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>Razor</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Razor Antispam System
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2703</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>Rdate</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Remote time retrieval (rdate)
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">37</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>Rsync</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Rsync server
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">873</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>SANE</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
SANE network scanning
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">6566</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>SMB</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Microsoft SMB traffic
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">udp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">135,445</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">udp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">137:139</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">udp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1024:65535</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">137</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">135,139,445</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>SMBswat</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Samba Web Administration Tool
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">901</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>SMTP</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Simple Mail Transfer Protocol
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">25</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>SMTPS</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Encrypted Simple Mail Transfer Protocol
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">465</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>SNMP</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Simple Network Management Protocol
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">udp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">161:162</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">161</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>SPAMD</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Spam Assassin SPAMD traffic
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">783</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>SPICEproxy</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Proxmox VE SPICE display proxy traffic
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3128</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>SSH</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Secure shell traffic
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">22</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>SVN</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Subversion server (svnserve)
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3690</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>SixXS</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
SixXS IPv6 Deployment and Tunnel Broker
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3874</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">udp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3740</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">41</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">udp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">5072,8374</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>Squid</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Squid web proxy traffic
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3128</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>Submission</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Mail message submission traffic
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">587</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>Syslog</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Syslog protocol (RFC 5424) traffic
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">udp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">514</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">514</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>TFTP</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Trivial File Transfer Protocol traffic
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">udp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">69</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>Telnet</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Telnet traffic
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">23</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>Telnets</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Telnet over SSL
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">992</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>Time</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
RFC 868 Time protocol
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">37</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>Trcrt</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Traceroute (for up to 30 hops) traffic
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">udp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">33434:33524</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">icmp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">echo-request</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>VNC</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
VNC traffic for VNC display’s 0 - 99
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">5900:5999</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>VNCL</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
VNC traffic from Vncservers to Vncviewers in listen mode
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">5500</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>Web</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
WWW traffic (HTTP and HTTPS)
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">80</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">443</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>Webcache</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Web Cache/Proxy traffic (port 8080)
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">8080</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>Webmin</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Webmin traffic
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">10000</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
<div class="hdlist"><table>
<tbody><tr>
<td class="hdlist1">
<em>Whois</em>
<br>
</td>
<td class="hdlist2">
<p style="margin-top: 0;">
Whois (nicname, RFC 3912) traffic
</p>
</td>
</tr>
</tbody></table></div>
<table class="tableblock frame-all grid-all" style="
width:100%;
">
<colgroup><col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup><thead>
<tr>
<th class="tableblock halign-left valign-top">Action</th>
<th class="tableblock halign-left valign-top">proto</th>
<th class="tableblock halign-left valign-top">dport</th>
<th class="tableblock halign-left valign-top">sport</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">PARAM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">tcp</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">43</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"></p></td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="sect1">
<h2 id="_markdown_primer">28. Appendix G: Markdown Primer</h2>
<div class="sectionbody">
<div class="quoteblock">
<div class="content">
<div class="paragraph">
<p>Markdown is a text-to-HTML conversion tool for web writers. Markdown allows you
to write using an easy-to-read, easy-to-write plain text format, then convert
it to structurally valid XHTML (or HTML).</p></div>
</div>
<div class="attribution">
<em>https://daringfireball.net/projects/markdown/</em><br>
— John Gruber
</div></div>
<div class="paragraph">
<p>The Proxmox VE web interface has support for using Markdown to rendering rich text
formatting in node and virtual guest notes.</p></div>
<div class="paragraph">
<p>Proxmox VE supports CommonMark with most extensions of GFM (GitHub Flavoured Markdown),
like tables or task-lists.</p></div>
<div class="sect2">
<h3 id="markdown_basics">
<span>28.1. Markdown Basics</span>
 <a class="headerlink" href="#markdown_basics" title="Permalink to this heading"></a>
</h3>
<div class="paragraph">
<p>Note that we only describe the basics here, please search the web for more
extensive resources, for example on <a href="https://www.markdownguide.org/">https://www.markdownguide.org/</a></p></div>
<div class="sect3">
<h4 id="_headings">28.1.1. Headings
 <a class="headerlink" href="#_headings" title="Permalink to this heading"></a>
</h4>
<div class="listingblock">
<div class="content monospaced">
<pre># This is a Heading h1
## This is a Heading h2
##### This is a Heading h5</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_emphasis">28.1.2. Emphasis
 <a class="headerlink" href="#_emphasis" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Use <span class="monospaced">*text*</span> or <span class="monospaced">_text_</span> for emphasis.</p></div>
<div class="paragraph">
<p>Use <span class="monospaced">**text**</span> or <span class="monospaced">__text__</span> for bold, heavy-weight text.</p></div>
<div class="paragraph">
<p>Combinations are also possible, for example:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>_You **can** combine them_</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_links">28.1.3. Links
 <a class="headerlink" href="#_links" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>You can use automatic detection of links, for example,
<span class="monospaced">https://forum.proxmox.com/</span> would transform it into a clickable link.</p></div>
<div class="paragraph">
<p>You can also control the link text, for example:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>Now, [the part in brackets will be the link text](https://forum.proxmox.com/).</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_lists">28.1.4. Lists
 <a class="headerlink" href="#_lists" title="Permalink to this heading"></a>
</h4>
<div class="sect4">
<h5 id="_unordered_lists">Unordered Lists
 <a class="headerlink" href="#_unordered_lists" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Use <span class="monospaced">*</span> or <span class="monospaced">-</span> for unordered lists, for example:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>* Item 1
* Item 2
* Item 2a
* Item 2b</pre>
</div></div>
<div class="paragraph">
<p>Adding an indentation can be used to created nested lists.</p></div>
</div>
<div class="sect4">
<h5 id="_ordered_lists">Ordered Lists
 <a class="headerlink" href="#_ordered_lists" title="Permalink to this heading"></a>
</h5>
<div class="listingblock">
<div class="content monospaced">
<pre>1. Item 1
1. Item 2
1. Item 3
  1. Item 3a
  1. Item 3b</pre>
</div></div>
<div class="admonitionblock">
<table><tbody><tr>
<td class="icon">
<img alt="Note" src="data:image/png;base64,
iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAJhUlEQVRoge2ZWWycVxXHf+fce7/v
m/GaGCde4pI0aQlJC0kRtE1L00JbLIjY4QkeUB9YHhAIJFCExAsKUkE8IAFFPIDUIqhBRSDRBUqC
CimFFBCBpCWx02IaZ3G2SdyxPZ7vHh6+mcnSZnFjKIge6Wj8zYzvPf9z/me5d8TM+F8WfbkNuFx5
BcDLLf/fAEZGRmx4eNh6enqsp6fHhoeHbWRk5D9aFeSlVqHNmzfb6H33sHnT7ZQmD5GfOMax6Sm+
Pl5h1Yc+xpYtW2SBbX1ReUkRGBkZsdH77mHLW95EOv4Ms3ueJh6YYPHUFF9aljJ63z3cf//9/5FI
vKQIDA8P293L2yhVjjH7t51ocDiviFecF46n7XzBreChhx4qNhH5t0XjJUVgx44ddGUZ9b/vIpQD
oRQIWSDJAiFL6B9axo4dO4gxAmANWVDLG+Ln82URMRGhVCqRHxonlAPqFXWKC4r6IhI6OMjMzBN4
/4LlTUQQEZxzZ32QJAlpmrb+p16vU6vVOHXq1AWjN18AnDj0F971vrs4OnmYJVkoDA4FCPUO172I
Cgnt7SV++4vvsGhRJx3tJbIsRVVpsUnOBBABBVFEClKYwbKr7sTM7EIUnBcA7z21k7t49x1X8JXv
bOWra7rw5QRtcN8PLCfvvZJvb9vJycpJpg4/hp/N0I4SMQs4Jw0A5zBXHGiCaIZIKABgpGlKjPEF
0TpT5pUDRXiVt99+Le03r+WzuytM1gO6pB/3+o0cbxvk8yOPMjW6i2iR2lxOjJDHSDMFogmGwzQ7
rRJAUpAENAGXIZq2AFzQpvkACCEQcahP+cRH3sKHn9zHXU+MM7rtGeD33NDXzaZynZU9gcezpUw9
X6OzIyOakkfF4QEpPG6nDRNNEA2FSgKimETSNCXPc0II57VpXhEolUqoOrxPSLOMT330Dv5SqfKD
NR388Y2L+caQsjITNv3pMBs3rOT56ZyZGaM+J0QUxDc0INrWUgggoRGBAOIRAt77hY1AmqaoeJxP
cN645jVDbNn8Hj73o8fZ/af9mEE9j9y2YRXt5YzZWmRmzjj1/BwhTXAKzitOHEbeWlc0AVwDnCv8
KoZzjotV33lTSL1HNKAuEtKM1169jM98/E6mTk3x4Nbd7Bk7TEdHRvAeVY+hmDqmZwx1kIkiqrhz
S2zL+AbNMC6l/80LgHMOEY9oQvBCks5RKpXo7JhFxbhz42pet2aQet1YtLiDJAkIDq8BHwJmwlwO
UaD0ojsrNKuUReIZyb9gABCHcwWFgg+0lTPyvIRToVzKWLpkMfV6REQplYvmZCj1uuBUSdJwTg8A
XBdoCZMOsBkQBeGi/J83gBgjmABC8AlJGsjzFLMyaXDM1etEA0VR50iCx6mSZhkiijpPjEpQD+SF
4WdJrTAewdCFB1CtVlFVVATnhMQnWJqC5aTBk+c5IIgWRoTgSZJAmiZAo1s7hwsppglI+fTiljeY
nyHkLQotKIAYI4igzpFHISQOiwEnKTEG8hhRVZw6YjRQLfJGHcF7jleqTBw8znXr12MABnv37efY
iSnesG4tiUsRUQwD7JIAzKsPqCqiRbVwweM04XdPjhJN+dvTBxgbn6G9q59yZx9/3HWEb33vN+zc
/RzOJ+w/eJLtO8Z5ZNtT7PvHIUQTvvv9X/Lc/mN0d3by3fseRLQwvWh0Fy+h8wbQ2VFG1KM+xfuA
qufo8So/fejPPD/rqJys8pvf7eLAoeNMHqnw2U9+kH3jVQ5MClMzKUla5obr13HliiEMmDx6gltv
uY7Vr1nBQF8PJopQ9AFTt/AROFfMjCW9XTy19xB33Hodb924jr1j+/nDk3/nzTdei4jw3nfeyCOP
bufa1y5jzeoVrcHM8HR3dfHlr/2Q+x94jFtuuh44/9B2PplXDryYLF3STXd3e+t5UXcHY89OsOH6
NS2Qed7wpM1Rm50G4MGHH2P961Zy3bqreXrPP5mrzwLt5y6/cACq1eoLCKm+TN/SAebmfn8aUG83
PYs7+cnPH+eqKwd5as8/edc7bi02847pmVkATk1VWbF8AOcca1Yv59DkqcYK0tCL02deACqVCldc
0YdIwLmEPM9RV6NnUZlPf3wT6oqJcePN6wHhzTeu4/CRCrfctJ4sSxBRli7pYfHEMUSU97/7Th75
1RP8eec+Yp5zzTVXM9DfDyogBvHS6HTJACYmJnjVoq5GFw0454gCEOnoaMfiNGZFFRFxJGkbywZ6
i1NWoy9kWYmbb1gHKCHApuGbisVFGyoYUswa5OR5ftF56JIBjI6OMtDfWwAQ35jnc8AVpRXBohVq
UowECKqK4RBxoE0W6gvGCcEjaOEAwEQWdpgbGxujt7erOLO2mk3R8i0Wz9EiuUGz+qlKEQEUaJbI
4lTHmTVePKgWzpDGJGpc8CDTlEsuo88++wxXDA0UIUbAOP23KGZKjI48KnkuhcbiPbPCOBoeBikO
Lk2VxjqNRilaAEqSZGEAbN261bZt+zW33XY7IAXXm6Ou0YhIMamaaUOl5WRrzg00viuKWWxpQUOH
NAZFQRpD48Xlkig0MTHBB95zG+VSylz1KCbWyDOh2XyK+56IqjWMKigkUnzWnPPFFFRRLZ29SQRU
ELOGY4pZ6LKOlM07mZ07d/KOtw1TcB4sGkTDiDQ9K1IkrKeYmQC08d7pZLSiRBpE5s7aS0XAHFEK
AGY51Wr18g80Zmb33nsvX/z8XdSmj2AWOXhwkrxe46+79jB55Dh/3T2GxUhHextdXW2sXN7PNWtW
IQKDA71FFBoAjIgQkVg/a5+oHrU5zIznDhxk964xKpXKggAoTlWW8+OfPorlOQ//cjsDy1bS2dFO
W+diVly1iL6+Pqanpzl5qsL4pPHwN3/G1InDlMsZ7Z1tDA30cfWqIa5dexV9fb2YnT7UTxw4xsHJ
o4yOjfOP8QOMjx/k4OQx7r777lY0zycXvJ02M4sxMjg4SL1eR0TYsGEDw8PD9PX10d7ejogUN3a1
GqpKCIE8z5mdnUVVqVarbN++nba2Nvbu3csDDzyAqrJ8+atb+zjn6e/vZ/Xq1axatYq1a9fS29tL
lmUMDQ1RKpXOm9EXvV6v1+tWr9eZnp5mZmaGWq1GjLHF62aiNZ+bnPfe45xrvTZzxMyYmZk56+LX
zKjX661DvHOOJElIkoRSqYT3/vLvRlW15eHCa4VxzdvmpjZDfubzuXeb3vuzqCEixBhbo0NTkyS5
PAr9L8j/96+U/w3yCoCXW14B8HLLvwDd67nwZIEPdgAAAABJRU5ErkJggg==">
</td>
<td class="content">The integer of ordered lists does not need to be correct, they will be numbered automatically.</td>
</tr></tbody></table>
</div>
</div>
<div class="sect4">
<h5 id="_task_lists">Task Lists
 <a class="headerlink" href="#_task_lists" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Task list use a empty box <span class="monospaced">[ ]</span> for unfinished tasks and a box with an <span class="monospaced">X</span> for finished tasks.</p></div>
<div class="paragraph">
<p>For example:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>- [X] First task already done!
- [X] Second one too
- [ ] This one is still to-do
- [ ] So is this one</pre>
</div></div>
</div>
</div>
<div class="sect3">
<h4 id="_tables">28.1.5. Tables
 <a class="headerlink" href="#_tables" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>Tables use the pipe symbol <span class="monospaced">|</span> to separate columns, and <span class="monospaced">-</span> to separate the
table header from the table body, in that separation one can also set the text
alignment, making one column left-, center-, or right-aligned.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>| Left columns  | Right columns |  Some  | More | Cols.| Centering Works Too
| ------------- |--------------:|--------|------|------|:------------------:|
| left foo      | right foo     | First  | Row  | Here | &gt;center&lt;           |
| left bar      | right bar     | Second | Row  | Here | 12345              |
| left baz      | right baz     | Third  | Row  | Here | Test               |
| left zab      | right zab     | Fourth | Row  | Here | ☁️☁️☁️              |
| left rab      | right rab     | And    | Last | Here | The End            |</pre>
</div></div>
<div class="paragraph">
<p>Note that you do not need to align the columns nicely with white space, but that makes
editing tables easier.</p></div>
</div>
<div class="sect3">
<h4 id="_block_quotes">28.1.6. Block Quotes
 <a class="headerlink" href="#_block_quotes" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>You can enter block quotes by prefixing a line with <span class="monospaced">&gt;</span>, similar as in plain-text emails.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>&gt; Markdown is a lightweight markup language with plain-text-formatting syntax,
&gt; created in 2004 by John Gruber with Aaron Swartz.
&gt;
&gt;&gt; Markdown is often used to format readme files, for writing messages in online discussion forums,
&gt;&gt; and to create rich text using a plain text editor.</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_code_and_snippets">28.1.7. Code and Snippets
 <a class="headerlink" href="#_code_and_snippets" title="Permalink to this heading"></a>
</h4>
<div class="paragraph">
<p>You can use backticks to avoid processing for a few word or paragraphs. That is useful for
avoiding that a code or configuration hunk gets mistakenly interpreted as markdown.</p></div>
<div class="sect4">
<h5 id="_inline_code">Inline code
 <a class="headerlink" href="#_inline_code" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>Surrounding part of a line with single backticks allows to write code inline,
for examples:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>This hosts IP address is `10.0.0.1`.</pre>
</div></div>
</div>
<div class="sect4">
<h5 id="_whole_blocks_of_code">Whole blocks of code
 <a class="headerlink" href="#_whole_blocks_of_code" title="Permalink to this heading"></a>
</h5>
<div class="paragraph">
<p>For code blocks spanning several lines you can use triple-backticks to start
and end such a block, for example:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>```
# This is the network config I want to remember here
auto vmbr2
iface vmbr2 inet static
        address 10.0.0.1/24
        bridge-ports ens20
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

```</pre>
</div></div>
</div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_gnu_free_documentation_license">29. Appendix H: GNU Free Documentation License</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Version 1.3, 3 November 2008</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>Copyright (C) 2000, 2001, 2002, 2007, 2008 Free Software Foundation, Inc.
    &lt;http://fsf.org/&gt;
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.</pre>
</div></div>
<div class="paragraph">
<div class="title">0. PREAMBLE</div><p>The purpose of this License is to make a manual, textbook, or other
functional and useful document "free" in the sense of freedom: to
assure everyone the effective freedom to copy and redistribute it,
with or without modifying it, either commercially or noncommercially.
Secondarily, this License preserves for the author and publisher a way
to get credit for their work, while not being considered responsible
for modifications made by others.</p></div>
<div class="paragraph">
<p>This License is a kind of "copyleft", which means that derivative
works of the document must themselves be free in the same sense.  It
complements the GNU General Public License, which is a copyleft
license designed for free software.</p></div>
<div class="paragraph">
<p>We have designed this License in order to use it for manuals for free
software, because free software needs free documentation: a free
program should come with manuals providing the same freedoms that the
software does.  But this License is not limited to software manuals;
it can be used for any textual work, regardless of subject matter or
whether it is published as a printed book.  We recommend this License
principally for works whose purpose is instruction or reference.</p></div>
<div class="paragraph">
<div class="title">1. APPLICABILITY AND DEFINITIONS</div><p>This License applies to any manual or other work, in any medium, that
contains a notice placed by the copyright holder saying it can be
distributed under the terms of this License.  Such a notice grants a
world-wide, royalty-free license, unlimited in duration, to use that
work under the conditions stated herein.  The "Document", below,
refers to any such manual or work.  Any member of the public is a
licensee, and is addressed as "you".  You accept the license if you
copy, modify or distribute the work in a way requiring permission
under copyright law.</p></div>
<div class="paragraph">
<p>A "Modified Version" of the Document means any work containing the
Document or a portion of it, either copied verbatim, or with
modifications and/or translated into another language.</p></div>
<div class="paragraph">
<p>A "Secondary Section" is a named appendix or a front-matter section of
the Document that deals exclusively with the relationship of the
publishers or authors of the Document to the Document’s overall
subject (or to related matters) and contains nothing that could fall
directly within that overall subject.  (Thus, if the Document is in
part a textbook of mathematics, a Secondary Section may not explain
any mathematics.)  The relationship could be a matter of historical
connection with the subject or with related matters, or of legal,
commercial, philosophical, ethical or political position regarding
them.</p></div>
<div class="paragraph">
<p>The "Invariant Sections" are certain Secondary Sections whose titles
are designated, as being those of Invariant Sections, in the notice
that says that the Document is released under this License.  If a
section does not fit the above definition of Secondary then it is not
allowed to be designated as Invariant.  The Document may contain zero
Invariant Sections.  If the Document does not identify any Invariant
Sections then there are none.</p></div>
<div class="paragraph">
<p>The "Cover Texts" are certain short passages of text that are listed,
as Front-Cover Texts or Back-Cover Texts, in the notice that says that
the Document is released under this License.  A Front-Cover Text may
be at most 5 words, and a Back-Cover Text may be at most 25 words.</p></div>
<div class="paragraph">
<p>A "Transparent" copy of the Document means a machine-readable copy,
represented in a format whose specification is available to the
general public, that is suitable for revising the document
straightforwardly with generic text editors or (for images composed of
pixels) generic paint programs or (for drawings) some widely available
drawing editor, and that is suitable for input to text formatters or
for automatic translation to a variety of formats suitable for input
to text formatters.  A copy made in an otherwise Transparent file
format whose markup, or absence of markup, has been arranged to thwart
or discourage subsequent modification by readers is not Transparent.
An image format is not Transparent if used for any substantial amount
of text.  A copy that is not "Transparent" is called "Opaque".</p></div>
<div class="paragraph">
<p>Examples of suitable formats for Transparent copies include plain
ASCII without markup, Texinfo input format, LaTeX input format, SGML
or XML using a publicly available DTD, and standard-conforming simple
HTML, PostScript or PDF designed for human modification.  Examples of
transparent image formats include PNG, XCF and JPG.  Opaque formats
include proprietary formats that can be read and edited only by
proprietary word processors, SGML or XML for which the DTD and/or
processing tools are not generally available, and the
machine-generated HTML, PostScript or PDF produced by some word
processors for output purposes only.</p></div>
<div class="paragraph">
<p>The "Title Page" means, for a printed book, the title page itself,
plus such following pages as are needed to hold, legibly, the material
this License requires to appear in the title page.  For works in
formats which do not have any title page as such, "Title Page" means
the text near the most prominent appearance of the work’s title,
preceding the beginning of the body of the text.</p></div>
<div class="paragraph">
<p>The "publisher" means any person or entity that distributes copies of
the Document to the public.</p></div>
<div class="paragraph">
<p>A section "Entitled XYZ" means a named subunit of the Document whose
title either is precisely XYZ or contains XYZ in parentheses following
text that translates XYZ in another language.  (Here XYZ stands for a
specific section name mentioned below, such as "Acknowledgements",
"Dedications", "Endorsements", or "History".)  To "Preserve the Title"
of such a section when you modify the Document means that it remains a
section "Entitled XYZ" according to this definition.</p></div>
<div class="paragraph">
<p>The Document may include Warranty Disclaimers next to the notice which
states that this License applies to the Document.  These Warranty
Disclaimers are considered to be included by reference in this
License, but only as regards disclaiming warranties: any other
implication that these Warranty Disclaimers may have is void and has
no effect on the meaning of this License.</p></div>
<div class="paragraph">
<div class="title">2. VERBATIM COPYING</div><p>You may copy and distribute the Document in any medium, either
commercially or noncommercially, provided that this License, the
copyright notices, and the license notice saying this License applies
to the Document are reproduced in all copies, and that you add no
other conditions whatsoever to those of this License.  You may not use
technical measures to obstruct or control the reading or further
copying of the copies you make or distribute.  However, you may accept
compensation in exchange for copies.  If you distribute a large enough
number of copies you must also follow the conditions in section 3.</p></div>
<div class="paragraph">
<p>You may also lend copies, under the same conditions stated above, and
you may publicly display copies.</p></div>
<div class="paragraph">
<div class="title">3. COPYING IN QUANTITY</div><p>If you publish printed copies (or copies in media that commonly have
printed covers) of the Document, numbering more than 100, and the
Document’s license notice requires Cover Texts, you must enclose the
copies in covers that carry, clearly and legibly, all these Cover
Texts: Front-Cover Texts on the front cover, and Back-Cover Texts on
the back cover.  Both covers must also clearly and legibly identify
you as the publisher of these copies.  The front cover must present
the full title with all words of the title equally prominent and
visible.  You may add other material on the covers in addition.
Copying with changes limited to the covers, as long as they preserve
the title of the Document and satisfy these conditions, can be treated
as verbatim copying in other respects.</p></div>
<div class="paragraph">
<p>If the required texts for either cover are too voluminous to fit
legibly, you should put the first ones listed (as many as fit
reasonably) on the actual cover, and continue the rest onto adjacent
pages.</p></div>
<div class="paragraph">
<p>If you publish or distribute Opaque copies of the Document numbering
more than 100, you must either include a machine-readable Transparent
copy along with each Opaque copy, or state in or with each Opaque copy
a computer-network location from which the general network-using
public has access to download using public-standard network protocols
a complete Transparent copy of the Document, free of added material.
If you use the latter option, you must take reasonably prudent steps,
when you begin distribution of Opaque copies in quantity, to ensure
that this Transparent copy will remain thus accessible at the stated
location until at least one year after the last time you distribute an
Opaque copy (directly or through your agents or retailers) of that
edition to the public.</p></div>
<div class="paragraph">
<p>It is requested, but not required, that you contact the authors of the
Document well before redistributing any large number of copies, to
give them a chance to provide you with an updated version of the
Document.</p></div>
<div class="paragraph">
<div class="title">4. MODIFICATIONS</div><p>You may copy and distribute a Modified Version of the Document under
the conditions of sections 2 and 3 above, provided that you release
the Modified Version under precisely this License, with the Modified
Version filling the role of the Document, thus licensing distribution
and modification of the Modified Version to whoever possesses a copy
of it.  In addition, you must do these things in the Modified Version:</p></div>
<div class="olist upperalpha"><ol class="upperalpha">
<li>
<p>
Use in the Title Page (and on the covers, if any) a title distinct
   from that of the Document, and from those of previous versions
   (which should, if there were any, be listed in the History section
   of the Document).  You may use the same title as a previous version
   if the original publisher of that version gives permission.
</p>
</li>
<li>
<p>
List on the Title Page, as authors, one or more persons or entities
   responsible for authorship of the modifications in the Modified
   Version, together with at least five of the principal authors of the
   Document (all of its principal authors, if it has fewer than five),
   unless they release you from this requirement.
</p>
</li>
<li>
<p>
State on the Title page the name of the publisher of the
   Modified Version, as the publisher.
</p>
</li>
<li>
<p>
Preserve all the copyright notices of the Document.
</p>
</li>
<li>
<p>
Add an appropriate copyright notice for your modifications
   adjacent to the other copyright notices.
</p>
</li>
<li>
<p>
Include, immediately after the copyright notices, a license notice
   giving the public permission to use the Modified Version under the
   terms of this License, in the form shown in the Addendum below.
</p>
</li>
<li>
<p>
Preserve in that license notice the full lists of Invariant Sections
   and required Cover Texts given in the Document’s license notice.
</p>
</li>
<li>
<p>
Include an unaltered copy of this License.
</p>
</li>
<li>
<p>
Preserve the section Entitled "History", Preserve its Title, and add
   to it an item stating at least the title, year, new authors, and
   publisher of the Modified Version as given on the Title Page.  If
   there is no section Entitled "History" in the Document, create one
   stating the title, year, authors, and publisher of the Document as
   given on its Title Page, then add an item describing the Modified
   Version as stated in the previous sentence.
</p>
</li>
<li>
<p>
Preserve the network location, if any, given in the Document for
   public access to a Transparent copy of the Document, and likewise
   the network locations given in the Document for previous versions
   it was based on.  These may be placed in the "History" section.
   You may omit a network location for a work that was published at
   least four years before the Document itself, or if the original
   publisher of the version it refers to gives permission.
</p>
</li>
<li>
<p>
For any section Entitled "Acknowledgements" or "Dedications",
   Preserve the Title of the section, and preserve in the section all
   the substance and tone of each of the contributor acknowledgements
   and/or dedications given therein.
</p>
</li>
<li>
<p>
Preserve all the Invariant Sections of the Document,
   unaltered in their text and in their titles.  Section numbers
   or the equivalent are not considered part of the section titles.
</p>
</li>
<li>
<p>
Delete any section Entitled "Endorsements".  Such a section
   may not be included in the Modified Version.
</p>
</li>
<li>
<p>
Do not retitle any existing section to be Entitled "Endorsements"
   or to conflict in title with any Invariant Section.
</p>
</li>
<li>
<p>
Preserve any Warranty Disclaimers.
</p>
</li>
</ol></div>
<div class="paragraph">
<p>If the Modified Version includes new front-matter sections or
appendices that qualify as Secondary Sections and contain no material
copied from the Document, you may at your option designate some or all
of these sections as invariant.  To do this, add their titles to the
list of Invariant Sections in the Modified Version’s license notice.
These titles must be distinct from any other section titles.</p></div>
<div class="paragraph">
<p>You may add a section Entitled "Endorsements", provided it contains
nothing but endorsements of your Modified Version by various
parties—for example, statements of peer review or that the text has
been approved by an organization as the authoritative definition of a
standard.</p></div>
<div class="paragraph">
<p>You may add a passage of up to five words as a Front-Cover Text, and a
passage of up to 25 words as a Back-Cover Text, to the end of the list
of Cover Texts in the Modified Version.  Only one passage of
Front-Cover Text and one of Back-Cover Text may be added by (or
through arrangements made by) any one entity.  If the Document already
includes a cover text for the same cover, previously added by you or
by arrangement made by the same entity you are acting on behalf of,
you may not add another; but you may replace the old one, on explicit
permission from the previous publisher that added the old one.</p></div>
<div class="paragraph">
<p>The author(s) and publisher(s) of the Document do not by this License
give permission to use their names for publicity for or to assert or
imply endorsement of any Modified Version.</p></div>
<div class="paragraph">
<div class="title">5. COMBINING DOCUMENTS</div><p>You may combine the Document with other documents released under this
License, under the terms defined in section 4 above for modified
versions, provided that you include in the combination all of the
Invariant Sections of all of the original documents, unmodified, and
list them all as Invariant Sections of your combined work in its
license notice, and that you preserve all their Warranty Disclaimers.</p></div>
<div class="paragraph">
<p>The combined work need only contain one copy of this License, and
multiple identical Invariant Sections may be replaced with a single
copy.  If there are multiple Invariant Sections with the same name but
different contents, make the title of each such section unique by
adding at the end of it, in parentheses, the name of the original
author or publisher of that section if known, or else a unique number.
Make the same adjustment to the section titles in the list of
Invariant Sections in the license notice of the combined work.</p></div>
<div class="paragraph">
<p>In the combination, you must combine any sections Entitled "History"
in the various original documents, forming one section Entitled
"History"; likewise combine any sections Entitled "Acknowledgements",
and any sections Entitled "Dedications".  You must delete all sections
Entitled "Endorsements".</p></div>
<div class="paragraph">
<div class="title">6. COLLECTIONS OF DOCUMENTS</div><p>You may make a collection consisting of the Document and other
documents released under this License, and replace the individual
copies of this License in the various documents with a single copy
that is included in the collection, provided that you follow the rules
of this License for verbatim copying of each of the documents in all
other respects.</p></div>
<div class="paragraph">
<p>You may extract a single document from such a collection, and
distribute it individually under this License, provided you insert a
copy of this License into the extracted document, and follow this
License in all other respects regarding verbatim copying of that
document.</p></div>
<div class="paragraph">
<div class="title">7. AGGREGATION WITH INDEPENDENT WORKS</div><p>A compilation of the Document or its derivatives with other separate
and independent documents or works, in or on a volume of a storage or
distribution medium, is called an "aggregate" if the copyright
resulting from the compilation is not used to limit the legal rights
of the compilation’s users beyond what the individual works permit.
When the Document is included in an aggregate, this License does not
apply to the other works in the aggregate which are not themselves
derivative works of the Document.</p></div>
<div class="paragraph">
<p>If the Cover Text requirement of section 3 is applicable to these
copies of the Document, then if the Document is less than one half of
the entire aggregate, the Document’s Cover Texts may be placed on
covers that bracket the Document within the aggregate, or the
electronic equivalent of covers if the Document is in electronic form.
Otherwise they must appear on printed covers that bracket the whole
aggregate.</p></div>
<div class="paragraph">
<div class="title">8. TRANSLATION</div><p>Translation is considered a kind of modification, so you may
distribute translations of the Document under the terms of section 4.
Replacing Invariant Sections with translations requires special
permission from their copyright holders, but you may include
translations of some or all Invariant Sections in addition to the
original versions of these Invariant Sections.  You may include a
translation of this License, and all the license notices in the
Document, and any Warranty Disclaimers, provided that you also include
the original English version of this License and the original versions
of those notices and disclaimers.  In case of a disagreement between
the translation and the original version of this License or a notice
or disclaimer, the original version will prevail.</p></div>
<div class="paragraph">
<p>If a section in the Document is Entitled "Acknowledgements",
"Dedications", or "History", the requirement (section 4) to Preserve
its Title (section 1) will typically require changing the actual
title.</p></div>
<div class="paragraph">
<div class="title">9. TERMINATION</div><p>You may not copy, modify, sublicense, or distribute the Document
except as expressly provided under this License.  Any attempt
otherwise to copy, modify, sublicense, or distribute it is void, and
will automatically terminate your rights under this License.</p></div>
<div class="paragraph">
<p>However, if you cease all violation of this License, then your license
from a particular copyright holder is reinstated (a) provisionally,
unless and until the copyright holder explicitly and finally
terminates your license, and (b) permanently, if the copyright holder
fails to notify you of the violation by some reasonable means prior to
60 days after the cessation.</p></div>
<div class="paragraph">
<p>Moreover, your license from a particular copyright holder is
reinstated permanently if the copyright holder notifies you of the
violation by some reasonable means, this is the first time you have
received notice of violation of this License (for any work) from that
copyright holder, and you cure the violation prior to 30 days after
your receipt of the notice.</p></div>
<div class="paragraph">
<p>Termination of your rights under this section does not terminate the
licenses of parties who have received copies or rights from you under
this License.  If your rights have been terminated and not permanently
reinstated, receipt of a copy of some or all of the same material does
not give you any rights to use it.</p></div>
<div class="paragraph">
<div class="title">10. FUTURE REVISIONS OF THIS LICENSE</div><p>The Free Software Foundation may publish new, revised versions of the
GNU Free Documentation License from time to time.  Such new versions
will be similar in spirit to the present version, but may differ in
detail to address new problems or concerns.  See
<a href="http://www.gnu.org/copyleft/">http://www.gnu.org/copyleft/</a>.</p></div>
<div class="paragraph">
<p>Each version of the License is given a distinguishing version number.
If the Document specifies that a particular numbered version of this
License "or any later version" applies to it, you have the option of
following the terms and conditions either of that specified version or
of any later version that has been published (not as a draft) by the
Free Software Foundation.  If the Document does not specify a version
number of this License, you may choose any version ever published (not
as a draft) by the Free Software Foundation.  If the Document
specifies that a proxy can decide which future versions of this
License can be used, that proxy’s public statement of acceptance of a
version permanently authorizes you to choose that version for the
Document.</p></div>
<div class="paragraph">
<div class="title">11. RELICENSING</div><p>"Massive Multiauthor Collaboration Site" (or "MMC Site") means any
World Wide Web server that publishes copyrightable works and also
provides prominent facilities for anybody to edit those works.  A
public wiki that anybody can edit is an example of such a server.  A
"Massive Multiauthor Collaboration" (or "MMC") contained in the site
means any set of copyrightable works thus published on the MMC site.</p></div>
<div class="paragraph">
<p>"CC-BY-SA" means the Creative Commons Attribution-Share Alike 3.0
license published by Creative Commons Corporation, a not-for-profit
corporation with a principal place of business in San Francisco,
California, as well as future copyleft versions of that license
published by that same organization.</p></div>
<div class="paragraph">
<p>"Incorporate" means to publish or republish a Document, in whole or in
part, as part of another Document.</p></div>
<div class="paragraph">
<p>An MMC is "eligible for relicensing" if it is licensed under this
License, and if all works that were first published under this License
somewhere other than this MMC, and subsequently incorporated in whole or
in part into the MMC, (1) had no cover texts or invariant sections, and
(2) were thus incorporated prior to November 1, 2008.</p></div>
<div class="paragraph">
<p>The operator of an MMC Site may republish an MMC contained in the site
under CC-BY-SA on the same site at any time before August 1, 2009,
provided the MMC is eligible for relicensing.</p></div>
</div>
</div>
</div>
<div id="footnotes"><hr><div class="footnote" id="_footnote_1"><a href="#_footnoteref_1" title="Return to text">1</a>. smartmontools homepage <a href="https://www.smartmontools.org">https://www.smartmontools.org</a></div><div class="footnote" id="_footnote_2"><a href="#_footnoteref_2" title="Return to text">2</a>. OpenZFS dRAID
<a href="https://openzfs.github.io/openzfs-docs/Basic%20Concepts/dRAID%20Howto.html">https://openzfs.github.io/openzfs-docs/Basic%20Concepts/dRAID%20Howto.html</a></div><div class="footnote" id="_footnote_3"><a href="#_footnoteref_3" title="Return to text">3</a>. Systems installed with Proxmox VE 6.4 or later,
EFI systems installed with Proxmox VE 5.4 or later</div><div class="footnote" id="_footnote_4"><a href="#_footnoteref_4" title="Return to text">4</a>. <a href="https://bugzilla.proxmox.com/show_bug.cgi?id=2350">https://bugzilla.proxmox.com/show_bug.cgi?id=2350</a></div><div class="footnote" id="_footnote_5"><a href="#_footnoteref_5" title="Return to text">5</a>. <a href="https://github.com/openzfs/zfs/issues/11688">https://github.com/openzfs/zfs/issues/11688</a></div><div class="footnote" id="_footnote_6"><a href="#_footnoteref_6" title="Return to text">6</a>. acme.sh <a href="https://github.com/acmesh-official/acme.sh">https://github.com/acmesh-official/acme.sh</a></div><div class="footnote" id="_footnote_7"><a href="#_footnoteref_7" title="Return to text">7</a>. These are all installs with root on <span class="monospaced">ext4</span> or <span class="monospaced">xfs</span> and installs
with root on ZFS on non-EFI systems</div><div class="footnote" id="_footnote_8"><a href="#_footnoteref_8" title="Return to text">8</a>. Booting ZFS on root with GRUB
<a href="https://github.com/zfsonlinux/zfs/wiki/Debian-Stretch-Root-on-ZFS">https://github.com/zfsonlinux/zfs/wiki/Debian-Stretch-Root-on-ZFS</a></div><div class="footnote" id="_footnote_9"><a href="#_footnoteref_9" title="Return to text">9</a>. GRUB Manual <a href="https://www.gnu.org/software/grub/manual/grub/grub.html">https://www.gnu.org/software/grub/manual/grub/grub.html</a></div><div class="footnote" id="_footnote_10"><a href="#_footnoteref_10" title="Return to text">10</a>. Systems using <span class="monospaced">proxmox-boot-tool</span> will call <span class="monospaced">proxmox-boot-tool
refresh</span> upon <span class="monospaced">update-grub</span>.</div><div class="footnote" id="_footnote_11"><a href="#_footnoteref_11" title="Return to text">11</a>. <span class="monospaced">votequorum_qdevice_master_wins</span> manual page
   <a href="https://manpages.debian.org/bookworm/libvotequorum-dev/votequorum_qdevice_master_wins.3.en.html">https://manpages.debian.org/bookworm/libvotequorum-dev/votequorum_qdevice_master_wins.3.en.html</a></div><div class="footnote" id="_footnote_12"><a href="#_footnoteref_12" title="Return to text">12</a>. <a href="https://docs.ceph.com/en/quincy/rados/operations/user-management/">Ceph User Management</a></div><div class="footnote" id="_footnote_13"><a href="#_footnoteref_13" title="Return to text">13</a>. RBD configuration reference
<a href="https://docs.ceph.com/en/quincy/rbd/rbd-config-ref/">https://docs.ceph.com/en/quincy/rbd/rbd-config-ref/</a></div><div class="footnote" id="_footnote_14"><a href="#_footnoteref_14" title="Return to text">14</a>. Ceph intro <a href="https://docs.ceph.com/en/quincy/start/intro/">https://docs.ceph.com/en/quincy/start/intro/</a></div><div class="footnote" id="_footnote_15"><a href="#_footnoteref_15" title="Return to text">15</a>. Ceph architecture <a href="https://docs.ceph.com/en/quincy/architecture/">https://docs.ceph.com/en/quincy/architecture/</a></div><div class="footnote" id="_footnote_16"><a href="#_footnoteref_16" title="Return to text">16</a>. Ceph glossary <a href="https://docs.ceph.com/en/quincy/glossary">https://docs.ceph.com/en/quincy/glossary</a></div><div class="footnote" id="_footnote_17"><a href="#_footnoteref_17" title="Return to text">17</a>. Full Mesh Network for Ceph <a href="https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server">https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server</a></div><div class="footnote" id="_footnote_18"><a href="#_footnoteref_18" title="Return to text">18</a>. Ceph Monitor <a href="https://docs.ceph.com/en/quincy/start/intro/">https://docs.ceph.com/en/quincy/start/intro/</a></div><div class="footnote" id="_footnote_19"><a href="#_footnoteref_19" title="Return to text">19</a>. Ceph Manager <a href="https://docs.ceph.com/en/quincy/mgr/">https://docs.ceph.com/en/quincy/mgr/</a></div><div class="footnote" id="_footnote_20"><a href="#_footnoteref_20" title="Return to text">20</a>. Ceph Bluestore <a href="https://ceph.com/community/new-luminous-bluestore/">https://ceph.com/community/new-luminous-bluestore/</a></div><div class="footnote" id="_footnote_21"><a href="#_footnoteref_21" title="Return to text">21</a>. PG calculator <a href="https://web.archive.org/web/20210301111112/http://ceph.com/pgcalc/">https://web.archive.org/web/20210301111112/http://ceph.com/pgcalc/</a></div><div class="footnote" id="_footnote_22"><a href="#_footnoteref_22" title="Return to text">22</a>. Placement Groups <a href="https://docs.ceph.com/en/quincy/rados/operations/placement-groups/">https://docs.ceph.com/en/quincy/rados/operations/placement-groups/</a></div><div class="footnote" id="_footnote_23"><a href="#_footnoteref_23" title="Return to text">23</a>. Automated Scaling <a href="https://docs.ceph.com/en/quincy/rados/operations/placement-groups/#automated-scaling">https://docs.ceph.com/en/quincy/rados/operations/placement-groups/#automated-scaling</a></div><div class="footnote" id="_footnote_24"><a href="#_footnoteref_24" title="Return to text">24</a>. Ceph pool operation
<a href="https://docs.ceph.com/en/quincy/rados/operations/pools/">https://docs.ceph.com/en/quincy/rados/operations/pools/</a></div><div class="footnote" id="_footnote_25"><a href="#_footnoteref_25" title="Return to text">25</a>. Ceph Erasure Coded Pool Recovery
<a href="https://docs.ceph.com/en/quincy/rados/operations/erasure-code/#erasure-coded-pool-recovery">https://docs.ceph.com/en/quincy/rados/operations/erasure-code/#erasure-coded-pool-recovery</a></div><div class="footnote" id="_footnote_26"><a href="#_footnoteref_26" title="Return to text">26</a>. Ceph Erasure Code Profile
<a href="https://docs.ceph.com/en/quincy/rados/operations/erasure-code/#erasure-code-profiles">https://docs.ceph.com/en/quincy/rados/operations/erasure-code/#erasure-code-profiles</a></div><div class="footnote" id="_footnote_27"><a href="#_footnoteref_27" title="Return to text">27</a>. CRUSH
<a href="https://ceph.com/wp-content/uploads/2016/08/weil-crush-sc06.pdf">https://ceph.com/wp-content/uploads/2016/08/weil-crush-sc06.pdf</a></div><div class="footnote" id="_footnote_28"><a href="#_footnoteref_28" title="Return to text">28</a>. CRUSH map <a href="https://docs.ceph.com/en/quincy/rados/operations/crush-map/">https://docs.ceph.com/en/quincy/rados/operations/crush-map/</a></div><div class="footnote" id="_footnote_29"><a href="#_footnoteref_29" title="Return to text">29</a>. Configuring multiple active MDS daemons
<a href="https://docs.ceph.com/en/quincy/cephfs/multimds/">https://docs.ceph.com/en/quincy/cephfs/multimds/</a></div><div class="footnote" id="_footnote_30"><a href="#_footnoteref_30" title="Return to text">30</a>. Ceph scrubbing <a href="https://docs.ceph.com/en/quincy/rados/configuration/osd-config-ref/#scrubbing">https://docs.ceph.com/en/quincy/rados/configuration/osd-config-ref/#scrubbing</a></div><div class="footnote" id="_footnote_31"><a href="#_footnoteref_31" title="Return to text">31</a>. Ceph log and debugging <a href="https://docs.ceph.com/en/quincy/rados/troubleshooting/log-and-debug/">https://docs.ceph.com/en/quincy/rados/troubleshooting/log-and-debug/</a></div><div class="footnote" id="_footnote_32"><a href="#_footnoteref_32" title="Return to text">32</a>. Ceph troubleshooting <a href="https://docs.ceph.com/en/quincy/rados/troubleshooting/">https://docs.ceph.com/en/quincy/rados/troubleshooting/</a></div><div class="footnote" id="_footnote_33"><a href="#_footnoteref_33" title="Return to text">33</a>. See
this benchmark on the KVM wiki <a href="https://www.linux-kvm.org/page/Using_VirtIO_NIC">https://www.linux-kvm.org/page/Using_VirtIO_NIC</a></div><div class="footnote" id="_footnote_34"><a href="#_footnoteref_34" title="Return to text">34</a>. See this benchmark for details
 <a href="https://events.static.linuxfound.org/sites/events/files/slides/CloudOpen2013_Khoa_Huynh_v3.pdf">https://events.static.linuxfound.org/sites/events/files/slides/CloudOpen2013_Khoa_Huynh_v3.pdf</a></div><div class="footnote" id="_footnote_35"><a href="#_footnoteref_35" title="Return to text">35</a>. TRIM, UNMAP, and discard
<a href="https://en.wikipedia.org/wiki/Trim_%28computing%29">https://en.wikipedia.org/wiki/Trim_%28computing%29</a></div><div class="footnote" id="_footnote_36"><a href="#_footnoteref_36" title="Return to text">36</a>. Meltdown Attack <a href="https://meltdownattack.com/">https://meltdownattack.com/</a></div><div class="footnote" id="_footnote_37"><a href="#_footnoteref_37" title="Return to text">37</a>. spectre-meltdown-checker <a href="https://meltdown.ovh/">https://meltdown.ovh/</a></div><div class="footnote" id="_footnote_38"><a href="#_footnoteref_38" title="Return to text">38</a>. PCID is now a critical performance/security feature on x86
<a href="https://groups.google.com/forum/m/#!topic/mechanical-sympathy/L9mHTbeQLNU">https://groups.google.com/forum/m/#!topic/mechanical-sympathy/L9mHTbeQLNU</a></div><div class="footnote" id="_footnote_39"><a href="#_footnoteref_39" title="Return to text">39</a>. <a href="https://en.wikipedia.org/wiki/Non-uniform_memory_access">https://en.wikipedia.org/wiki/Non-uniform_memory_access</a></div><div class="footnote" id="_footnote_40"><a href="#_footnoteref_40" title="Return to text">40</a>. if the command
<span class="monospaced">numactl --hardware | grep available</span> returns more than one node, then your host
system has a NUMA architecture</div><div class="footnote" id="_footnote_41"><a href="#_footnoteref_41" title="Return to text">41</a>. A good explanation of the inner workings of the balloon driver can be found here <a href="https://rwmj.wordpress.com/2010/07/17/virtio-balloon/">https://rwmj.wordpress.com/2010/07/17/virtio-balloon/</a></div><div class="footnote" id="_footnote_42"><a href="#_footnoteref_42" title="Return to text">42</a>. <a href="https://www.kraxel.org/blog/2014/10/qemu-using-cirrus-considered-harmful/">https://www.kraxel.org/blog/2014/10/qemu-using-cirrus-considered-harmful/</a>
qemu: using cirrus considered harmful</div><div class="footnote" id="_footnote_43"><a href="#_footnoteref_43" title="Return to text">43</a>. See the OVMF Project <a href="https://github.com/tianocore/tianocore.github.io/wiki/OVMF">https://github.com/tianocore/tianocore.github.io/wiki/OVMF</a></div><div class="footnote" id="_footnote_44"><a href="#_footnoteref_44" title="Return to text">44</a>. Alex
Williamson has a good blog entry about this
<a href="https://vfio.blogspot.co.at/2014/08/primary-graphics-assignment-without-vga.html">https://vfio.blogspot.co.at/2014/08/primary-graphics-assignment-without-vga.html</a></div><div class="footnote" id="_footnote_45"><a href="#_footnoteref_45" title="Return to text">45</a>. Looking Glass: <a href="https://looking-glass.io/">https://looking-glass.io/</a></div><div class="footnote" id="_footnote_46"><a href="#_footnoteref_46" title="Return to text">46</a>. Official
<em>vmgenid</em> Specification
<a href="https://docs.microsoft.com/en-us/windows/desktop/hyperv_v2/virtual-machine-generation-identifier">https://docs.microsoft.com/en-us/windows/desktop/hyperv_v2/virtual-machine-generation-identifier</a></div><div class="footnote" id="_footnote_47"><a href="#_footnoteref_47" title="Return to text">47</a>. Online GUID generator <a href="http://guid.one/">http://guid.one/</a></div><div class="footnote" id="_footnote_48"><a href="#_footnoteref_48" title="Return to text">48</a>. <a href="https://docs.microsoft.com/en-us/windows-server/identity/ad-ds/get-started/virtual-dc/virtualized-domain-controller-architecture">https://docs.microsoft.com/en-us/windows-server/identity/ad-ds/get-started/virtual-dc/virtualized-domain-controller-architecture</a></div><div class="footnote" id="_footnote_49"><a href="#_footnoteref_49" title="Return to text">49</a>. this includes all newest major versions of container
templates shipped by Proxmox VE</div><div class="footnote" id="_footnote_50"><a href="#_footnoteref_50" title="Return to text">50</a>. for example Alpine Linux</div><div class="footnote" id="_footnote_51"><a href="#_footnoteref_51" title="Return to text">51</a>. /etc/os-release replaces the multitude of per-distribution
release files <a href="https://manpages.debian.org/stable/systemd/os-release.5.en.html">https://manpages.debian.org/stable/systemd/os-release.5.en.html</a></div><div class="footnote" id="_footnote_52"><a href="#_footnoteref_52" title="Return to text">52</a>. AppId <a href="https://developers.yubico.com/U2F/App_ID.html">https://developers.yubico.com/U2F/App_ID.html</a></div><div class="footnote" id="_footnote_53"><a href="#_footnoteref_53" title="Return to text">53</a>. Multi-facet apps: <a href="https://developers.yubico.com/U2F/App_ID.html">https://developers.yubico.com/U2F/App_ID.html</a></div><div class="footnote" id="_footnote_54"><a href="#_footnoteref_54" title="Return to text">54</a>. Lempel–Ziv–Oberhumer a lossless data compression algorithm
<a href="https://en.wikipedia.org/wiki/Lempel-Ziv-Oberhumer">https://en.wikipedia.org/wiki/Lempel-Ziv-Oberhumer</a></div><div class="footnote" id="_footnote_55"><a href="#_footnoteref_55" title="Return to text">55</a>. gzip -
based on the DEFLATE algorithm <a href="https://en.wikipedia.org/wiki/Gzip">https://en.wikipedia.org/wiki/Gzip</a></div><div class="footnote" id="_footnote_56"><a href="#_footnoteref_56" title="Return to text">56</a>. Zstandard a lossless data compression algorithm
<a href="https://en.wikipedia.org/wiki/Zstandard">https://en.wikipedia.org/wiki/Zstandard</a></div><div class="footnote" id="_footnote_57"><a href="#_footnoteref_57" title="Return to text">57</a>. pigz - parallel implementation of gzip
<a href="https://zlib.net/pigz/">https://zlib.net/pigz/</a></div><div class="footnote" id="_footnote_58"><a href="#_footnoteref_58" title="Return to text">58</a>. see <span class="monospaced">man 7 systemd.time</span> for more information</div></div>
<div id="footer">
<div id="footer-text">
Version 8.2.3<br>
Last updated
 Wed Jul 31 16:58:41 CEST 2024
</div>
</div>


</body></html>