1
0

gitfs.rst 39 KB

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374757677787980818283848586878889909192939495969798991001011021031041051061071081091101111121131141151161171181191201211221231241251261271281291301311321331341351361371381391401411421431441451461471481491501511521531541551561571581591601611621631641651661671681691701711721731741751761771781791801811821831841851861871881891901911921931941951961971981992002012022032042052062072082092102112122132142152162172182192202212222232242252262272282292302312322332342352362372382392402412422432442452462472482492502512522532542552562572582592602612622632642652662672682692702712722732742752762772782792802812822832842852862872882892902912922932942952962972982993003013023033043053063073083093103113123133143153163173183193203213223233243253263273283293303313323333343353363373383393403413423433443453463473483493503513523533543553563573583593603613623633643653663673683693703713723733743753763773783793803813823833843853863873883893903913923933943953963973983994004014024034044054064074084094104114124134144154164174184194204214224234244254264274284294304314324334344354364374384394404414424434444454464474484494504514524534544554564574584594604614624634644654664674684694704714724734744754764774784794804814824834844854864874884894904914924934944954964974984995005015025035045055065075085095105115125135145155165175185195205215225235245255265275285295305315325335345355365375385395405415425435445455465475485495505515525535545555565575585595605615625635645655665675685695705715725735745755765775785795805815825835845855865875885895905915925935945955965975985996006016026036046056066076086096106116126136146156166176186196206216226236246256266276286296306316326336346356366376386396406416426436446456466476486496506516526536546556566576586596606616626636646656666676686696706716726736746756766776786796806816826836846856866876886896906916926936946956966976986997007017027037047057067077087097107117127137147157167177187197207217227237247257267277287297307317327337347357367377387397407417427437447457467477487497507517527537547557567577587597607617627637647657667677687697707717727737747757767777787797807817827837847857867877887897907917927937947957967977987998008018028038048058068078088098108118128138148158168178188198208218228238248258268278288298308318328338348358368378388398408418428438448458468478488498508518528538548558568578588598608618628638648658668678688698708718728738748758768778788798808818828838848858868878888898908918928938948958968978988999009019029039049059069079089099109119129139149159169179189199209219229239249259269279289299309319329339349359369379389399409419429439449459469479489499509519529539549559569579589599609619629639649659669679689699709719729739749759769779789799809819829839849859869879889899909919929939949959969979989991000100110021003100410051006100710081009101010111012101310141015101610171018101910201021102210231024102510261027102810291030103110321033103410351036103710381039104010411042104310441045104610471048104910501051105210531054105510561057105810591060106110621063106410651066106710681069107010711072107310741075107610771078107910801081108210831084108510861087108810891090109110921093109410951096109710981099110011011102110311041105110611071108110911101111111211131114111511161117111811191120112111221123112411251126112711281129113011311132113311341135113611371138113911401141114211431144114511461147114811491150115111521153115411551156115711581159116011611162116311641165116611671168
  1. .. _tutorial-gitfs:
  2. ==================================
  3. Git Fileserver Backend Walkthrough
  4. ==================================
  5. .. note::
  6. This walkthrough assumes basic knowledge of Salt. To get up to speed, check
  7. out the :ref:`Salt Walkthrough <tutorial-salt-walk-through>`.
  8. The gitfs backend allows Salt to serve files from git repositories. It can be
  9. enabled by adding ``git`` to the :conf_master:`fileserver_backend` list, and
  10. configuring one or more repositories in :conf_master:`gitfs_remotes`.
  11. Branches and tags become Salt fileserver environments.
  12. .. note::
  13. Branching and tagging can result in a lot of potentially-conflicting
  14. :ref:`top files <states-top>`, for this reason it may be useful to set
  15. :conf_minion:`top_file_merging_strategy` to ``same`` in the minions' config
  16. files if the top files are being managed in a GitFS repo.
  17. .. _gitfs-dependencies:
  18. Installing Dependencies
  19. =======================
  20. Both pygit2_ and GitPython_ are supported Python interfaces to git. If
  21. compatible versions of both are installed, pygit2_ will be preferred. In these
  22. cases, GitPython_ can be forced using the :conf_master:`gitfs_provider`
  23. parameter in the master config file.
  24. .. note::
  25. It is recommended to always run the most recent version of any the below
  26. dependencies. Certain features of GitFS may not be available without
  27. the most recent version of the chosen library.
  28. .. _pygit2: https://github.com/libgit2/pygit2
  29. .. _GitPython: https://github.com/gitpython-developers/GitPython
  30. pygit2
  31. ------
  32. The minimum supported version of pygit2_ is 0.20.3. Availability for this
  33. version of pygit2_ is still limited, though the SaltStack team is working to
  34. get compatible versions available for as many platforms as possible.
  35. For the Fedora/EPEL versions which have a new enough version packaged, the
  36. following command would be used to install pygit2_:
  37. .. code-block:: bash
  38. # yum install python-pygit2
  39. Provided a valid version is packaged for Debian/Ubuntu (which is not currently
  40. the case), the package name would be the same, and the following command would
  41. be used to install it:
  42. .. code-block:: bash
  43. # apt-get install python-pygit2
  44. If pygit2_ is not packaged for the platform on which the Master is running, the
  45. pygit2_ website has installation instructions
  46. `here <pygit2-install-instructions>`_. Keep in mind however that
  47. following these instructions will install libgit2_ and pygit2_ without system
  48. packages. Additionally, keep in mind that :ref:`SSH authentication in pygit2
  49. <pygit2-authentication-ssh>` requires libssh2_ (*not* libssh) development
  50. libraries to be present before libgit2_ is built. On some Debian-based distros
  51. ``pkg-config`` is also required to link libgit2_ with libssh2.
  52. .. note::
  53. If you are receiving the error "Unsupported URL Protocol" in the Salt Master
  54. log when making a connection using SSH, review the libssh2 details listed
  55. above.
  56. Additionally, version 0.21.0 of pygit2 introduced a dependency on python-cffi_,
  57. which in turn depends on newer releases of libffi_. Upgrading libffi_ is not
  58. advisable as several other applications depend on it, so on older LTS linux
  59. releases pygit2_ 0.20.3 and libgit2_ 0.20.0 is the recommended combination.
  60. .. warning::
  61. pygit2_ is actively developed and `frequently makes
  62. non-backwards-compatible API changes <pygit2-version-policy>`_, even in
  63. minor releases. It is not uncommon for pygit2_ upgrades to result in errors
  64. in Salt. Please take care when upgrading pygit2_, and pay close attention
  65. to the changelog_, keeping an eye out for API changes. Errors can be
  66. reported on the `SaltStack issue tracker <saltstack-issue-tracker>`_.
  67. .. _pygit2-version-policy: http://www.pygit2.org/install.html#version-numbers
  68. .. _changelog: https://github.com/libgit2/pygit2#changelog
  69. .. _saltstack-issue-tracker: https://github.com/saltstack/salt/issues
  70. .. _pygit2-install-instructions: http://www.pygit2.org/install.html
  71. .. _libgit2: https://libgit2.github.com/
  72. .. _libssh2: http://www.libssh2.org/
  73. .. _python-cffi: https://pypi.python.org/pypi/cffi
  74. .. _libffi: http://sourceware.org/libffi/
  75. RedHat Pygit2 Issues
  76. ~~~~~~~~~~~~~~~~~~~~
  77. The release of RedHat/CentOS 7.3 upgraded both ``python-cffi`` and
  78. ``http-parser``, both of which are dependencies for pygit2_/libgit2_. Both
  79. ``pygit2`` and ``libgit2`` packages (which are from the EPEL repository) should
  80. be upgraded to the most recent versions, at least to ``0.24.2``.
  81. The below errors will show up in the master log if an incompatible
  82. ``python-pygit2`` package is installed:
  83. .. code-block:: text
  84. 2017-02-10 09:07:34,892 [salt.utils.gitfs ][ERROR ][11211] Import pygit2 failed: CompileError: command 'gcc' failed with exit status 1
  85. 2017-02-10 09:07:34,907 [salt.utils.gitfs ][ERROR ][11211] gitfs is configured but could not be loaded, are pygit2 and libgit2 installed?
  86. 2017-02-10 09:07:34,907 [salt.utils.gitfs ][CRITICAL][11211] No suitable gitfs provider module is installed.
  87. 2017-02-10 09:07:34,912 [salt.master ][CRITICAL][11211] Master failed pre flight checks, exiting
  88. The below errors will show up in the master log if an incompatible ``libgit2``
  89. package is installed:
  90. .. code-block:: text
  91. 2017-02-15 18:04:45,211 [salt.utils.gitfs ][ERROR ][6211] Error occurred fetching gitfs remote 'https://foo.com/bar.git': No Content-Type header in response
  92. A restart of the ``salt-master`` daemon and gitfs cache directory clean up may
  93. be required to allow http(s) repositories to continue to be fetched.
  94. GitPython
  95. ---------
  96. GitPython_ 0.3.0 or newer is required to use GitPython for gitfs. For
  97. RHEL-based Linux distros, a compatible version is available in EPEL, and can be
  98. easily installed on the master using yum:
  99. .. code-block:: bash
  100. # yum install GitPython
  101. Ubuntu 14.04 LTS and Debian Wheezy (7.x) also have a compatible version packaged:
  102. .. code-block:: bash
  103. # apt-get install python-git
  104. GitPython_ requires the ``git`` CLI utility to work. If installed from a system
  105. package, then git should already be installed, but if installed via pip_ then
  106. it may still be necessary to install git separately. For MacOS users,
  107. GitPython_ comes bundled in with the Salt installer, but git must still be
  108. installed for it to work properly. Git can be installed in several ways,
  109. including by installing XCode_.
  110. .. _pip: http://www.pip-installer.org/
  111. .. _XCode: https://developer.apple.com/xcode/
  112. .. warning::
  113. Keep in mind that if GitPython has been previously installed on the master
  114. using pip (even if it was subsequently uninstalled), then it may still
  115. exist in the build cache (typically ``/tmp/pip-build-root/GitPython``) if
  116. the cache is not cleared after installation. The package in the build cache
  117. will override any requirement specifiers, so if you try upgrading to
  118. version 0.3.2.RC1 by running ``pip install 'GitPython==0.3.2.RC1'`` then it
  119. will ignore this and simply install the version from the cache directory.
  120. Therefore, it may be necessary to delete the GitPython directory from the
  121. build cache in order to ensure that the specified version is installed.
  122. .. warning::
  123. GitPython_ 2.0.9 and newer is not compatible with Python 2.6. If installing
  124. GitPython_ using pip on a machine running Python 2.6, make sure that a
  125. version earlier than 2.0.9 is installed. This can be done on the CLI by
  126. running ``pip install 'GitPython<2.0.9'``, or in a :py:func:`pip.installed
  127. <salt.states.pip_state.installed>` state using the following SLS:
  128. .. code-block:: yaml
  129. GitPython:
  130. pip.installed:
  131. - name: 'GitPython < 2.0.9'
  132. Simple Configuration
  133. ====================
  134. To use the gitfs backend, only two configuration changes are required on the
  135. master:
  136. 1. Include ``gitfs`` in the :conf_master:`fileserver_backend` list in the
  137. master config file:
  138. .. code-block:: yaml
  139. fileserver_backend:
  140. - gitfs
  141. .. note::
  142. ``git`` also works here. Prior to the 2018.3.0 release, *only* ``git``
  143. would work.
  144. 2. Specify one or more ``git://``, ``https://``, ``file://``, or ``ssh://``
  145. URLs in :conf_master:`gitfs_remotes` to configure which repositories to
  146. cache and search for requested files:
  147. .. code-block:: yaml
  148. gitfs_remotes:
  149. - https://github.com/saltstack-formulas/salt-formula.git
  150. SSH remotes can also be configured using scp-like syntax:
  151. .. code-block:: yaml
  152. gitfs_remotes:
  153. - git@github.com:user/repo.git
  154. - ssh://user@domain.tld/path/to/repo.git
  155. Information on how to authenticate to SSH remotes can be found :ref:`here
  156. <gitfs-authentication>`.
  157. 3. Restart the master to load the new configuration.
  158. .. note::
  159. In a master/minion setup, files from a gitfs remote are cached once by the
  160. master, so minions do not need direct access to the git repository.
  161. Multiple Remotes
  162. ================
  163. The ``gitfs_remotes`` option accepts an ordered list of git remotes to
  164. cache and search, in listed order, for requested files.
  165. A simple scenario illustrates this cascading lookup behavior:
  166. If the ``gitfs_remotes`` option specifies three remotes:
  167. .. code-block:: yaml
  168. gitfs_remotes:
  169. - git://github.com/example/first.git
  170. - https://github.com/example/second.git
  171. - file:///root/third
  172. And each repository contains some files:
  173. .. code-block:: yaml
  174. first.git:
  175. top.sls
  176. edit/vim.sls
  177. edit/vimrc
  178. nginx/init.sls
  179. second.git:
  180. edit/dev_vimrc
  181. haproxy/init.sls
  182. third:
  183. haproxy/haproxy.conf
  184. edit/dev_vimrc
  185. Salt will attempt to lookup the requested file from each gitfs remote
  186. repository in the order in which they are defined in the configuration. The
  187. :strong:`git://github.com/example/first.git` remote will be searched first.
  188. If the requested file is found, then it is served and no further searching
  189. is executed. For example:
  190. * A request for the file :strong:`salt://haproxy/init.sls` will be served from
  191. the :strong:`https://github.com/example/second.git` git repo.
  192. * A request for the file :strong:`salt://haproxy/haproxy.conf` will be served from the
  193. :strong:`file:///root/third` repo.
  194. .. note::
  195. This example is purposefully contrived to illustrate the behavior of the
  196. gitfs backend. This example should not be read as a recommended way to lay
  197. out files and git repos.
  198. The :strong:`file://` prefix denotes a git repository in a local directory.
  199. However, it will still use the given :strong:`file://` URL as a remote,
  200. rather than copying the git repo to the salt cache. This means that any
  201. refs you want accessible must exist as *local* refs in the specified repo.
  202. .. warning::
  203. Salt versions prior to 2014.1.0 are not tolerant of changing the
  204. order of remotes or modifying the URI of existing remotes. In those
  205. versions, when modifying remotes it is a good idea to remove the gitfs
  206. cache directory (``/var/cache/salt/master/gitfs``) before restarting the
  207. salt-master service.
  208. .. _gitfs-per-remote-config:
  209. Per-remote Configuration Parameters
  210. ===================================
  211. .. versionadded:: 2014.7.0
  212. The following master config parameters are global (that is, they apply to all
  213. configured gitfs remotes):
  214. * :conf_master:`gitfs_base`
  215. * :conf_master:`gitfs_root`
  216. * :conf_master:`gitfs_ssl_verify`
  217. * :conf_master:`gitfs_mountpoint` (new in 2014.7.0)
  218. * :conf_master:`gitfs_user` (**pygit2 only**, new in 2014.7.0)
  219. * :conf_master:`gitfs_password` (**pygit2 only**, new in 2014.7.0)
  220. * :conf_master:`gitfs_insecure_auth` (**pygit2 only**, new in 2014.7.0)
  221. * :conf_master:`gitfs_pubkey` (**pygit2 only**, new in 2014.7.0)
  222. * :conf_master:`gitfs_privkey` (**pygit2 only**, new in 2014.7.0)
  223. * :conf_master:`gitfs_passphrase` (**pygit2 only**, new in 2014.7.0)
  224. * :conf_master:`gitfs_refspecs` (new in 2017.7.0)
  225. * :conf_master:`gitfs_disable_saltenv_mapping` (new in 2018.3.0)
  226. * :conf_master:`gitfs_ref_types` (new in 2018.3.0)
  227. * :conf_master:`gitfs_update_interval` (new in 2018.3.0)
  228. .. note::
  229. pygit2 only supports disabling SSL verification in versions 0.23.2 and
  230. newer.
  231. These parameters can now be overridden on a per-remote basis. This allows for a
  232. tremendous amount of customization. Here's some example usage:
  233. .. code-block:: yaml
  234. gitfs_provider: pygit2
  235. gitfs_base: develop
  236. gitfs_remotes:
  237. - https://foo.com/foo.git
  238. - https://foo.com/bar.git:
  239. - root: salt
  240. - mountpoint: salt://bar
  241. - base: salt-base
  242. - ssl_verify: False
  243. - update_interval: 120
  244. - https://foo.com/bar.git:
  245. - name: second_bar_repo
  246. - root: other/salt
  247. - mountpoint: salt://other/bar
  248. - base: salt-base
  249. - ref_types:
  250. - branch
  251. - http://foo.com/baz.git:
  252. - root: salt/states
  253. - user: joe
  254. - password: mysupersecretpassword
  255. - insecure_auth: True
  256. - disable_saltenv_mapping: True
  257. - saltenv:
  258. - foo:
  259. - ref: foo
  260. - http://foo.com/quux.git:
  261. - all_saltenvs: master
  262. .. important::
  263. There are two important distinctions which should be noted for per-remote
  264. configuration:
  265. 1. The URL of a remote which has per-remote configuration must be suffixed
  266. with a colon.
  267. 2. Per-remote configuration parameters are named like the global versions,
  268. with the ``gitfs_`` removed from the beginning. The exception being the
  269. ``name``, ``saltenv``, and ``all_saltenvs`` parameters, which are only
  270. available to per-remote configurations.
  271. The ``all_saltenvs`` parameter is new in the 2018.3.0 release.
  272. In the example configuration above, the following is true:
  273. 1. The first and fourth gitfs remotes will use the ``develop`` branch/tag as the
  274. ``base`` environment, while the second and third will use the ``salt-base``
  275. branch/tag as the ``base`` environment.
  276. 2. The first remote will serve all files in the repository. The second
  277. remote will only serve files from the ``salt`` directory (and its
  278. subdirectories). The third remote will only server files from the
  279. ``other/salt`` directory (and its subdirectories), while the fourth remote
  280. will only serve files from the ``salt/states`` directory (and its
  281. subdirectories).
  282. 3. The third remote will only serve files from branches, and not from tags or
  283. SHAs.
  284. 4. The fourth remote will only have two saltenvs available: ``base`` (pointed
  285. at ``develop``), and ``foo`` (pointed at ``foo``).
  286. 5. The first and fourth remotes will have files located under the root of the
  287. Salt fileserver namespace (``salt://``). The files from the second remote
  288. will be located under ``salt://bar``, while the files from the third remote
  289. will be located under ``salt://other/bar``.
  290. 6. The second and third remotes reference the same repository and unique names
  291. need to be declared for duplicate gitfs remotes.
  292. 7. The fourth remote overrides the default behavior of :ref:`not authenticating
  293. to insecure (non-HTTPS) remotes <gitfs-insecure-auth>`.
  294. 8. Because ``all_saltenvs`` is configured for the fifth remote, files from the
  295. branch/tag ``master`` will appear in every fileserver environment.
  296. .. note::
  297. The use of ``http://`` (instead of ``https://``) is permitted here
  298. *only* because authentication is not being used. Otherwise, the
  299. ``insecure_auth`` parameter must be used (as in the fourth remote) to
  300. force Salt to authenticate to an ``http://`` remote.
  301. 9. The first remote will wait 120 seconds between updates instead of 60.
  302. .. _gitfs-per-saltenv-config:
  303. Per-Saltenv Configuration Parameters
  304. ====================================
  305. .. versionadded:: 2016.11.0
  306. For more granular control, Salt allows the following three things to be
  307. overridden for individual saltenvs within a given repo:
  308. - The :ref:`mountpoint <gitfs-walkthrough-mountpoint>`
  309. - The :ref:`root <gitfs-walkthrough-root>`
  310. - The branch/tag to be used for a given saltenv
  311. Here is an example:
  312. .. code-block:: yaml
  313. gitfs_root: salt
  314. gitfs_saltenv:
  315. - dev:
  316. - mountpoint: salt://gitfs-dev
  317. - ref: develop
  318. gitfs_remotes:
  319. - https://foo.com/bar.git:
  320. - saltenv:
  321. - staging:
  322. - ref: qa
  323. - mountpoint: salt://bar-staging
  324. - dev:
  325. - ref: development
  326. - https://foo.com/baz.git:
  327. - saltenv:
  328. - staging:
  329. - mountpoint: salt://baz-staging
  330. Given the above configuration, the following is true:
  331. 1. For all gitfs remotes, files for the ``dev`` saltenv will be located under
  332. ``salt://gitfs-dev``.
  333. 2. For the ``dev`` saltenv, files from the first remote will be sourced from
  334. the ``development`` branch, while files from the second remote will be
  335. sourced from the ``develop`` branch.
  336. 3. For the ``staging`` saltenv, files from the first remote will be located
  337. under ``salt://bar-staging``, while files from the second remote will be
  338. located under ``salt://baz-staging``.
  339. 4. For all gitfs remotes, and in all saltenvs, files will be served from the
  340. ``salt`` directory (and its subdirectories).
  341. .. _gitfs-custom-refspecs:
  342. Custom Refspecs
  343. ===============
  344. .. versionadded:: 2017.7.0
  345. GitFS will by default fetch remote branches and tags. However, sometimes it can
  346. be useful to fetch custom refs (such as those created for `GitHub pull
  347. requests`__). To change the refspecs GitFS fetches, use the
  348. :conf_master:`gitfs_refspecs` config option:
  349. .. __: https://help.github.com/articles/checking-out-pull-requests-locally/
  350. .. code-block:: yaml
  351. gitfs_refspecs:
  352. - '+refs/heads/*:refs/remotes/origin/*'
  353. - '+refs/tags/*:refs/tags/*'
  354. - '+refs/pull/*/head:refs/remotes/origin/pr/*'
  355. - '+refs/pull/*/merge:refs/remotes/origin/merge/*'
  356. In the above example, in addition to fetching remote branches and tags,
  357. GitHub's custom refs for pull requests and merged pull requests will also be
  358. fetched. These special ``head`` refs represent the head of the branch which is
  359. requesting to be merged, and the ``merge`` refs represent the result of the
  360. base branch after the merge.
  361. .. important::
  362. When using custom refspecs, the destination of the fetched refs *must* be
  363. under ``refs/remotes/origin/``, preferably in a subdirectory like in the
  364. example above. These custom refspecs will map as environment names using
  365. their relative path underneath ``refs/remotes/origin/``. For example,
  366. assuming the configuration above, the head branch for pull request 12345
  367. would map to fileserver environment ``pr/12345`` (slash included).
  368. Refspecs can be configured on a :ref:`per-remote basis
  369. <gitfs-per-remote-config>`. For example, the below configuration would only
  370. alter the default refspecs for the *second* GitFS remote. The first remote
  371. would only fetch branches and tags (the default).
  372. .. code-block:: yaml
  373. gitfs_remotes:
  374. - https://domain.tld/foo.git
  375. - https://domain.tld/bar.git:
  376. - refspecs:
  377. - '+refs/heads/*:refs/remotes/origin/*'
  378. - '+refs/tags/*:refs/tags/*'
  379. - '+refs/pull/*/head:refs/remotes/origin/pr/*'
  380. - '+refs/pull/*/merge:refs/remotes/origin/merge/*'
  381. .. _gitfs-global-remotes:
  382. Global Remotes
  383. ==============
  384. .. versionadded:: 2018.3.0
  385. The ``all_saltenvs`` per-remote configuration parameter overrides the logic
  386. Salt uses to map branches/tags to fileserver environments (i.e. saltenvs). This
  387. allows a single branch/tag to appear in *all* GitFS saltenvs.
  388. .. note::
  389. ``all_saltenvs`` only works *within* GitFS. That is, files in a branch
  390. configured using ``all_saltenvs`` will *not* show up in a fileserver
  391. environment defined via some other fileserver backend (e.g.
  392. :conf_master:`file_roots`).
  393. This is very useful in particular when working with :ref:`salt formulas
  394. <conventions-formula>`. Prior to the addition of this feature, it was necessary
  395. to push a branch/tag to the remote repo for each saltenv in which that formula
  396. was to be used. If the formula needed to be updated, this update would need to
  397. be reflected in all of the other branches/tags. This is both inconvenient and
  398. not scalable.
  399. With ``all_saltenvs``, it is now possible to define your formula once, in a
  400. single branch.
  401. .. code-block:: yaml
  402. gitfs_remotes:
  403. - http://foo.com/quux.git:
  404. - all_saltenvs: anything
  405. .. _gitfs-update-intervals:
  406. Update Intervals
  407. ================
  408. Prior to the 2018.3.0 release, GitFS would update its fileserver backends as part
  409. of a dedicated "maintenance" process, in which various routine maintenance
  410. tasks were performed. This tied the update interval to the
  411. :conf_master:`loop_interval` config option, and also forced all fileservers to
  412. update at the same interval.
  413. Now it is possible to make GitFS update at its own interval, using
  414. :conf_master:`gitfs_update_interval`:
  415. .. code-block:: yaml
  416. gitfs_update_interval: 180
  417. gitfs_remotes:
  418. - https://foo.com/foo.git
  419. - https://foo.com/bar.git:
  420. - update_interval: 120
  421. Using the above configuration, the first remote would update every three
  422. minutes, while the second remote would update every two minutes.
  423. Configuration Order of Precedence
  424. =================================
  425. The order of precedence for GitFS configuration is as follows (each level
  426. overrides all levels below it):
  427. 1. Per-saltenv configuration (defined under a per-remote ``saltenv``
  428. param)
  429. .. code-block:: yaml
  430. gitfs_remotes:
  431. - https://foo.com/bar.git:
  432. - saltenv:
  433. - dev:
  434. - mountpoint: salt://bar
  435. 2. Global per-saltenv configuration (defined in :conf_master:`gitfs_saltenv`)
  436. .. code-block:: yaml
  437. gitfs_saltenv:
  438. - dev:
  439. - mountpoint: salt://bar
  440. 3. Per-remote configuration parameter
  441. .. code-block:: yaml
  442. gitfs_remotes:
  443. - https://foo.com/bar.git:
  444. - mountpoint: salt://bar
  445. 4. Global configuration parameter
  446. .. code-block:: yaml
  447. gitfs_mountpoint: salt://bar
  448. .. note::
  449. The one exception to the above is when :ref:`all_saltenvs
  450. <gitfs-global-remotes>` is used. This value overrides all logic for mapping
  451. branches/tags to fileserver environments. So, even if
  452. :conf_master:`gitfs_saltenv` is used to globally override the mapping for a
  453. given saltenv, :ref:`all_saltenvs <gitfs-global-remotes>` would take
  454. precedence for any remote which uses it.
  455. It's important to note however that any ``root`` and ``mountpoint`` values
  456. configured in :conf_master:`gitfs_saltenv` (or :ref:`per-saltenv
  457. configuration <gitfs-per-saltenv-config>`) would be unaffected by this.
  458. .. _gitfs-walkthrough-root:
  459. Serving from a Subdirectory
  460. ===========================
  461. The :conf_master:`gitfs_root` parameter allows files to be served from a
  462. subdirectory within the repository. This allows for only part of a repository
  463. to be exposed to the Salt fileserver.
  464. Assume the below layout:
  465. .. code-block:: text
  466. .gitignore
  467. README.txt
  468. foo/
  469. foo/bar/
  470. foo/bar/one.txt
  471. foo/bar/two.txt
  472. foo/bar/three.txt
  473. foo/baz/
  474. foo/baz/top.sls
  475. foo/baz/edit/vim.sls
  476. foo/baz/edit/vimrc
  477. foo/baz/nginx/init.sls
  478. The below configuration would serve only the files under ``foo/baz``, ignoring
  479. the other files in the repository:
  480. .. code-block:: yaml
  481. gitfs_remotes:
  482. - git://mydomain.com/stuff.git
  483. gitfs_root: foo/baz
  484. The root can also be configured on a :ref:`per-remote basis
  485. <gitfs-per-remote-config>`.
  486. .. _gitfs-walkthrough-mountpoint:
  487. Mountpoints
  488. ===========
  489. .. versionadded:: 2014.7.0
  490. The :conf_master:`gitfs_mountpoint` parameter will prepend the specified path
  491. to the files served from gitfs. This allows an existing repository to be used,
  492. rather than needing to reorganize a repository or design it around the layout
  493. of the Salt fileserver.
  494. Before the addition of this feature, if a file being served up via gitfs was
  495. deeply nested within the root directory (for example,
  496. ``salt://webapps/foo/files/foo.conf``, it would be necessary to ensure that the
  497. file was properly located in the remote repository, and that all of the
  498. parent directories were present (for example, the directories
  499. ``webapps/foo/files/`` would need to exist at the root of the repository).
  500. The below example would allow for a file ``foo.conf`` at the root of the
  501. repository to be served up from the Salt fileserver path
  502. ``salt://webapps/foo/files/foo.conf``.
  503. .. code-block:: yaml
  504. gitfs_remotes:
  505. - https://mydomain.com/stuff.git
  506. gitfs_mountpoint: salt://webapps/foo/files
  507. Mountpoints can also be configured on a :ref:`per-remote basis
  508. <gitfs-per-remote-config>`.
  509. Using gitfs in Masterless Mode
  510. ==============================
  511. Since 2014.7.0, gitfs can be used in masterless mode. To do so, simply add the
  512. gitfs configuration parameters (and set :conf_master:`fileserver_backend`) in
  513. the _minion_ config file instead of the master config file.
  514. Using gitfs Alongside Other Backends
  515. ====================================
  516. Sometimes it may make sense to use multiple backends; for instance, if ``sls``
  517. files are stored in git but larger files are stored directly on the master.
  518. The cascading lookup logic used for multiple remotes is also used with multiple
  519. backends. If the :conf_master:`fileserver_backend` option contains multiple
  520. backends:
  521. .. code-block:: yaml
  522. fileserver_backend:
  523. - roots
  524. - git
  525. Then the ``roots`` backend (the default backend of files in ``/srv/salt``) will
  526. be searched first for the requested file; then, if it is not found on the
  527. master, each configured git remote will be searched.
  528. .. note::
  529. This can be used together with `file_roots` accepting `__env__` as a catch-all
  530. environment, since 2018.3.5 and 2019.2.1:
  531. .. code-block:: yaml
  532. file_roots:
  533. base:
  534. - /srv/salt
  535. __env__:
  536. - /srv/salt
  537. Branches, Environments, and Top Files
  538. =====================================
  539. When using the GitFS backend, branches, and tags will be mapped to environments
  540. using the branch/tag name as an identifier.
  541. There is one exception to this rule: the ``master`` branch is implicitly mapped
  542. to the ``base`` environment.
  543. So, for a typical ``base``, ``qa``, ``dev`` setup, the following branches could
  544. be used:
  545. .. code-block:: yaml
  546. master
  547. qa
  548. dev
  549. ``top.sls`` files from different branches will be merged into one at runtime.
  550. Since this can lead to overly complex configurations, the recommended setup is
  551. to have a separate repository, containing only the ``top.sls`` file with just
  552. one single ``master`` branch.
  553. To map a branch other than ``master`` as the ``base`` environment, use the
  554. :conf_master:`gitfs_base` parameter.
  555. .. code-block:: yaml
  556. gitfs_base: salt-base
  557. The base can also be configured on a :ref:`per-remote basis
  558. <gitfs-per-remote-config>`.
  559. .. _gitfs-whitelist-blacklist:
  560. Environment Whitelist/Blacklist
  561. ===============================
  562. .. versionadded:: 2014.7.0
  563. The :conf_master:`gitfs_saltenv_whitelist` and
  564. :conf_master:`gitfs_saltenv_blacklist` parameters allow for greater control
  565. over which branches/tags are exposed as fileserver environments. Exact matches,
  566. globs, and regular expressions are supported, and are evaluated in that order.
  567. If using a regular expression, ``^`` and ``$`` must be omitted, and the
  568. expression must match the entire branch/tag.
  569. .. code-block:: yaml
  570. gitfs_saltenv_whitelist:
  571. - base
  572. - v1.*
  573. - 'mybranch\d+'
  574. .. note::
  575. ``v1.*``, in this example, will match as both a glob and a regular
  576. expression (though it will have been matched as a glob, since globs are
  577. evaluated before regular expressions).
  578. The behavior of the blacklist/whitelist will differ depending on which
  579. combination of the two options is used:
  580. * If only :conf_master:`gitfs_saltenv_whitelist` is used, then **only**
  581. branches/tags which match the whitelist will be available as environments
  582. * If only :conf_master:`gitfs_saltenv_blacklist` is used, then the
  583. branches/tags which match the blacklist will **not** be available as
  584. environments
  585. * If both are used, then the branches/tags which match the whitelist, but do
  586. **not** match the blacklist, will be available as environments.
  587. .. _gitfs-authentication:
  588. Authentication
  589. ==============
  590. pygit2
  591. ------
  592. .. versionadded:: 2014.7.0
  593. Both HTTPS and SSH authentication are supported as of version 0.20.3, which is
  594. the earliest version of pygit2_ supported by Salt for gitfs.
  595. .. note::
  596. The examples below make use of per-remote configuration parameters, a
  597. feature new to Salt 2014.7.0. More information on these can be found
  598. :ref:`here <gitfs-per-remote-config>`.
  599. HTTPS
  600. ~~~~~
  601. For HTTPS repositories which require authentication, the username and password
  602. can be provided like so:
  603. .. code-block:: yaml
  604. gitfs_remotes:
  605. - https://domain.tld/myrepo.git:
  606. - user: git
  607. - password: mypassword
  608. .. _gitfs-insecure-auth:
  609. If the repository is served over HTTP instead of HTTPS, then Salt will by
  610. default refuse to authenticate to it. This behavior can be overridden by adding
  611. an ``insecure_auth`` parameter:
  612. .. code-block:: yaml
  613. gitfs_remotes:
  614. - http://domain.tld/insecure_repo.git:
  615. - user: git
  616. - password: mypassword
  617. - insecure_auth: True
  618. .. _pygit2-authentication-ssh:
  619. SSH
  620. ~~~
  621. SSH repositories can be configured using the ``ssh://`` protocol designation,
  622. or using scp-like syntax. So, the following two configurations are equivalent:
  623. * ``ssh://git@github.com/user/repo.git``
  624. * ``git@github.com:user/repo.git``
  625. Both :conf_master:`gitfs_pubkey` and :conf_master:`gitfs_privkey` (or their
  626. :ref:`per-remote counterparts <gitfs-per-remote-config>`) must be configured in
  627. order to authenticate to SSH-based repos. If the private key is protected with
  628. a passphrase, it can be configured using :conf_master:`gitfs_passphrase` (or
  629. simply ``passphrase`` if being configured :ref:`per-remote
  630. <gitfs-per-remote-config>`). For example:
  631. .. code-block:: yaml
  632. gitfs_remotes:
  633. - git@github.com:user/repo.git:
  634. - pubkey: /root/.ssh/id_rsa.pub
  635. - privkey: /root/.ssh/id_rsa
  636. - passphrase: myawesomepassphrase
  637. Finally, the SSH host key must be :ref:`added to the known_hosts file
  638. <gitfs-ssh-fingerprint>`.
  639. .. note::
  640. There is a known issue with public-key SSH authentication to Microsoft
  641. Visual Studio (VSTS) with pygit2. This is due to a bug or lack of support
  642. for VSTS in older libssh2 releases. Known working releases include libssh2
  643. 1.7.0 and later, and known incompatible releases include 1.5.0 and older.
  644. At the time of this writing, 1.6.0 has not been tested.
  645. Since upgrading libssh2 would require rebuilding many other packages (curl,
  646. etc.), followed by a rebuild of libgit2 and a reinstall of pygit2, an
  647. easier workaround for systems with older libssh2 is to use GitPython with a
  648. passphraseless key for authentication.
  649. GitPython
  650. ---------
  651. HTTPS
  652. ~~~~~
  653. For HTTPS repositories which require authentication, the username and password
  654. can be configured in one of two ways. The first way is to include them in the
  655. URL using the format ``https://<user>:<password>@<url>``, like so:
  656. .. code-block:: yaml
  657. gitfs_remotes:
  658. - https://git:mypassword@domain.tld/myrepo.git
  659. The other way would be to configure the authentication in ``~/.netrc``:
  660. .. code-block:: text
  661. machine domain.tld
  662. login git
  663. password mypassword
  664. If the repository is served over HTTP instead of HTTPS, then Salt will by
  665. default refuse to authenticate to it. This behavior can be overridden by adding
  666. an ``insecure_auth`` parameter:
  667. .. code-block:: yaml
  668. gitfs_remotes:
  669. - http://git:mypassword@domain.tld/insecure_repo.git:
  670. - insecure_auth: True
  671. SSH
  672. ~~~
  673. Only passphrase-less SSH public key authentication is supported using
  674. GitPython. **The auth parameters (pubkey, privkey, etc.) shown in the pygit2
  675. authentication examples above do not work with GitPython.**
  676. .. code-block:: yaml
  677. gitfs_remotes:
  678. - ssh://git@github.com/example/salt-states.git
  679. Since GitPython_ wraps the git CLI, the private key must be located in
  680. ``~/.ssh/id_rsa`` for the user under which the Master is running, and should
  681. have permissions of ``0600``. Also, in the absence of a user in the repo URL,
  682. GitPython_ will (just as SSH does) attempt to login as the current user (in
  683. other words, the user under which the Master is running, usually ``root``).
  684. If a key needs to be used, then ``~/.ssh/config`` can be configured to use
  685. the desired key. Information on how to do this can be found by viewing the
  686. manpage for ``ssh_config``. Here's an example entry which can be added to the
  687. ``~/.ssh/config`` to use an alternate key for gitfs:
  688. .. code-block:: text
  689. Host github.com
  690. IdentityFile /root/.ssh/id_rsa_gitfs
  691. The ``Host`` parameter should be a hostname (or hostname glob) that matches the
  692. domain name of the git repository.
  693. It is also necessary to :ref:`add the SSH host key to the known_hosts file
  694. <gitfs-ssh-fingerprint>`. The exception to this would be if strict host key
  695. checking is disabled, which can be done by adding ``StrictHostKeyChecking no``
  696. to the entry in ``~/.ssh/config``
  697. .. code-block:: text
  698. Host github.com
  699. IdentityFile /root/.ssh/id_rsa_gitfs
  700. StrictHostKeyChecking no
  701. However, this is generally regarded as insecure, and is not recommended.
  702. .. _gitfs-ssh-fingerprint:
  703. Adding the SSH Host Key to the known_hosts File
  704. -----------------------------------------------
  705. To use SSH authentication, it is necessary to have the remote repository's SSH
  706. host key in the ``~/.ssh/known_hosts`` file. If the master is also a minion,
  707. this can be done using the :mod:`ssh.set_known_host
  708. <salt.modules.ssh.set_known_host>` function:
  709. .. code-block:: bash
  710. # salt mymaster ssh.set_known_host user=root hostname=github.com
  711. mymaster:
  712. ----------
  713. new:
  714. ----------
  715. enc:
  716. ssh-rsa
  717. fingerprint:
  718. 16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48
  719. hostname:
  720. |1|OiefWWqOD4kwO3BhoIGa0loR5AA=|BIXVtmcTbPER+68HvXmceodDcfI=
  721. key:
  722. AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==
  723. old:
  724. None
  725. status:
  726. updated
  727. If not, then the easiest way to add the key is to su to the user (usually
  728. ``root``) under which the salt-master runs and attempt to login to the
  729. server via SSH:
  730. .. code-block:: text
  731. $ su -
  732. Password:
  733. # ssh github.com
  734. The authenticity of host 'github.com (192.30.252.128)' can't be established.
  735. RSA key fingerprint is 16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48.
  736. Are you sure you want to continue connecting (yes/no)? yes
  737. Warning: Permanently added 'github.com,192.30.252.128' (RSA) to the list of known hosts.
  738. Permission denied (publickey).
  739. It doesn't matter if the login was successful, as answering ``yes`` will write
  740. the fingerprint to the known_hosts file.
  741. Verifying the Fingerprint
  742. ~~~~~~~~~~~~~~~~~~~~~~~~~
  743. To verify that the correct fingerprint was added, it is a good idea to look it
  744. up. One way to do this is to use ``nmap``:
  745. .. code-block:: bash
  746. $ nmap -p 22 github.com --script ssh-hostkey
  747. Starting Nmap 5.51 ( http://nmap.org ) at 2014-08-18 17:47 CDT
  748. Nmap scan report for github.com (192.30.252.129)
  749. Host is up (0.17s latency).
  750. Not shown: 996 filtered ports
  751. PORT STATE SERVICE
  752. 22/tcp open ssh
  753. | ssh-hostkey: 1024 ad:1c:08:a4:40:e3:6f:9c:f5:66:26:5d:4b:33:5d:8c (DSA)
  754. |_2048 16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48 (RSA)
  755. 80/tcp open http
  756. 443/tcp open https
  757. 9418/tcp open git
  758. Nmap done: 1 IP address (1 host up) scanned in 28.78 seconds
  759. Another way is to check one's own ``known_hosts`` file, using this one-liner:
  760. .. code-block:: bash
  761. $ ssh-keygen -l -f /dev/stdin <<<`ssh-keyscan github.com 2>/dev/null` | awk '{print $2}'
  762. 16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48
  763. .. warning::
  764. AWS tracks usage of nmap and may flag it as abuse. On AWS hosts, the
  765. ``ssh-keygen`` method is recommended for host key verification.
  766. .. note::
  767. As of `OpenSSH 6.8`_ the SSH fingerprint is now shown as a base64-encoded
  768. SHA256 checksum of the host key. So, instead of the fingerprint looking
  769. like ``16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48``, it would look
  770. like ``SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8``.
  771. .. _`OpenSSH 6.8`: http://www.openssh.com/txt/release-6.8
  772. Refreshing gitfs Upon Push
  773. ==========================
  774. By default, Salt updates the remote fileserver backends every 60 seconds.
  775. However, if it is desirable to refresh quicker than that, the :ref:`Reactor
  776. System <reactor>` can be used to signal the master to update the fileserver on
  777. each push, provided that the git server is also a Salt minion. There are three
  778. steps to this process:
  779. 1. On the master, create a file **/srv/reactor/update_fileserver.sls**, with
  780. the following contents:
  781. .. code-block:: yaml
  782. update_fileserver:
  783. runner.fileserver.update
  784. 2. Add the following reactor configuration to the master config file:
  785. .. code-block:: yaml
  786. reactor:
  787. - 'salt/fileserver/gitfs/update':
  788. - /srv/reactor/update_fileserver.sls
  789. 3. On the git server, add a `post-receive hook`_
  790. a. If the user executing `git push` is the same as the minion user, use the following hook:
  791. .. code-block:: bash
  792. #!/usr/bin/env sh
  793. salt-call event.fire_master update salt/fileserver/gitfs/update
  794. b. To enable other git users to run the hook after a `push`, use sudo in the hook script:
  795. .. code-block:: bash
  796. #!/usr/bin/env sh
  797. sudo -u root salt-call event.fire_master update salt/fileserver/gitfs/update
  798. 4. If using sudo in the git hook (above), the policy must be changed to permit
  799. all users to fire the event. Add the following policy to the sudoers file
  800. on the git server.
  801. .. code-block:: bash
  802. Cmnd_Alias SALT_GIT_HOOK = /bin/salt-call event.fire_master update salt/fileserver/gitfs/update
  803. Defaults!SALT_GIT_HOOK !requiretty
  804. ALL ALL=(root) NOPASSWD: SALT_GIT_HOOK
  805. The ``update`` argument right after :mod:`event.fire_master
  806. <salt.modules.event.fire_master>` in this example can really be anything, as it
  807. represents the data being passed in the event, and the passed data is ignored
  808. by this reactor.
  809. Similarly, the tag name ``salt/fileserver/gitfs/update`` can be replaced by
  810. anything, so long as the usage is consistent.
  811. The ``root`` user name in the hook script and sudo policy should be changed to
  812. match the user under which the minion is running.
  813. .. _`post-receive hook`: http://www.git-scm.com/book/en/Customizing-Git-Git-Hooks#Server-Side-Hooks
  814. .. _git-as-ext_pillar:
  815. Using Git as an External Pillar Source
  816. ======================================
  817. The git external pillar (a.k.a. git_pillar) has been rewritten for the 2015.8.0
  818. release. This rewrite brings with it pygit2_ support (allowing for access to
  819. authenticated repositories), as well as more granular support for per-remote
  820. configuration. This configuration schema is detailed :ref:`here
  821. <git-pillar-configuration>`.
  822. .. _faq-gitfs-bug:
  823. Why aren't my custom modules/states/etc. syncing to my Minions?
  824. ===============================================================
  825. In versions 0.16.3 and older, when using the :mod:`git fileserver backend
  826. <salt.fileserver.gitfs>`, certain versions of GitPython may generate errors
  827. when fetching, which Salt fails to catch. While not fatal to the fetch process,
  828. these interrupt the fileserver update that takes place before custom types are
  829. synced, and thus interrupt the sync itself. Try disabling the git fileserver
  830. backend in the master config, restarting the master, and attempting the sync
  831. again.
  832. This issue is worked around in Salt 0.16.4 and newer.