diff --git a/advanced/option-database-tuning.rst b/advanced/option-database-tuning.rst deleted file mode 100644 index c4496da1a..000000000 --- a/advanced/option-database-tuning.rst +++ /dev/null @@ -1,29 +0,0 @@ -.. _option-databaseTuning: - -databaseTuning -============== - -.. versionadded:: 1.3.0 - -The ``databaseTuning`` option controls how Syncthing uses the backend -key-value database that stores the index data and other persistent data -Syncthing needs. In most cases this database is fairly small (hundred -megabytes or less) and the default tuning is optimized for this. However in -cases with large folders or many devices the database may grow, and updates -may be more frequent. In these cases it's better to use larger buffers, -allow more memory to be used for cache, allow a larger amount of overhead on -disk in order to improve compaction performance, and so on. - -The ``databaseTuning`` option can have one of three values: - -- ``small``: This is the old set of tuning parameters, recommended for small - databases. -- ``large``: This is a new set of tuning parameters, recommended for large - databases. - -- ``auto``: Syncthing will automatically use either the large or small - tuning parameters depending on the database size on disk. A database that is - 200 MiB or more in size is considered "large" for the purposes of this - option. - -The default value is ``auto``. diff --git a/dev/building.rst b/dev/building.rst index d514d1bff..879521174 100644 --- a/dev/building.rst +++ b/dev/building.rst @@ -32,7 +32,10 @@ Prerequisites - The latest stable version of Go. The previous stable version should also work; older versions will likely not work. This largely follows Go's `Release Policy `__. -- Git +- Usually, a C compiler. Syncthing uses SQLite for storing data and the + most common implementation is written in C. +- Git, unless you're building from a downloaded source tarball that includes + a ``RELEASE`` marker with the version information. - If you want to build Debian packages FPM is required. See FPM's `installation information `__. - To build Windows executables, installing `goversioninfo @@ -44,11 +47,6 @@ If you're not already a Go developer, the easiest way to get going is to download the latest version of Go as instructed in https://go.dev/doc/install. -.. note:: - Because Syncthing uses Go modules you do not need to set or care about "GOPATH". - However, the GOPATH still defaults to ``~/go`` and you'd be best to *not* - put your Syncthing source in there, for now. - Building (Unix) --------------- @@ -109,7 +107,13 @@ The following ``build.go`` subcommands and options exist. ``go run build.go build`` Builds just the named target, or ``syncthing`` by default, to the current directory. Use this when cross compiling, with parameters for what to cross - compile to: ``go run build.go --goos linux --goarch 386 build``. + compile to: ``go run build.go --goos linux --goarch 386 build``. When + cross compiling you need to have a C compiler for the target platform, and + you need to let Go know how to invoke it. The ``--cc`` flag to build.go + controls this, together with the environment variable ``CGO_ENABLED=1``. + For example, to cross build from Linux to Windows using Zig as the C + compiler, ``go run build.go -goos windows -goarch amd64 -cc "zig cc + -target x86_64-windows" build``. ``go run build.go test`` Runs the tests. @@ -120,11 +124,13 @@ The following ``build.go`` subcommands and options exist. ``go run build.go tar`` Creates a Syncthing tar.gz dist file in the current directory. Assumes a - Unixy build. + Unixy build. Otherwise same considerations and parameters as for the + ``build`` subcommand. ``go run build.go zip`` Creates a Syncthing zip dist file in the current directory. Assumes a - Windows build. + Windows build. Otherwise same considerations and parameters as for the + ``build`` subcommand. The options ``--no-upgrade``, ``--goos`` and ``--goarch`` can be given to influence ``build``, ``tar`` and ``zip``. Examples: diff --git a/includes/env-vars.rst b/includes/env-vars.rst index e0fa06aeb..c28fcece3 100644 --- a/includes/env-vars.rst +++ b/includes/env-vars.rst @@ -8,8 +8,6 @@ STTRACE Main and operational facilities: config Configuration loading and saving. - db - The database layer. main Main package. model @@ -51,6 +49,8 @@ STTRACE REST API. sha256 SHA256 hashing package (this facility currently unused). + sqlite + The SQLite database stats Persistent device and folder statistics. sync @@ -63,65 +63,14 @@ STTRACE all All of the above. -STBLOCKPROFILE - Write block profiles to ``block-$pid-$timestamp.pprof`` every 20 seconds. - -STCPUPROFILE - Write a CPU profile to ``cpu-$pid.pprof`` on exit. - -STDEADLOCKTIMEOUT - Used for debugging internal deadlocks; sets debug sensitivity. Use only - under direction of a developer. - STLOCKTHRESHOLD Used for debugging internal deadlocks; sets debug sensitivity. Use only under direction of a developer. -STGUIADDRESS - Override GUI listen address. Equivalent to passing :option:`--gui-address`. - -STGUIAPIKEY - Override the API key needed to access the GUI / REST API. Equivalent to - passing :option:`--gui-apikey`. - -STGUIASSETS - Directory to load GUI assets from. Overrides compiled in assets. Useful for - developing webgui, commonly use ``STGUIASSETS=gui bin/syncthing``. - -STHEAPPROFILE - Write heap profiles to ``heap-$pid-$timestamp.pprof`` each time heap usage - increases. - -STNODEFAULTFOLDER - Don't create a default folder when starting for the first time. This - variable will be ignored anytime after the first run. Equivalent to the - :option:`--no-default-folder` flag. - -STNORESTART - Equivalent to the :option:`--no-restart` flag. - -STNOUPGRADE - Disable automatic upgrades. Equivalent to the :option:`--no-upgrade` flag. - -STPROFILER - Set to a listen address such as "127.0.0.1:9090" to start the profiler with - HTTP access, which then can be reached at - http://localhost:9090/debug/pprof. See ``go tool pprof`` for more - information. - -STPERFSTATS - Write running performance statistics to ``perf-$pid.csv``. Not supported on - Windows. - -STRECHECKDBEVERY - Time before folder statistics (file, dir, ... counts) are recalculated from - scratch. The given duration must be parseable by Go's ``time.ParseDuration``. If - missing or not parseable, the default value of 1 month is used. To force - recalculation on every startup, set it to ``1s``. - -STGCINDIRECTEVERY - Sets the time interval in between database garbage collection runs. The - given duration must be parseable by Go's ``time.ParseDuration``. +STVERSIONEXTRA + Add extra information to the version string in logs and the version line + in the GUI. Can be set to the name of a wrapper or tool controlling + syncthing to communicate this to the end user. GOMAXPROCS Set the maximum number of CPU cores to use. Defaults to all available CPU diff --git a/rest/system-config-get.rst b/rest/system-config-get.rst index 02723f9c7..17a3891b1 100644 --- a/rest/system-config-get.rst +++ b/rest/system-config-get.rst @@ -50,9 +50,7 @@ Returns the current configuration. "pullerPauseS": 0, "maxConflicts": -1, "disableSparseFiles": false, - "disableTempIndexes": false, "paused": false, - "weakHashThresholdPct": 25, "markerName": ".stfolder", "copyOwnershipFromParent": false, "modTimeWindowS": 0, @@ -171,7 +169,6 @@ Returns the current configuration. "stunServers": [ "default" ], - "databaseTuning": "auto", "maxConcurrentIncomingRequestKiB": 0, "announceLANAddresses": true, "sendFullIndexOnUpgrade": false, @@ -226,9 +223,7 @@ Returns the current configuration. "pullerPauseS": 0, "maxConflicts": 10, "disableSparseFiles": false, - "disableTempIndexes": false, "paused": false, - "weakHashThresholdPct": 25, "markerName": ".stfolder", "copyOwnershipFromParent": false, "modTimeWindowS": 0, diff --git a/rest/system-paths-get.rst b/rest/system-paths-get.rst index f27f3f321..edb858c23 100644 --- a/rest/system-paths-get.rst +++ b/rest/system-paths-get.rst @@ -16,7 +16,7 @@ and others. "certFile": "/home/user/.config/syncthing/cert.pem", "config": "/home/user/.config/syncthing/config.xml", "csrfTokens": "/home/user/.config/syncthing/csrftokens.txt", - "database": "/home/user/.local/share/syncthing/index-v0.14.0.db", + "database": "/home/user/.local/share/syncthing/index-v2", "defFolder": "/home/user/Sync", "guiAssets": "/home/user/src/syncthing/gui", "httpsCertFile": "/home/user/.config/syncthing/https-cert.pem", diff --git a/specs/bep-v1.rst b/specs/bep-v1.rst index a384769ec..1818b47a5 100644 --- a/specs/bep-v1.rst +++ b/specs/bep-v1.rst @@ -235,13 +235,11 @@ Protocol Buffer Schema } message Folder { - string id = 1; - string label = 2; - bool read_only = 3; - bool ignore_permissions = 4; - bool ignore_delete = 5; - bool disable_temp_indexes = 6; - bool paused = 7; + string id = 1; + string label = 2; + FolderType type = 3; + FolderStopReason stop_reason = 7; + reserved 4 to 6; repeated Device devices = 16; } @@ -265,6 +263,18 @@ Protocol Buffer Schema ALWAYS = 2; } + enum FolderType { + FOLDER_TYPE_SEND_RECEIVE = 0; + FOLDER_TYPE_SEND_ONLY = 1; + FOLDER_TYPE_RECEIVE_ONLY = 2; + FOLDER_TYPE_RECEIVE_ENCRYPTED = 3; + } + + enum FolderStopReason { + FOLDER_STOP_REASON_RUNNING = 0; + FOLDER_STOP_REASON_PAUSED = 1; + } + Fields (Cluster Config Message) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -284,20 +294,12 @@ the folder. The **label** field contains the folder label, the human readable name of the folder. -The **read_only** field is set for folders that the device will accept no -updates from the network for. - -The **ignore_permissions** field is set for folders that the device will not -accept or announce file permissions for. - -The **ignore_delete** field is set for folders that the device will ignore -deletes for. - -The **disable_temp_indexes** field is set for folders that will not dispatch -and do not wish to receive progress updates about partially downloaded files -via Download Progress messages. +The **folder_type** field contains the type of the folder as defined by the +FolderType enumeration. -The **paused** field is set for folders that are currently paused. +The **stop_reason** field is set for folders that are currently stopped for +any reason. The zero means that the folder is not stopped; further reasons +are defined by the FolderStopReason enumeration. The **devices** field is a list of devices participating in sharing this folder. @@ -401,10 +403,10 @@ Protocol Buffer Schema } message BlockInfo { - int64 offset = 1; - int32 size = 2; - bytes hash = 3; - uint32 weak_hash = 4; + int64 offset = 1; + int32 size = 2; + bytes hash = 3; + reserved 4; } message Vector { @@ -511,6 +513,8 @@ Protocol Buffer Schema int32 size = 5; bytes hash = 6; bool from_temporary = 7; + int32 block_no = 9; + reserved 8; } Fields diff --git a/users/config.rst b/users/config.rst index 823f5869b..7553c1495 100644 --- a/users/config.rst +++ b/users/config.rst @@ -63,9 +63,9 @@ corresponding environment variables (``$STDATADIR`` or ``STHOMEDIR``). The database directory contains the following files, among others: -:file:`index-{*}.db` - A directory holding the database with metadata and hashes of the files - currently on disk and available from peers. +:file:`index-{*}` + The database with metadata and hashes of the files currently on disk and + available from peers. :file:`syncthing.log` Log output, on some systems. @@ -112,9 +112,7 @@ The following shows an example of a default configuration file (IDs will differ) 0 -1 false - false false - 25 .stfolder false 0 @@ -188,7 +186,6 @@ The following shows an example of a default configuration file (IDs will differ) 180 20 default - auto 0 true false @@ -217,9 +214,7 @@ The following shows an example of a default configuration file (IDs will differ) 0 10 false - false false - 25 .stfolder false 0 @@ -305,9 +300,7 @@ Folder Element 0 -1 false - false false - 25 .stfolder false 0 @@ -512,23 +505,10 @@ The following child elements may exist: to be sparse on filesystems that support this feature. When set to ``true``, sparse files will not be created. -.. option:: folder.disableTempIndexes - - By default, devices exchange information about blocks available in - transfers that are still in progress, which allows other devices to - download parts of files that are not yet fully downloaded on your own - device, essentially making transfers more torrent like. When set to - ``true``, such information is not exchanged for this folder. - .. option:: folder.paused True if this folder is (temporarily) suspended. -.. option:: folder.weakHashThresholdPct - - Use weak hash if more than the given percentage of the file has changed. Set - to ``-1`` to always use weak hash. Default is ``25``. - .. option:: folder.markerName Name of a directory or file in the folder root to be used as a marker - see @@ -1049,7 +1029,6 @@ Options Element 180 20 default - auto 0 true false @@ -1301,13 +1280,6 @@ The ``options`` element contains all other global configuration options. feature. Set ``false`` to keep Syncthing from sending panic logs on serious troubles. Defaults to ``true``, to help the developers troubleshoot. -.. option:: options.databaseTuning - - Controls how Syncthing uses the backend key-value database that stores the - index data and other persistent data it needs. The available options and - implications are explained in a :doc:`separate chapter - `. - .. option:: options.maxConcurrentIncomingRequestKiB This limits how many bytes we have "in the air" in the form of response data @@ -1391,9 +1363,7 @@ Defaults Element 0 10 false - false false - 25 .stfolder false 0 diff --git a/users/syncing.rst b/users/syncing.rst index 9d6034556..e6af5ffa5 100644 --- a/users/syncing.rst +++ b/users/syncing.rst @@ -69,8 +69,8 @@ version of the file. This version is called the *global* version and is the one that each device strives to be up to date with. This information is kept in the *index database*, which is stored in the -configuration directory and called ``index-vx.y.z.db`` (for some version -x.y.z which may not be exactly the version of Syncthing you're running). +configuration or data directory and called ``index-*``, with some version +number in place of the asterisk. When new index data is received from other devices Syncthing recalculates which version for each file should be the global version, and compares this diff --git a/users/syncthing.rst b/users/syncthing.rst index e22e1c9e3..7fa52432e 100644 --- a/users/syncthing.rst +++ b/users/syncthing.rst @@ -10,33 +10,60 @@ Synopsis :: syncthing [serve] - [--audit] [--auditfile=] [--browser-only] [--device-id] - [--generate=] [--gui-address=
] [--gui-apikey=] [--home= | --config= --data=] + [--allow-newer-config] [--audit] [--auditfile=] + [--db-maintenance-interval=] + [--db-delete-retention-interval=] + [--gui-address=
] [--gui-apikey=] [--logfile=] [--logflags=] [--log-max-old-files=] [--log-max-size=] - [--no-browser] [--no-console] [--no-restart] [--paths] [--paused] - [--no-default-folder] [--skip-port-probing] - [--reset-database] [--reset-deltas] [--unpaused] [--allow-newer-config] - [--upgrade] [--no-upgrade] [--upgrade-check] [--upgrade-to=] + [--no-browser] [--no-console] + [--no-port-probing] [--no-restart] [--no-upgrade] + [--paused] [--unpaused] [--verbose] [--version] [--help] [--debug-*] - syncthing generate - [--home= | --config=] - [--gui-user=] [--gui-password=] - [--no-default-folder] [--skip-port-probing] [--no-console] + syncthing cli + [--home= | --config= --data=] + [--gui-address=
] [--gui-apikey=] + [--help] + [command options...] [arguments...] + + syncthing browser + [--home= | --config= --data=] [--help] syncthing decrypt (--to= | --verify-only) + [--home= | --config= --data=] [--password=] [--folder-id=] [--token-path=] - [--continue] [--verbose] [--version] [--help] + [--continue] [--verbose] [--help] - syncthing cli + syncthing device-id + [--home= | --config= --data=] + [--help] + + syncthing generate + [--home= | --config= --data=] + [--gui-user=] [--gui-password=] + [--no-port-probing] + [--help] + + syncthing paths + [--home= | --config= --data=] + [--help] + + syncthing upgrade + [--home= | --config= --data=] + [--check-only] [--from=] + [--help] + + syncthing version + [--home= | --config= --data=] + [--help] + + syncthing debug [--home= | --config= --data=] - [--gui-address=
] [--gui-apikey=] [--help] - [command options...] [arguments...] Description ----------- @@ -56,8 +83,34 @@ other apps like graphical system integration helpers can use as well, for greatest flexibility. A link to reach the GUI and API is printed among the first few log messages. -Options -------- +Common options +-------------- + +.. cmdoption:: --home=, -H + + Set common configuration and data directory. The default configuration + directory is ``$XDG_STATE_HOME/syncthing`` or + ``$HOME/.local/state/syncthing`` (Unix-like), + ``$HOME/Library/Application Support/Syncthing`` (Mac) and + ``%LOCALAPPDATA%\Syncthing`` (Windows). + +.. cmdoption:: --config=, -C + + Set configuration directory. Alternative to ``--home`` and must be used + together with ``--data``. + +.. cmdoption:: --data=, -D + + Set data (e.g. database) directory. Alternative to ``--home`` and must be used + together with ``--config``. + +.. cmdoption:: --help, -h + + Show help text about command line usage. Context-sensitive depending on the + given subcommand. + +Serve options +------------- .. cmdoption:: --allow-newer-config @@ -73,17 +126,15 @@ Options Use specified file or stream (``"-"`` for stdout, ``"--"`` for stderr) for audit events, rather than the timestamped default file name. -.. cmdoption:: --browser-only - - Open the web UI in a browser for an already running Syncthing instance. +.. cmdoption:: --db-maintenance-interval= -.. cmdoption:: --device-id + Database maintenance interval -- internal database maintenance routines + run this often. - Print device ID to command line. +.. cmdoption:: --db-delete-retention-interval= -.. cmdoption:: --generate= - - Generate key and config in specified dir, then exit. + Database deleted item retention interval -- deleted items are forgotten + from the database after this interval. .. cmdoption:: --gui-address=
@@ -98,41 +149,6 @@ Options Override the API key needed to access the GUI / REST API. -.. cmdoption:: --gui-password= - - Specify new GUI authentication password, to update the config file. Read - from the standard input stream if only a single dash (``-``) is given. A - plaintext password is hashed before writing to the config file, but an - already bcrypt-hashed input is stored verbatim. As a special case, giving - the existing password hash as password will leave it untouched. - -.. cmdoption:: --gui-user= - - Specify new GUI authentication user name, to update the config file. - -.. cmdoption:: --help, -h - - Show help text about command line usage. Context-sensitive depending on the - given subcommand. - -.. cmdoption:: --home= - - Set common configuration and data directory. The default configuration - directory is ``$XDG_STATE_HOME/syncthing`` or - ``$HOME/.local/state/syncthing`` (Unix-like), - ``$HOME/Library/Application Support/Syncthing`` (Mac) and - ``%LOCALAPPDATA%\Syncthing`` (Windows). - -.. cmdoption:: --config= - - Set configuration directory. Alternative to ``--home`` and must be used - together with ``--data``. - -.. cmdoption:: --data= - - Set data (e.g. database) directory. Alternative to ``--home`` and must be used - together with ``--config``. - .. cmdoption:: --logfile= Set destination filename for logging (use ``"-"`` for stdout, which is the @@ -170,10 +186,10 @@ Options Hide the console window. (On Windows only) -.. cmdoption:: --no-default-folder +.. cmdoption:: --no-port-probing - Don't create a default folder when generating an initial configuration / - starting for the first time. + Don't try to find unused random ports for the GUI and listen address when + generating an initial configuration / starting for the first time. .. cmdoption:: --no-restart @@ -185,32 +201,10 @@ Options Disable automatic upgrades. Equivalent to the ``STNOUPGRADE`` environment variable, see below. -.. cmdoption:: --paths - - Print the paths used for configuration, keys, database, GUI overrides, - default sync folder and the log file. - .. cmdoption:: --paused Start with all devices and folders paused. -.. cmdoption:: --reset-database - - Reset the database, forcing a full rescan and resync. Create `.stfolder` - folders in each sync folder if they do not already exist. **Caution**: - Ensure that all sync folders which are mountpoints are already mounted. - Inconsistent versions may result if the mountpoint is later mounted and - contains older versions. - -.. cmdoption:: --reset-deltas - - Reset delta index IDs, forcing a full index exchange. - -.. cmdoption:: --skip-port-probing - - Don't try to find unused random ports for the GUI and listen address when - generating an initial configuration / starting for the first time. - .. cmdoption:: --unpaused Start with all devices and folders unpaused. @@ -219,21 +213,12 @@ Options Perform upgrade. -.. cmdoption:: --upgrade-check - - Check for available upgrade. - -.. cmdoption:: --upgrade-to= - - Force upgrade directly from specified URL. - .. cmdoption:: --verbose Print verbose log output. -.. cmdoption:: --version - - Show version. +Decrypt options +--------------- .. cmdoption:: --to= @@ -261,6 +246,33 @@ Options Continue processing next file in case of error, instead of aborting. +Generate options +---------------- + +.. cmdoption:: --gui-password= + + Specify new GUI authentication password, to update the config file. Read + from the standard input stream if only a single dash (``-``) is given. A + plaintext password is hashed before writing to the config file, but an + already bcrypt-hashed input is stored verbatim. As a special case, giving + the existing password hash as password will leave it untouched. + +.. cmdoption:: --gui-user= + + Specify new GUI authentication user name, to update the config file. + +Upgrade options +--------------- + +.. cmdoption:: --check-only + + Do not upgrade, only indicate whether an upgrade is available. + +.. cmdoption:: --from= + + Upgrade to the Syncthing version available from loading the package at + the given URL. + Exit Codes ---------- diff --git a/users/tuning.rst b/users/tuning.rst index cf34b192c..d5d8788ed 100644 --- a/users/tuning.rst +++ b/users/tuning.rst @@ -32,11 +32,6 @@ is on an SSD. First some general options: performance. If the underlying storage is a large array of disk, then syncing many folders concurrently may be beneficial. -- :opt:`databaseTuning` - Set to ``large``. Regardless of the size of the actual database, this - increases a number of buffers and settings to optimize for higher - throughput. - - :opt:`maxConcurrentIncomingRequestKiB` This sets the maximum amount of data being processed (loaded from disk and transmitted over the network) concurrently at any given @@ -76,13 +71,6 @@ These options are folder specific and should be set on each folder: is effort that could be spent on something more productive. If you don't need to see scan progress, set this to -1 to disable it. -- :opt:`weakHashThresholdPct` - Syncthing will by default look for rolling (weak) hash matches to detect - data shifted in a file if a lot of data has changed in the file. If your - use case doesn't cause data to shift in a file, and if the files are - large (movies, VM images, ...) it is unnecessary to spend time checking - weak hashes. Set the threshold to 101% to disable use of weak hashes. - - :opt:`maxConcurrentWrites` Synchting limits the number of outstanding write system calls at any given time to avoid overloading the I/O system. If you increased @@ -152,11 +140,6 @@ General options: - :opt:`maxFolderConcurrency` Set to 1 to sync folders sequentially, reducing the peak memory usage. -- :opt:`databaseTuning` - Set to ``small``. Regardless of the size of the actual database size, - this reduces the size of a number of buffers to optimize for reduced - memory usage. - - :opt:`maxConcurrentIncomingRequestKiB` Set to 32 MiB to reduce the amount of memory used for buffering responses to incoming requests. @@ -179,10 +162,6 @@ Folders options: Set to -1 to disable scan progress updates. Keeping track of scan progress uses memory and CPU. -- :opt:`weakHashThresholdPct` - Set to 101% to disable use of weak hashes. Using weak hashes has a - memory cost. - - :opt:`copyRangeMethod` If your underlying filesystem supports it, using copyrange is more efficient than having Syncthing do the data copying.