Invalid output in RPC API docs for /api/v0/dag/import?stats=true

[crimeminister]

Checklist

• This is a bug report, not a question. Ask questions on discuss.ipfs.io.
• I have searched on the issue tracker for my bug.
• I am running the latest go-ipfs version or have an issue updating.

Installation method

ipfs-update or dist.ipfs.io

Version

go-ipfs version: 0.11.0
Repo version: 11
System version: amd64/linux
Golang version: go1.16.12

Config

{
  "API": {
    "HTTPHeaders": {}
  },
  "Addresses": {
    "API": "/ip4/127.0.0.1/tcp/5001",
    "Announce": [],
    "AppendAnnounce": null,
    "Gateway": "/ip4/127.0.0.1/tcp/8080",
    "NoAnnounce": [],
    "Swarm": [
      "/ip4/0.0.0.0/tcp/4001",
      "/ip6/::/tcp/4001",
      "/ip4/0.0.0.0/udp/4001/quic",
      "/ip6/::/udp/4001/quic"
    ]
  },
  "AutoNAT": {},
  "Bootstrap": [
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt",
    "/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
    "/ip4/104.131.131.82/udp/4001/quic/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ"
  ],
  "DNS": {
    "Resolvers": null
  },
  "Datastore": {
    "BloomFilterSize": 0,
    "GCPeriod": "1h",
    "HashOnRead": false,
    "Spec": {
      "mounts": [
        {
          "child": {
            "path": "blocks",
            "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2",
            "sync": true,
            "type": "flatfs"
          },
          "mountpoint": "/blocks",
          "prefix": "flatfs.datastore",
          "type": "measure"
        },
        {
          "child": {
            "compression": "none",
            "path": "datastore",
            "type": "levelds"
          },
          "mountpoint": "/",
          "prefix": "leveldb.datastore",
          "type": "measure"
        }
      ],
      "type": "mount"
    },
    "StorageGCWatermark": 90,
    "StorageMax": "10GB"
  },
  "Discovery": {
    "MDNS": {
      "Enabled": true,
      "Interval": 10
    }
  },
  "Experimental": {
    "AcceleratedDHTClient": false,
    "FilestoreEnabled": false,
    "GraphsyncEnabled": false,
    "Libp2pStreamMounting": false,
    "P2pHttpProxy": false,
    "StrategicProviding": false,
    "UrlstoreEnabled": false
  },
  "Gateway": {
    "APICommands": [],
    "HTTPHeaders": {
      "Access-Control-Allow-Headers": [
        "X-Requested-With",
        "Range",
        "User-Agent"
      ],
      "Access-Control-Allow-Methods": [
        "GET"
      ],
      "Access-Control-Allow-Origin": [
        "*"
      ]
    },
    "NoDNSLink": false,
    "NoFetch": false,
    "PathPrefixes": [],
    "PublicGateways": null,
    "RootRedirect": "",
    "Writable": false
  },
  "Identity": {
    "PeerID": "12D3KooWGn8esyGN8DB1UCsVTZPWxkr6ELGBP9AffLNQqbrfejv7"
  },
  "Internal": {},
  "Ipns": {
    "RecordLifetime": "",
    "RepublishPeriod": "",
    "ResolveCacheSize": 128
  },
  "Migration": {
    "DownloadSources": null,
    "Keep": ""
  },
  "Mounts": {
    "FuseAllowOther": false,
    "IPFS": "/ipfs",
    "IPNS": "/ipns"
  },
  "Peering": {
    "Peers": [
      {
        "Addrs": [
          "/ip4/3.134.223.177/udp/6746/quic"
        ],
        "ID": "12D3KooWN8vAoGd6eurUSidcpLYguQiGZwt4eVgDvbgaS7kiGTup"
      },
      {
        "Addrs": [
          "/ip4/147.75.49.71/udp/6746/quic",
          "/ip4/147.75.49.71/tcp/6745"
        ],
        "ID": "12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien"
      },
      {
        "Addrs": [
          "/ip4/147.75.86.255/udp/6746/quic",
          "/ip4/147.75.86.255/tcp/6745"
        ],
        "ID": "12D3KooWFrnuj5o3tx4fGD2ZVJRyDqTdzGnU3XYXmBbWbc8Hs8Nd"
      },
      {
        "Addrs": [
          "/ip4/212.6.53.9/udp/6746/quic",
          "/ip4/212.6.53.9/tcp/6745"
        ],
        "ID": "12D3KooWNNuTcH39tagS4MyddgQP7jBNSHJn4cVuTKuY4bjNZ6cw"
      }
    ]
  },
  "Pinning": {
    "RemoteServices": {
      "pinata": {
        "API": {
          "Endpoint": "https://api.pinata.cloud/psa"
        },
        "Policies": {
          "MFS": {
            "Enable": false,
            "PinName": "",
            "RepinInterval": ""
          }
        }
      }
    }
  },
  "Plugins": {
    "Plugins": null
  },
  "Provider": {
    "Strategy": ""
  },
  "Pubsub": {
    "DisableSigning": false,
    "Router": ""
  },
  "Reprovider": {
    "Interval": "12h",
    "Strategy": "all"
  },
  "Routing": {
    "Type": "dht"
  },
  "Swarm": {
    "AddrFilters": null,
    "ConnMgr": {
      "GracePeriod": "20s",
      "HighWater": 900,
      "LowWater": 600,
      "Type": "basic"
    },
    "DisableBandwidthMetrics": false,
    "DisableNatPortMap": false,
    "RelayClient": {},
    "RelayService": {},
    "Transports": {
      "Multiplexers": {},
      "Network": {},
      "Security": {}
    }
  }
}

Description

I used curl to import a .car file with the stats flag set to true:

$ curl -X POST -F file=@output.car "http://127.0.0.1:5001/api/v0/dag/import?stats=true"

The docs indicate that I should get back the following response body:

{
  "Root": {
    "Cid": {
      "/": "<cid-string>"
    },
    "PinErrorMsg": "<string>"
  },
  "Stats": {
    "BlockBytesCount": "<uint64>",
    "BlockCount": "<uint64>"
  }
}

Instead I get two sequential JSON documents in the response body:

{
  "Root": {
    "Cid": {
      "/": "bafyreia5oy46mxuw2bplcncfdekg7y4mxcyebnssj7vaxokcauupmqp2i4"
    },
    "PinErrorMsg": ""
  }
}
{
  "Stats": {
    "BlockCount": 1,
    "BlockBytesCount": 15
  }
}

[welcome]

Thank you for submitting your first issue to this repository! A maintainer will be here shortly to triage and review.
In the meantime, please double-check that you have provided all the necessary information to make this process easy! Any information that can help save additional round trips is useful! We currently aim to give initial feedback within two business days. If this does not happen, feel free to leave a comment.
Please keep an eye on how this issue will be labeled, as labels give an overview of priorities, assignments and additional actions requested by the maintainers:

• "Priority" labels will show how urgent this is for the team.
• "Status" labels will show if this is ready to be worked on, blocked, or in progress.
• "Need" labels will indicate if additional input or analysis is required.

Finally, remember to use https://discuss.ipfs.io if you just need general support.

[lidel]

This one is tricky, dag/import returns multiple CarImportOutput events even if there are no stats – one for every imported root. If there are stats, then its +1 event.

Most of the CAR files have only one root, and stats are opt-in, so we did not notice this before.

I think we should fix https://github.com/ipfs/ipfs-docs/tree/main/tools/http-api-docs generator somehow, so it is able to correctly indicate when a command response type is ndjson / https://jsonlines.org (streaming response, one JSON Object per line)

Right now the "output example" at https://docs.ipfs.io/reference/http/api/#api-v0-dag-import is based on the Type field of cmds.Command:

https://github.com/ipfs/go-ipfs/blob/d59730f51eb537a5776091503887edb76bbc1bf6/core/commands/dag/dag.go#L212

https://github.com/ipfs/go-ipfs/blob/d59730f51eb537a5776091503887edb76bbc1bf6/core/commands/dag/dag.go#L58-L73

Adding this to the best effort track, as we should fix this in generic fashion + apply to any other streaming commands.