Timeouts when trying to connect to mongo (after it works for a random time)

[Nasicus]

We use the mongo image to run our e2e test stack, together with cypress.

Basically what we have is a various set of tests:

• we access the mongo db from the tests (via js)
• we access the mongo db from the api (via c#)

Since a long time we have again and again issues, that we get timeouts after the tests are running for a while... at some point I could fix this by specifically using the image mongo:3.6.17.... however now it doesn't work again...

I'm not sure if it's even the problem of the image, but I'm desperate - I already lost so many hours trying to fix this and figure out what's going on.

The errors look something like this (this is now by using command: --serviceExecutor adaptive, but it doesn't matter, since it also happens without this):
(also note the IP is generated by docker-compose, not hardcoded)

MongoServerSelectionError: connection 5 to 192.168.224.3:27017 timed out
    at Timeout._onTimeout (/node_modules/mongodb/lib/core/sdam/topology.js:428:30)

While the tests are running I can normally also connect to this mongo image by using this command:

mongo mongodb://192.168.224.3:27017

However once I try this after the first timeout from above occurred then it hangs forever and it never connects.

Maybe of interest is also, that I can still ping this IP-address.

Any help and or pointers are greatly appreciated.

This is my docker-compose file:

version: '3.7'
services:
  blob:
    image: mcr.microsoft.com/azure-storage/azurite
    expose:
      - '10000'
      - '10001'

  mongo:
    image: mongo
    expose:
      - '27017'

  redis:
    image: redis
    expose:
      - '6379'

  api:
    build:
      context: ../api
      args:
        - VERSION=0.0.0.1
    expose:
      - '5000'

    environment:
      - ASPNETCORE_ENVIRONMENT=ci-tests
      - ASPNETCORE_IS_DOCKER=TRUE
      - ASPNETCORE_URLS=http://+:5000
      - Database__Global__Connection__ConnectionString=mongodb://mongo
      - Database__Content__Connections__default__ConnectionString=mongodb://mongo
      - Database__Content__Connections__special__ConnectionString=mongodb://mongo
      - Azure__Scheduler__Disabled=true
      - SendGrid__IsDisabled=true
      - Azure__Redis__ConnectionString=redis
      - Azure__Storage__ConnectionString=DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://blob:10000/devstoreaccount1;QueueEndpoint=http://blob:10001/devstoreaccount1;

    depends_on:
      - 'redis'
      - 'mongo'
      - 'blob'

  web:
    build:
      context: ../web

    expose:
      - '3000'

    environment:
      - ASPNETCORE_ENVIRONMENT=ci-tests
      - ASPNETCORE_IS_DOCKER=TRUE
      - ASPNETCORE_URLS=http://+:3000
      - Client__ApiEndpoint=http://api:5000/api/v1

    depends_on:
      - 'api'

  test-api:
    build:
      context: ../api
      dockerfile: Dockerfile-TestApi

    expose:
      - '5001'

    environment:
      - ASPNETCORE_ENVIRONMENT=ci-tests
      - ASPNETCORE_IS_DOCKER=TRUE
      - ASPNETCORE_URLS=http://+:5001
      - Database__Global__Connection__ConnectionString=mongodb://mongo
      - Database__Content__Connections__default__ConnectionString=mongodb://mongo
      - Database__Content__Connections__special__ConnectionString=mongodb://mongo
      - Azure__Redis__ConnectionString=redis

    depends_on:
      - 'mongo'

  cypress:
    depends_on:
      - 'api'
      - 'test-api'
      - 'web'
      - mongo

    build: .

    environment:
      - NODE_ENV=CI
      - CYPRESS_MY_PROJECT_API_BASE=http://api:5000
      - CYPRESS_MY_PROJECT_API=http://api:5000/api/v1
      - CYPRESS_MY_PROJECT_ADMIN_API=http://api:5000/admin-api/v1
      - CYPRESS_MY_PROJECT_TEST_API=http://api:5000/test-api/v1
      - CYPRESS_MY_PROJECT_WEBJOBS_API=http://test-api:5001
      - CYPRESS_MY_PROJECT_WEB=http://web:3000
      - CYPRESS_CYPRESS_BASE_URL=http://web:3000
      - CYPRESS_DATABASE_HOST=mongo
      - CYPRESS_BLOB_HOST_NAME=blob
      - TZ=Europe/Paris

This is how we connect from the client (js). Note that we try to cache connections, and NEVER close them manually (until docker-compose is done and we remove the images again), I read that it is "best practice":

let databases = {};
async function getClient(host, database) {
  const cacheKey = host + database;
  if (databases[cacheKey] != null) {
    return databases[cacheKey];
  }

  return (databases[cacheKey] = MongoClient.connect(
    `mongodb://${host}/${database}`,
    {
      useUnifiedTopology: true,
      useNewUrlParser: true
    }
  ));
}

This is how we connect from the api (c#) - the service is a singleton, so the connection should only be opened once:

    private static MongoClientSettings CreateMongoClientSettings(DatabaseConnectionConfig config)
    {
        return MongoClientSettings.FromUrl(new MongoUrl(config.ConnectionString));
    }

EDIT:

Now I ran it with

command: --serviceExecutor adaptive -vvvvv

And it seemd to crash even earlier, attached the logs I got via docker logs <imagename>

mongo-log.txt

[Nasicus]

After more desperate tries, I started to think (for the first time....haha) what we do what a "heavy-used-mongo-docker" normally doesn't do.

We have around 3000 tests, each of these tests obviously needs clean test data, therefore we call almost in all tests and often even more than once (among others) these commands:

await db.collection(collectionName).drop();

// and then...

// either... (if we have no documents)
await db.createCollection(collectionName);

//.. or...
 await db.collection(collectionName).insertMany(documents);

What I did now I replaced the drop with:

await db.collection(collectionName).remove();

and the timeouts do not appear randomly anymore....however for me it's too early to celebrate. There were also other times when I thought I solved it and suddenly it happened again.

Anyway is it possible that mongo cannot handle a lot of drops and recreations of the same collections in a short amount of time?

[thomasrstorey]

I am experiencing a very similar issue. Using mongodb in docker for an integration test suite. I can get through all the tests if I run them serially, but if I run them with any parallelization the mongod process on the mongodb container stops accepting connections. I can shell into the container and poke around - it's not using any resources, it's just doing nothing. Seems like a network/connection issue to me? Not sure how to debug further.

[tianon]

Honestly, I'm having a hard time seeing anything in our image that could be causing this -- we're installing upstream's stock packages, and at the time you are all seeing issues it's long-since running mongod directly, so it can't really be particular to anything that we've done/added.

What I'd recommend is trying a community support forum that's dedicated to MongoDB and asking for help debugging further there (since we're clearly not going to be much use, given the amount of time it's taken me to even get back to this issue 😅).