Installing GetOutline - An Emerging Knowledge Management System for your second brain

[Hydraulics]

Hi all, I came across a company called GetOutline that has developed a "knowledge management" database type system similar to Notion, Roam Research, Obsidian. www.getoutline.com

These applications like Roam, Notion, Outline, etc are part of a new wave of applications that allow for people to collaborate together and form as a "second brain" for your notes, research, studies, etc. These apps are more than just basic wiki's. Apps like Roam Research and Obsidian allow for what's called "bidirectional linking" which is a bottom up level of organizing notes and thoughts together. However, some apps like Roam have a SaaS model, and others like Obsidian are free in beta but is not open source and cannot be self-hosted.

The company behind GetOutline made their solution available to be self-hosted as a docker container: outline/outline

This app has a number of dependencies which makes it difficult to install (requires slack/google authentication, Amazon S3, Postgresql, etc.) However, I saw someone who managed to install it! He used Minio to substitute the Amazon S3 database and is using Caddy as the reverse proxy instead of NPM. For those interested, he documented his steps here: Deploying Outline with Docker and Caddy – Guru Computing Blog

I wanted to install this knowledge management system and try it out, but I'm running into a problem to install this using Portainer and the docker compose file he created. My docker compose references 3 .env files ( you can see my picture below).

When I try to install Outline using the .env file I get a deployment error from Portainer saying that it cannot find the .env file. I mapped the path to my system where the .env file can be found. Is there issues in getting Portainer on Synology to run and install .env files?

Would anyone be able to help me understand what is wrong? This is my first time dealing with .env files. Usually in docker containers I've had specific environmental variables so I havent had this issue yet. I made sure to make the .env file properly by creating a textfile in notepad and saving it with the .env extension and selecting "all files" in the dropdown. Windows seems to identify it as an env file so I dont think I've done anything wrong there. Must be some config thing I'm not aware of.

T

 

[technorabilia]

Maybe this will help?

When creating a stack, the env file is not found by absolute path. · Issue #4600 · portainer/portainer

Bug description When creating a stack, the env file is not found by absolute path. Expected behavior When I specify an Env file by absolute path, I expect the file to be read and the values to be s...

github.com

You should not use the portainer_data volume, but just to illustrate...

After you have copied your .env file to the portainer volume and made it available to the portainer container:

docker run -v portainer_data:/data --name helper busybox true
docker cp your.env helper:/data
docker rm helper

You can reference the .env with:

env_file: /data/your.env

 

[Hydraulics]

Thanks @technorabilia I did see that github post you referenced but the content was a bit too technical for me to follow (still a bit of a noob but learning).

You mentioned the following:

You should not use the portainer_data volume, but just to illustrate...

After you have copied your .env file to the portainer volume and made it available to the portainer container:

1) Sorry I didnt quite understand what you mean by 'portainer_data volume'. Where is this portainer volume? Do you mean the portainer builder in the main docker folder on Synology? When I navigate to my /volume1/docker/portainer-ce sub-folder, I only see folders for "bin" "compose" and "tls". So no clue what you mean by portainer data volume.

2) Is there a way to do any troubleshooting using the GUI or using my Synology directory system? I would like to avoid command line if possible.

 

[technorabilia]

It seems that Portainer can only reference .env files that are accessibly in the Portainer container itself. Portainer cannot access files on the host.

I have installed Portainer from here and have created a volume from the command line where Portainer stores its data. In the example from my first reply I have copied the .env file to that volume so Portainer can reference it.

I don't know how you installed Portainer, but you can go to DSM, Docker, Portainer, Details, Overview, Volume and see if you have a folder mapped on the host where the Portainer data is stored. If the .env file is stored there, you can reference it from Portainer.

Example
If the host directory /volume1/portainer/data is mapped to /data and you create for example a new directory /volume1/portainer/data/env_files with a your.env file in it, then it can be referenced from Portainer as follows.

env_file: /data/env_files/your.env

Hope this makes sense...

BTW
If you specify a .env file without a full path like env_file: your.env Portainer will look in the directory where the docker-compose.yaml file is stored. This is the yaml content you specify in Portainer. This yaml content is saved as a docker-compose.yaml file in the Portainer container (and also on the host if a volume directory is mapped).

You can give this a try. It will not work, but maybe this will give you further insight in how this actually works.

BTW 2
IF you start a "stack" Portainer will start the stack with the following command:
docker-compose -f /path/to/docker-compose.yaml up -d
If you stop the stack the following command is used:
docker-compose -f /path/to/docker-compose.yaml down

Basically these are the same commands that you use when you are on the command line...

 

[Hydraulics]

Thanks so much for helping me understand this @technorabilia ! The example you used helped a ton. So now I understand that the Portainer/data folder is where the .env files needed to go. I thought they should be put in a subfolder but once I put the 3 .env files in the main portainer folder (which is what is mapped in Docker) then the env files were recognized.

I did have another related question if that's ok. In my picture of the docker compsoe file below, the container "outline" has a number of dependencies: redis, minio, and postgresql.

I already have redis installed as a separate container for another application (authelia in this case).

Will this docker compose file generate a new redis container called "outline_redis"? If so, what if I wanted to use my existing redis container called "redis"? Can I do that? Could you advise how I would modify this docker compose to use my existing redis instance?

I'm not clear on how to reuse previous containers for new docker container installations.

Much appreciate your assistance,

 

[technorabilia]

Yes, a new redis container will be created. It is only possible to reference a service within a docker-compose.yaml file.

If the service is external, you will have to reference it by ip-address. Also, depends_on will have to be implemented in the container itself (some sort of wait-for).

At least, this is my understanding.

Maybe this will help?

Docker Compose: Depends_On with an external container?

I'm trying to add depends_on for an external MySQL container named "mysql" (I don't want to create a new mysql container for this stack; I want to use the existing container). My code so far looks...

stackoverflow.com

 

[one-eyed-king]

Actualy an .env file is just a way to externalize variables that would be in the environment: block instead - you can safely "merge" the keys/values back into the compose.yml, underneath a service's environment: block.

Referencing services by their name is actualy not exclusive to containers created by docker-compose.yml. It is a feature of docker's user defined networks called service discoverey, which is nothing else than a simple (docker network scoped) dns service for services/containers.

min.io is object store compliant to the S3 api, which even uses the exact same bucket policies for permissions like AWS's Cloud Service uses. It's a great on-prem replacement for S3.

 

[technorabilia]

Yes, docker-compose config will validate the compose file and I believe also uses the variables in the .env file and move them into the environment section.

Sometimes I find it useful to obscure variables or share them between compose files.

 

[one-eyed-king]

It would make sence if docker-compose config would do that.
Personaly, I find it more usefull to keep everything tied together and less obfuscated as possible In-6-months-me realy appreciates if its able to understand things with a blink of an eye.

When I use docker-compose files, I I create Makefiles and use envsubst to archive an easy templating (Sadly Syno lacks make and the gettext package which includes envsubst), which is easy to use and has no limitation of where variable substitution can be applied to. docker-compose could realy need a deployment management like helm provides for kubernetes deployments.

 

[technorabilia]

Sorry for the off-topic...

True, but sometimes I don't want stuff like passwords in my compose files that end up in git.

I did notice that compose does not do full environment variable substitution like the shell.

Eval will work on every Synology and does do full substitution.

root@DiskStation:~/lsio/darktable# grep config docker-compose.yaml
- ${DARKTABLE_CONFIG_DIR:-${BASE_DIR:-/volume1/docker}/darktable/config}:/config
root@DiskStation:~/lsio/darktable# . .env && eval "echo \"$(cat docker-compose.yaml)\"" | grep config
- /volume1/docker/darktable/config:/config
root@DiskStation:~/lsio/darktable

 

[one-eyed-king]

It does, but make sure to escape any single and double quotes Been there done that, and I don't realy like it - though better then nothing, isn't it?

 

[Rusty]

If the host directory /volume1/portainer/data is mapped to /data and you create for example a new directory /volume1/portainer/data/env_files with a your.env file in it, then it can be referenced from Portainer as follows.

I tried this in the past and it doesn't work for me. Not in the root of data folder or in the data/sub/...

Not reading the absolute path location of ENV files is classified as a bug and in the long run, this is a pain (to run the env files in portainer volume), but in 2.6 (end of May/early June) this should be implemented according to what the DEVs (Anthony Lapenna) are talking.

Add support for `env_file` when creating stacks, services or containers · Issue #1857 · portainer/portainer

Is your feature request related to a problem? Please describe. We deploy to many environments and we have many secrets that are unique to each environment. We don't use swarm secrets because we als...

github.com

 

[Rusty]

Just an update from portainer official twitter account on the matter of env_file usage:

Still on track.. however if you want to have an early access to the changes available via #4175 you can use the portainerci/portainer:pr4175 image

 

[technorabilia]

Thanks, good to know!

In the mean time as a workaround you could start Portainer as follows:

docker run -d -p 8000:8000 -p 9000:9000 --name=portainer \
--restart=always \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /volume1/docker/portainer/data:/data \
-v /volume1/docker:/volume1/docker \
portainer/portainer-ce

Now you can reference the .env files with absolute paths as if they were on the host system. Once the Portainer fix is there you can just drop the -v /volume1/docker:/volume1/docker and should not have to change anything.

After adding -v /volume1/docker:/volume1/docker the following stack will successfully deploy.

---
version: "2.1"
services:
  domoticz:
    image: ghcr.io/linuxserver/domoticz
    container_name: domoticz
    env_file: /volume1/docker/domoticz/config/your.env

Tested on Portainer 2.11.

 

[Rusty]

As a workaround is using their dev image . Full throttle!

 

[technorabilia]

Not for me. I know all about dev builds. Let someone else test them.

 

[Hydraulics]

Thanks all,

I got around to testing this out. Thankfully the .env file path works! So that's great. But I'm falling into another stumbling block related to the redis container. For those of you that use redis container in your stack, I was hoping if you could help me understand something:

When I check the error logs of the GetOutline container (which has redis as a dependency), I'm getting the following error:

Redis error: connect ECONNREFUSED 127.0.0.1:6377....Redis error: Reached the max retries per request limit (which is 20).
Refer to "maxRetriesPerRequest" option for details.. Shutting down now.,
Retrying redis connection: attempt 21.
at Socket.emit (events.js:315:20), at Object.onceWrapper (events.js:422:26), at Socket.EventEmitter.emit (domain.js:467:12), at TCP.<anonymous> (net.js:673:12),
error Command failed with exit code 1.,
at Socket.<anonymous> (/opt/outline/node_modules/ioredis/built/redis/event_handler.js:173:37)

However, when I check the error logs of the Redis container, I'm seeing a different mesage in the Redis log below.
Any idea what the problem is?

1:C 07 Jun 2021 21:56:21.640 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo,
1:C 07 Jun 2021 21:56:21.640 # Redis version=6.2.3, bits=64, commit=00000000, modified=0, pid=1, just started,
1:C 07 Jun 2021 21:56:21.640 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf,
1:M 07 Jun 2021 21:56:21.640 * monotonic clock: POSIX clock_gettime, 1:M 07 Jun 2021 21:56:21.641 * Running mode=standalone, port=6379.,
1:M 07 Jun 2021 21:56:21.641 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.,
1:M 07 Jun 2021 21:56:21.641 # Server initialized, 1:M 07 Jun 2021 21:56:21.641 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.,
1:M 07 Jun 2021 21:56:21.641 * Loading RDB produced by version 6.2.3, 1:M 07 Jun 2021 21:56:21.641 * RDB age 3 seconds, 1:M 07 Jun 2021 21:56:21.641 * RDB memory usage when created 0.77 Mb,
1:M 07 Jun 2021 21:56:21.641 * DB loaded from disk: 0.000 seconds,
1:M 07 Jun 2021 21:56:21.641 * Ready to accept connections,

(Note: I have another Redis server being used for my Authelia container.) Am I able to reuse that Redis container for the Getoutline instance? If so, how would I reuse the existing Redis server in my Docker Compose file? I have taken a picture at the bottom of my current docker compose file for GetOutline (which is based on your instructions).





Below is my redis container that's already setup to be used by Authelia on port 6379

 

[one-eyed-king]

In order to use redis, the outline service needs to be a member of a network redis is member of. It's the same approach that you alreay apply for the nginx_default network: identify the name of the network and declare it as exernal network in your outline compose yml.

Typicaly you would create shared networks manually, and declared them as external in your compose.yml files. Networks created within a docker-compose.yml use the project name as prefix. If no network is declared the registered name of the network will be ${project name}_default.

 

[Hydraulics]

In order to use redis, the outline service needs to be a member of a network redis is member of. It's the same approach that you alreay apply for the nginx_default network: identify the name of the network and declare it as exernal network in your outline compose yml.

Typicaly you would create shared networks manually, and declared them as external in your compose.yml files. Networks created within a docker-compose.yml use the project name as prefix. If no network is declared the registered name of the network will be ${project name}_default.

Thanks for this @one-eyed-king , can you explain further what you mean? Are you saying I need to do the following:

  outline-internal:
    external: true

at the bottom of the getoutline docker compose file above? Cause I think the reason this was not made external is cause there is an internal docker network for Outline, Redis, and Postgres to talk internally. I tried this and manually made an internal network called "outline-internal" to match the code I put above and I still got the same Redis error ECONNREFUSED problem.

Redis error: connect ECONNREFUSED 127.0.0.1:6377
(node:27) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see Command-line options | Node.js v16.3.0 Documentation). (rejection id: 9),
(node:27) UnhandledPromiseRejectionWarning: SequelizeHostNotFoundError: getaddrinfo ENOTFOUND 0utl1ne,
at Client._handleErrorWhileConnecting (/opt/outline/node_modules/pg/lib/client.js:305:19),
at Client._connectionCallback (/opt/outline/node_modules/sequelize/lib/dialects/postgres/connection-manager.js:175:24),
at Connection.emit (events.js:315:20),
at Client._handleErrorEvent (/opt/outline/node_modules/pg/lib/client.js:315:19),
at Socket.emit (events.js:315:20),
at Socket.reportStreamError (/opt/outline/node_modules/pg/lib/connection.js:53:12),
at Connection.EventEmitter.emit (domain.js:467:12),
at emitErrorNT (internal/streams/destroy.js:106:8),
at Socket.EventEmitter.emit (domain.js:467:12),
at emitErrorCloseNT (internal/streams/destroy.js:74:3),
at processTicksAndRejections (internal/process/task_queues.js:80:21),
(node:27) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see Command-line options | Node.js v16.3.0 Documentation). (rejection id: 10)

Or are you saying I need to add the "outline-internal external: true" to the bottom to my already running redis instance that's being used for Authelia which is below?

  outline-internal:
    external: true

 

[one-eyed-king]

Did you create the network outline-internal manually (as in with Portainer, the Syno Docker-UI or on the command line)?

If you'd stop posting images/screenshots and instead start posting the content in </> code blocks, I'd be able to copy/paste its content and enrich what needs to be done