One month with TrueNAS Scale (now in RC-1 stage) and Kubernetes

[jeyare]

Tl;Dr : This is a long shot for those who are interested in learning more.
If you want to use TrueNAS SCALE for purposes such as Minecraft server, this article is not for you.
Note:
TrueNAS Core is a successful running product - based on FreeBSD, which is diff. from the TrueNAS SCALE (in RC-11 stage now).
This reading will help you save time when you will decide to test something new.
--------------------------------

I want to share my monthly experiences from TrueNAS SCALE on 22.02-RC.1-2 running in one of my VM nodes. For better segmentation of my thoughts:
My primary intent is to check the Scale platform to reuse its capabilities in SME/SMB segment in companies up to 100 employees. To understand me correctly, this includes small teams of 10 people for specific purposes (not a single holy grail) as a replacement for the existing Synology platform (my own and also for my customers).
My secondary intent: to check this platform as a replacement for my mixed home environment based on Synology and TrueNAS CORE (TNC). Each of them had its advantages because neither of them is perfect. But that's life. Nothing is perfect.


First – TrueNAS “honeypot” for people like me:

1. Pure Debian environment (5.10.70 kernel in RC-1) vs FreeBSD in both mentioned platforms (Syno or TNC).

2. Virtualization based on both Containers and VM in a single node or across multiple nodes (up to setup).

3. Enterprise Support. Yes, nothing is for free, and it is welcome for people like me to get in touch with another level of support than for SoHo. Of course, like many of you, I grew up in public communities, which I also actively use. Sometimes I feel that they know more than 2nd level NAS vendor support in those communities, especially if it is a downloaded package, which is part of the system but is not directly from the NAS vendor.

From any point of view, the SCALE product is more oriented to the SME / SMB segment than to SoHo. We will probably agree on that. This does not mean that people cannot use such a product at home. However, the primary goal is essential. And I hope that such a thing was also defined when creating the SCALE.

4. OpenZFS and Gluster technologies for hardening of the stable storage operation. Which seems to be a more suitable solution compared to LVM + BTRFS in Synology ecosystems.

5. Freedom to decide which HW to use for a particular environment based on the requirements of the environment itself and not the restrictions of the NAS system vendor. Over the years, it has been confirmed that freedom in this decision-making is more valuable than the comfort of one support centre for everything that still does not work well enough in an SME). And especially in today's world, where HW requirements change every year, and new available technologies are shifting at missile speed, tying up one HW vendor is a burden rather than an advantage. Once again - I do not describe a large enterprise environment like banks and the like.
This freedom has just been lost to the owners of Synology enterprise NASes, where due to the no longer supported Facebook Flaxcache (unchanged since 2014) and the already mentioned LVM + BTRFS connection, it has reached the point that Synology has severely limited the use of non-Synology branded (firmware tuned) HDD, SSD ( aka Toshiba), Synology RAM modules, Synology NIC and Synology PCIe card for NVMe Cache. One would say that it is suitable for the enterprise = this right choice should guarantee all. But, to be honest, it doesn't work. And right at all. Especially if you understand the details of disk behaviour and that not every PCIe card is the same.
This is exactly all I discovered in the SCALE. Well, that's about all the positive parts of this consideration. Only disappointment follows.

Second – the Containers “honeypot” was massively promoted by iX.

From the official SCALE web: TrueNAS SCALE provides simple access to the well-established Linux container ecosystem and makes application deployment easy. With support for KVM virtual machines, Kubernetes, and Docker containers, it’s easy to customize and add applications to suit a wide variety of needs.

Reality:

iX deployed Kubernetes as the primary containers orchestrator only. Docker (swarm, compose) does not officially support at all, and it is literally up to the user to adapt the system himself to use another orchestration (Docker-based). What will be valid until the next upgrade of the SCALE (there is a workaround already, described below).

The SCALE includes the ability to run Docker containers using Kubernetes (written in iX web, Last Modified 2021-04-02). Ok.
To be more consistent, iX used a simplified modified version of K8S in the form of K3S. And here comes the first problem. K3S itself is a platform initially defined as a Lightweight Kubernetes orchestrator designed for production workloads in unattended, resource-constrained, remote locations or inside IoT appliances.
How it manifests itself in the SCALE practice:

1. It is not possible to use native commands created by Ranchers. E.g. try to use <kubectl>; you must search the Internet for why it doesn't recognize this command. Someone iX decided to use <k3s kubectl>. Yes, ta workaround is there (an alias) - but in RC-1??
2. It is not possible to use a network other than the "host network", which is a significant issue regarding the security or operation of segmented networks. Even for SoHo users who understand what this causes is a problem. Not to mention SME / SMB.
3. Kubernetes Administration / Monitoring. This is an entirely unmanaged part of the SCALE product. It does not support native elements from Ranchers such as Dashboard. So, for now, you can forget that you will orchestrate your containers through a complex environment such as Dashboard. You don't even have to try deployment for Dashboard. I tried it - it doesn't work. This was also confirmed by a TrueCharts representative on the TN-forum. This is a failure for SMEs / SMBs, even for SoHo power users.
4. SCALE APPs. As I learned on the TN-forum and not on the official iX website, this is a "glue" created by TrueChart group (not part of iX) based on the simple deployment of freely available containers from the Docker hub environment, the purpose of which is "touch the button and play ”. In translation, this is a K3S deployment of existing Docker containers to be installed even for a novice in the world of virtualization which does not understand how to deploy containers and does not want to waste time with it. But that's still the definition of a SoHo user. Representative TrueCharts explained in TN-forum that it has added value for the user in case of update, rollback of the container. It is said that it is possible to request TrueChart support directly. What support? Binding container to K3S? In such setup of K3S within the SCALE? Are you serious? It has nothing to do with SME / SMB. To be sure, the SCALE is in RC-1 stage and not in Beta.
5. Heavy security issue with the SCALE APPs. For the reasons described in point 4, K3S deployments are set up so that only the user selects the appropriate APP from the catalogue, and all predefined settings have already been made for him in TrueChart. To make you understand - including setting up root / psw, usr / psw to a database which is a part of full-stack containers under single "APP name". This is an unacceptable and utterly wrong attitude. Even for the SoHo segment, it's across the line. This reality has also been confirmed by the TrueCharts representative on the TN-forum here.
6. There is no freedom to choose containers. You are strictly dependent on what SCALE APPS, TrueChart source, provides. An example - try to install a monitoring stack defined by Telegraf+InfluxDB+Grafana. First, you can't find the Telegraf or Grafana in the SCALE APP library. When you do not want to use the GUI - part of the SCALE APPs, you should turn it off. It's unworthy of the other quality features that the SCALE contains. Unfortunately, this is another example of how a good idea in the wrong hands ends tragically.

The SCALE APPs catalogue is available here:
App Request List - TrueCharts
Compare that to what's available on the Docker Hub.

Unless you're tired of reading, we're reaching our goal of using the Docker regularly and more securely through the management and monitoring tools, as we do in 2021.
First, you need docker daemon to be active, which is part of SCALE( but only passively). The reasons have been described above. Then, the following procedure will help you:
Using Docker on TrueNAS SCALE (no Kubernetes)

Secondly, you need to create a secure TLS docker node in SCALE for Portainer which is able to manage directly Docker containers (swarm, compose) and Kubernetes:
Portainer: Managing Docker Engines remotely over TCP socket (TLS)
Of course, there are a few opponents of this method, so I'm happy to see their suggestions for comprehensive container lifecycle management in SME/SMB especially in this stage of the SCALE.

Done.
And this could have been done on the iX side a long time ago. Easy choice for the user:
- Unusable ecosystem in the form of K3S SCALE APPS from TrueCharts
- Or something that really works, including setting up your own root pswds, network setup / ENV options, Nginx Reverse proxy, ...

I still need to play with the docker socket to make everything work 100%, including access to the container console from Portainer. I have already run my first containers on TrueNAS SCALE. However, as I wrote in the introduction, I will watch the SCALE heading because the RC-1 is dedicated to a storage target only rather than the central host for container virtualization. It is still a long way from this goal, if at all. So what is the missed potential of this project. But I'm not the iX shareholder, so I can't comment otherwise.

Nothing is perfect.

Cheers

 

[Coop777]

Did you build your own hardware or purchase a prebuilt TrueNAS box?

 

[jeyare]

TrueNas Core is running on my own old PC, till the time I will find a final decision.

Scale is running in VM.

 

[jeyare]

For the TrueNas Core or Scale you don’t need a special HW. It is running also on each VM platform also even in Oracle Virtualbox. There is no need create a bootloaders like for the sh.ty Xpendology. Because it is official product.

The installation is really easy for every from the Linux part of the world (no need to be an iron man). You need read more about ZFS/Zvols pooling to achieve best config, then you can “slice” the final “datasets” for each (independent) purpose. When you understand principles you can’t lose a way. Then it is better to train it in snapshoted VM environment to avoid time or cost loses.

What is great, that they Jira is publicly open = you can read all tickets and statuses = something what Synology doesn’t understand. I have few tickets there re Docker integration in the Scale.

TrueNAS - Issues - iXsystems Jira

jira.ixsystems.com

-- post merged: 3. Dec 2021 --

what is great from the TNScale point of view is fully running Docker Swarm, what is really missing in Syno “proprietary” docker engine.
But it is a time to learn more about the K3S.
@one-eyed-king … what is your experience?

 

[one-eyed-king]

I am afraid you have to be a little bit more specific about what experience?

In my job everything takes place in AWS Where ever possible I stick to managed services, as they simpley tend to work. For instance the managed kuberentes solution EKS, provides a high available control plane for a fixed costs around 72EUR/month. Additionaly you pay the price of the worker nodes you need to run your payload, which of course depends on the instance type. I

Regardings ks3: I expect it to provide the same set of configuration files that could be used with kubectl on remote systems like every "grown up" kubernetes distribution des. For time beeing many key-turn distributions still depend on docker, even though kubernetes deprecated the docker shim. If I'd install kubernetes today, I would use a solution that depends on cri-o or containerd directly - there is no benefit in introducing deprecated dependencies that need to be migrated in the near future. We never install kubectl on the management or worker nodes.

I was planning to replace my homelab kubeadm cluster (that depends on docker) with RKE2 on top of containerd. In my homelab I use portworx for distributed storage, which basicly turns each worker node into a node of a portworx storage cluster. Portworx takes care of volume replication and bringing the volume to the node your pods are running.

Depending on the payloadS swarm might be perfectly sufficient (portworx works with swarm as well btw) and even be preferable with a smaller team - it will be easier to teach to new employers. Though, in my experience all mayor enterprises and government entities (at least in germany) silently agreed on kubernetes beeing the only accapted orchestrator when it commes to containers.

 

[one-eyed-king]

There is an ansible role that should help to easily bootsrap a RKE2 cluster: GitHub - lablabs/ansible-role-rke2: Ansible Role to install RKE2 Kubernetes.
It just requires the inventory to define the master and worker nodes, and some further declaration (like a vip for the k8s-api) that typicaly end up beeing part of the groupvars. As a result you get a pre-installed RKE2 cluster with as many manager and worker nodes as you want. Of course it will create a kubeconf that can be used with kubectl to controll the cluster from where ever the api-vip is reachable.

That's what I am going to setup in my homelab during christmas and NYE.

 

[jeyare]

Today I finished last part of the test:
- I can run Swarm and K3S in single host as “container virtualisation centre”
- I can manage both of them by Portainer
- as host not primary exposed to web (behind load balancer, reverse proxy and vpn)
- more demanding on k3s
- microservices on docker
This makes sense for me in SME/SMB.

I have also in operation PowerBI gateway and Azure B2B for expose PowerBI Apps to customers. But all the main ELT, data logic, analytics, interpretation on my environment.

Services, where is more flexible to use AWS will run there.

 

[one-eyed-king]

- more demanding on k3s

I assume you mean that the ressource consumption with k3s is higher than with plain docker or swarm?

Indeed it is, while the docker swarm control plane doesn't add much load, the kubernetes core services do - especialy on the manager nodes. That's the price to pay if you want/need fine grained control (permissions/operation scenarions) of your container zoo.

 

[jeyare]

more demanding services on k3s - it was the missing part of my consideration
but your point is ofc correct

 

[one-eyed-king]

I must admit, If only I wouldn't have recently migrated my homelab from ESXi to Proxmox PVE, I might have considered TrueNAS Scale.

Both, PVE and Scale, base on Debian 11, use KVM, provide ZFS filesystems + CEPH. Though, the UI of Scale is much nicer and they use libvirt as api to drive KVM. PVE has build-in container support, but only for vm-like LXC ontainers - it lacks native support for Docker, K?s or even an "appstore".

KVM vm's stored on Ceph can be quickly migrated to another PVE box, if the node where it's running goes down. I assume Scale provides something similar? The only drawback with Ceph is, that you will want to at least use10gpbs connections or even better use 40gbps or higher. It seems the hight bandwith's are operated on higher frequencies, which result in a smaller latency.

 

[jeyare]

with the Scale they create a marriage between pure enterprise NAS storage, which is from any principle intended for use out of internet “exposed” environment and VM host with all the current needs (from the basic block to object) and containers hub.

10g interface for such purpose is must and not costly in comparison of the gains. The advantage of this concept is that no one limits your HW infrastructure elements, and you don’t need to wait for PCIe 4.0 with a sufficient number of free 8 lanes.

So, from today I now have mixed NASes farm (Syno, TrueNAS) and mixed containers operation within the farm (docker, swarm, kubernetes), operated from a single point of management - Portainer:

I already sent the guide to TrueNAS jira to prepare it as a standardized solution for the next release. They need just create portainer Pod within the default node (what is a few lines of code in bash script).


Useful for ALL Syno disgusted power users to use benefits Syno platform (I have still just one - SynoDrive, till the time when I will finish test of NextCloud) and TrueNas platform in one powerful environment.


Then need to test a few migration scenarios of the containers between the platforms. And learn more in k3s.

 

[WST16]

disgusted power users

I might consider myself a moderate power user that is not yet fully disgusted (I’d say moderately disgusted with the drives lock-in move).

Are you at the stage of a fully disgusted power user? You know that there’s no going back at that stage.

 

[jeyare]

Need I explain this topic better?

After seeing the fully functional version of the DSM7, I understand that something is wrong. Two years of promising a revolutionary new system have put in mind what is happening. I've written enough about it here. I just understood that comprehensive vision is more appropriate than tunnel vision. I'm not angry with Synology. It is their free choice where they have decided to go. I'm not one of the people who change their cell phone every year when they see a new one. Last month I exchanged my iPhone 6s for 12 after six years. It was already slow that I accepted it.
Similarly to Synology. After more than ten years and currently seven middle-class NASes, I have decided to follow the path of greater freedom (HW) and much broader possibilities (SW) to get a +2 level higher for a better cost.

Nothing is perfect. Me neither.
@SynoMan knows me from all of you much better. I will help him to support this forum with my ideas. This is not a swan song.

 

[WST16]

I'm not running a Fortune 500 company or anything like that. I use them for a few of our business projects and of course at home. So far I'm amazed by what I can do with them, especially with the addition of Docker.

I'd say that these devices are really good if you understand (and accept) the limitations. However, one thing that they’re extremely good at without a doubt is pushing –the willing– to learn. They're able to do so many things. To me whenever I try to justify adding a new one to the growing collection, this is the thing I reflect on, the knowledge I gained about so many technologies meshed together in an easy to use box. That's when I say, it's priceless.

On the flip side, if one goes looking for a breaking point then one will find it (in anything or anyone in your life). I'm not saying that you're doing this, of course you might have your own reasons (business or personal) as a very advanced user that are beyond the capability of these devices. Could be value related too as you said, and for you to try something else is part of the journey.

It's one's reasons and choice at the end

 

[jeyare]

you can’t pour diesel to your petrol engine
you can’t expect, that your Prius will survive in rough environment
you can use it for the defined purpose, what is absolutely ok

Because all valuable services I’m running on containers or VMs the migration is quite optimistic. No hurry. I have time and patience. Just be ready when tomorrow one of my big NASes will die I know where I will put my money. Step ahead.

 

[jeyare]

A few details illustrate my point of view.

iXsystems is a company that has several business lines:
- Storage (Storage system SW + NAS HW)
- Custom build servers
- Support.

The revenue stream comes from the sale of NAS HW, servers and support. SW storage system is free.

I haven't found guaranteed accurate Revenue data anywhere, but it's around $50-70M USD. So, for simplified comparison, $70M / $ 300 (per avg Syno NAS) = 233k such NASes supplied annually worldwide.

However, the average price of iXsystems appliances is several times higher. So if the support from their revenue stream is 30%, which would be about $21M, then HW sales would be $49M. This would mean that their annual sales at $2000 / appliance would be around 24.5k units.

Why do I count it all? An experienced person will find an anomaly in these numbers - how can an excellent Dev ecosystem (R&D, architecture, dev, test, support) pay from this revenue? Realise that if you have your own development of two completely different storage system platforms, such as Core and Scale, you need a comprehensive Dev ecosystem (R&D, architecture, dev, test, support) for each SW product line.

That's a minimum of $20M a year (and I'm very reticent in these estimates). So HW sales of $49M with a profitability of 50% = $25M. So they have max. $5M per year = $417k per month. So, so.

However, suppose their total revenue is at the level of $ 50M only, then according to the above high-level procedure, they will have $17.5M left on the OPEX of the entire company, including the essential = 2x Dev team of their main Storage system products. Therefore, they must have reduced capacities, reduced quality capacities, or all together.

The Core itself has been an excellent system for years. This is true, and I have to confirm it. However, with the Scale product, the rule was confirmed that without sufficient preparation and definition of the goal of the new product, the whole development is one big trouble. If you look at the TrueNAS forum, it will confirm that it is not going in the direction it was announced on the TrueNAS website.

And one more experience in terms of support from TrueNAS (Scale):
- you can use a public forum, which, however, as the representatives of iXsystems wrote, is not under the supervision of people from the Dev team.

- However, a representative of the Product line (VP in (iXsystems) works daily on this public forum. Nowadays, every small company also has x-VP positions (it sells better).

- They still have a parallel Discord channel, where you find the same people. That's really great.

- However, the problem with the public forum is that they immediately lock the thread for an in-depth analysis of the state of deployment = only praise is expected.

- The people from iXsystems themselves recommend using their Jira to define the ticket (improvements, discovered shortcomings, bugs, ...). Another problem is that there are many tickets and few people on the support side. As some of you know from the practice, the average time of an open ticket is evaluated. Therefore they close the tickets = because "we will not spoil success statistics". Logically - ticketing for Core and Scale is free, but it costs iXsystems (people, systems behind).

So this is my experience. But it's my point of view. If I were a regular NAS user who only needs simple storage, including a Plex server, to enjoy such an appliance, I would love Scale right now. No doubts.

I have prepared this post for those who expect immediate salvation in the TrueNAS Scale solution. Unfortunately, the car is still not finished. And if you've read emphatically, it doesn't look like an endless source of resources for a sophisticated way forward. Patience.

 

[jeyare]

the mentioned Discord channel: TrueNAS Community

 

[jeyare]

And just imagine how iXsystems VP for the TrueNAS product announced the last version of the Scale (22.02.1 aka release version) on May 5th, 2022:

link:

TrueNAS SCALE 22.02.1

TrueNAS SCALE 22.02.0 (“Angelfish”) was released on “Twosday”, 2/22/22 and now gets its first major update after being deployed on over 16,000 active systems.

www.truenas.com

with a single problem in reality:
- HELM and Docker Compose aren't supported
There is just TrueCharts APP choice within the Scale described in my previous post - suitable for newbies (Plug & Play), not suitable for power users (Custom parameters). (TrueCharts is an independent project from iXsystems and has just an integration to the Scale).
And to be sure - the Scale product is defined primarily for enterprise usage, not for home/Plex users.

Here is a link to iXsystems JIRA:

[NAS-115010] Disable the docker-compose binary - iX - Bug Tracker (Jira)

jira.ixsystems.com

which proves that Docker Compose was DISABLED in the Scale

and here is the statement of the TrueCharts representative:

which denied the support of the Docker Compose

When the left hand doesn't know what the right hand is doing ....

BTW, you can sometimes upgrade to the new version of Scale with a lot of luck. But unfortunately, the condition with many error messages is much more common.

This leads me to confirm my reasoning that there is no such thing as a clearly described scope - for whom we actually do it and why. A very common situation where a group of people tell each other that "we will code this within two days".

 

[WST16]

Thank you for your insight @jeyare.

 

[jeyare]

Thanks, @WST16
I feel responsible for suggesting to people here that the TrueNAS SCALE is the quick way out of the Synology heading. Sometimes we need to correct our mistakes. People should help each other. Any false notions that our data will save by any vendors are wrong. And it doesn't matter if there is a label like Syno, Qnap, TrueNAS, ...

I have already been rewarded twice for my open opinions on the TrueNAS Scale forum (Nov2021). I started two threads where I explained that they didn't use that excellent knowledge of TrueNAS Core and went in exactly the same directions as Synology. So both of these threads were locked, and one of them is available just for authorised users (the link is inactive for ordinary users) = I understand why, I described the complete instructions as proof that I can use k3s and Swarm on one Scale appliance - each for its own specific advantage. Both are managed through Portainer, which is free and does not need to waste time and cost to create by iXsystems Devs again with a new label.

However, this does not change my willingness to be patient and wait for more users who contribute to the TrueNAS forum with the dissatisfaction with the state of containerisation and lock on a single vendor only who can deliver containers in the pre-prepared TrueCharts APP. Without Helm support. ...That's crazy = a single point of failure of the entire containerisation.