Recent by Daniel Nashed
Domino CertMgr GitHub Repository with additional material 
By Daniel Nashed | 9/27/23 1:21 AM | Infrastructure - Notes / Domino | Added by Oliver Busse
Documentation is always a challenge. This is specially true when it comes to complex topics like SSL/TLS certificates.
Many admins still use their old cook books to get certificates created.
When HCL introduced CertMgr in Domino 12.0 the team asked for feedback in the early code drops.
And the team is keeping asking in public and private forums since then.
We really need your help to get it right. We need detailed feedback and questions.
My new plan is to turn questions into FAQs and Howto documents in this GitHub repository.
Who moved my Domino keyfile.kyr files?
By Daniel Nashed | 9/25/23 2:00 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
Domino security in 2023
Domino 12.0 introduced a new, standards based and open way to work with web server certificates.
Instead of using command-line tools like OpenSSL and the Domino kyrtool you can now manage all web server certificates in a domain wide certstore.nsf.
The new functionality based on the well known text based PEM standard for certificates provides simplified flows and automation options for all type of certificates.
Domino 12 also introduces the more modern ECDSA (sometimes referred as ECC) keys/certificates which are based on elliptic crypto which has dramatically less overhead.
Moving from keyfile.kyr to certstore.nsf
The legacy kyr files can be automatically imported into certstore.nsf with a single command-line operation (load certmgr -importkyr all).
Domino V14 backup for notes.ini
By Daniel Nashed | 6/1/23 2:03 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
Domino backup is around since 12.0 and it got improvements in every release.
There are not many current AHA ideas for Domino Backup & Restore.
One smaller feature you can see in EAP1 is the backup of the notes.ini.
How to use Domino OTS on Kubernetes to import an existing TLS Certificate
By Daniel Nashed | 5/30/23 12:05 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
Domino One Touch Setup has been designed with flexibility in mind, with special focus on getting a server up in a secure way.
On Docker you can just mount PEM files into the container. On Kubernetes TLS Certificates and Keys are stored in secrets.
Personally I am not a big fan of storing PEM files on disk. But you could at least set a password on the PEM file you import.
Here is a basic example how to create a secret on K8s and reference it in OTS.
Even the simple environment variable setup supports the security settings for CertMgr.
Of course the same functionality is also available with the more flexible JSON based configuration.
Importing trusted MicroCA Roots for a Nomad Lab environment
By Daniel Nashed | 5/29/23 12:39 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
Yesterday I worked on a lab configuration based on Windows Sandbox, Domino and Nomad Web.
The biggest challenge is to have a trusted certificate for Nomad Web.
Nomad Server running with the Micro CA
A Nomad Server can use Domino CertMgr Micro CA Certs. But the root is not trusted in your browser.
I took a closer look and came up with a simple solution. which makes the import dramatically easier.
No more searching for the right trust store and handling PEM files manually.
Get prepared for Notes/Domino V14 Early Access Code Drop 1
By Daniel Nashed | 5/26/23 9:32 AM | Business - Events / People | Added by Oliver Busse
You can get hands on experience with Notes/Domino V14 end of this month. Here are some tips to get prepared.
All of the software is only intended for non-production use! So you should prepare a VM to get started.
But you should really take a look and have a try. Specially for business partners this is a call for action to test their applications with the updated back-end components.
As announced earlier Notes/Domino moved to up to date compilers and a newer Java version. Also the client is 64bit only.
I hope to see many of you in the EAP forum or at DNUG conference face to face.
Tuning Domino Servers for TLS sessions
By Daniel Nashed | 5/24/23 2:13 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
My previous post was mainly about HTTP traffic and I mentioned TLS/SSL don't use the maximum number of connections settings, because they have a SSL/TLS session.
Establishing a new TLS session has significant overhead! And you have to make sure in any application, that those sessions are cached and resumed.
I revisited a blog post from 2012 where I explained a fix, which went into 8.5.3. And was enabled in 8.5.4 by default (which turned into the 9.0 release when shipped as far I recall).
There was an issue with the session cache and a new cache had been implemented in 8.5.3. Today the new cache is the default and SSL_USE_ADDSESSION2=1 does not exist any more.
Debugging program crashes with gdb on Linux
By Daniel Nashed | 4/10/23 9:37 AM | Development - Notes / Domino | Added by Roberto Boccadoro
This bugged me for a while because I had no idea what was happening.
One of my OpenSSL-based tools in C crashed once in a while.
I was only able to find out once wrote my own small tool to check the server listener of my other application.
The crash happened very intermittently in different places when I opened and closed the connection very quickly.
Adding a delay of 1 ms stopped the crash. But if you are running a service on the internet with port scanners around, you better find out in detail.
It turned out to be the SIGPIPE event causing my program to terminate.
But the interesting part is how I found out about the crash.
TLS/SSL Cipher Troubleshooting
By Daniel Nashed | 4/10/23 9:35 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
Every Domino release adds more TLS ciphers to the weak list to ensure poper security. We can expect the next versions also to have less ciphers available.
Domino ensures for clients and servers, that the list of ciphers provided is safe. In addition the default behavior is that the server decides the order of ciphers to pick.
And only allows secure renegotiation to prevent the client to pick a less secure cipher.
Usually this doesn't cause a lot of trouble for inbound connections. Modern browsers support modern ciphers. But outgoing connections for LDAP and ICAP could be a challenge.
I had to look into an ICAP connection problem this week. To demonstrate how the TLS handshake works, I wrote a small OpenSSL demo program in C. This turned into a quite flexible troubleshooting tool over the weekend.
Picking the right Linux Distribution
By Daniel Nashed | 4/5/23 10:42 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
There isn't one best or right distribution in general. There are many variations. For example Redhat/CentOS based platforms have flavors like Rocky and Alma Linux.
The HCL Domino community project looks into many possible combinations as the base image --> https://opensource.hcltechsw.com/domino-container/concept_environments/
There are basically three different main flavors with different toppings:
- Redhat/CentOS based (with yum and dnf in later versions to manage packages)
- Ubuntu/Debian (with apt to manage packages)
- SUSE Enterprise/Leap etc. (with zypper to manage packages)
The right distribution is really depending on your needs.
Nomad Web server connection options
By Daniel Nashed | 1/25/23 7:30 AM | Infrastructure - Notes / Domino | Added by Oliver Busse
Nomad Web is a modern HCL client offering in form of a Progressive Web Application (PWA) running in your web browser.
In addition to Windows or Mac, it also works on Ubuntu and other Linux distributions! So there is finally a client offering for Linux clients again!
The Nomad Web application is installed on a server providing the required files for download.
Those files can be stored on a SafeLinx or Domino/Nomad Web server.
Windows Sandbox - A feature you should know
By Daniel Nashed | 1/23/23 2:20 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
The sandbox can be a very useful tool for many different situations.
I am often using it for Domino server or client install tests.
But there are many other use cases including training environments etc.
It's a full throw away sandbox environment recreated every time you start it.
The only limitation is that you can't reboot the Windows for example after a software update.
But even installing the Windows re-distributable run-time package does not require a boot.
Most applications like Notes/Domino install it on their own.
I needed it to test my own applications. But there is an easy way to download and silent install it:
NGINX TCP Stream with SNI support. More than helpful for lab environments
By Daniel Nashed | 1/23/23 2:15 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
In production you usually want centralized certificate handling and off-loading TLS termination to a load-balancer.
I posted scripts to have NGINX realod certs automatically from Domino CertMgr via HTTPS to leverage Domino's Let's Encrypt implementation.
But sometimes you really want all your servers directly exposed over TLS.
For example in a lab environment with limited resources and only one IP, you might want to still have each of the hosts expose their services on their own.
How to get the error message for a Notes error code
By Daniel Nashed | 12/13/22 1:34 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
Sometimes scripts or Domino server commands only return an error code and you would like to know the error message.
There is an easy way to get the error message back from a server command.
"show message [module]
In most cases you don't need server tasks specific error messages and just use the decimal error code.
Restic – Command Line Tool supporting Windows VSS
By Daniel Nashed | 11/24/22 2:03 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
Now that Domino 12.0.2 is has a native VSS Writer, we can look into new interesting integrations.
In my session at SUTOL conference this week, I showed a first version of a Restic integration for Domino 12.0.2 via VSS.
Restic is a very interesting application (https://restic.net)
It's a single binary written in GO
And uses a approach like Borg Backup uses. But in contrast to Borg Backup it has full Windows support.
This includes VSS Writer + AutoRecovery support!
It is Open Source, efficient, flexible & secure.
And very simple to setup & use!
Docker cp with permissions and owner change
By Daniel Nashed | 10/11/22 2:22 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
By the default the ownership of a file copied into a running container is always root:root.
Depending on how you want to use the copied file, this ownership isn't what you want.
You can't pass user/group or permissions to the docker cp command.
Changing the owner or mode would need root permissions inside the container.
Containers usually run with an unprivileged application user. For Domino this is notes:notes with the IDs 1000:1000.
Domino 12.0.2 One Touch setup with Let’s Encrypt certificates
By Daniel Nashed | 9/15/22 3:32 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
One Touch setup (OTS) is quite powerful tool. But sometimes you need to know exactly what happens and combine functionality to make best use of it.
With Domino 12.0.2 OTS creates certstore.nsf automatically and you can let it create a MicroCA for you.
But what if you want to use a Let's Encrypt certificate instead?
There is a quite simple way to just find and update the existing document with a appConfiguration.
And if you specify notes.ini CertMgr_ACCEPT_TOU=1 the ACME account license agreement will be automatically accepted (already part of 12.0.0).
Download certificate chain without OpenSSL
By Daniel Nashed | 9/12/22 3:17 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
Usually OpenSSL is the tool of choice for all type of certificate operations.
But what if no OpenSSL command line is available? Like in a Domino container where you can't install software?
After some research, I came up with the keytool, which is part of the JVM Domino ships.
K3s, Podman and a registry
By Daniel Nashed | 9/5/22 2:03 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
Rancher Desktop is a great all-in-one desktop environment.
When running it with the Docker back-end you have all in one environment for development and run-time.
For a server, K3s (https://k3s.io) is my platform of choice. It is production ready and easy to deploy.
For Kubernetes, you always need a registry to pull images.
As soon you need custom images, you will need a registry to upload and download your image.
K3s allows you to configure private registries.
You could use any registry. I am just running the registry Docker image on Podman in my environment.
Domino Community Image - New Nomad Server install option
By Daniel Nashed | 7/30/22 10:22 AM | Infrastructure - Notes / Domino | Added by Oliver Busse
The Nomad server is a new offering to directly add Nomad support to your Domino server instead of using a SafeLinx server.
Recently I added a SafeLinx container to the Domino community project.
Now I am adding the Nomad Server to the Domino image as a new build options.
Domino Container automation testing
By Daniel Nashed | 7/25/22 12:20 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
Containers are not only a good way to run Domino. It is also the perfect environment for automation testing.
Domino 12 introduced OneTouch Setup to automate deployments, which also lets you create reproducible Domino test server scenarios.
As a starting point I am building an automation test for the Domino image itself.
So in future for every commit on the Git repository I can run automation to ensure the image works.
The test automation can be used in your own environment as well after an image built in your environment.
It can be also be extended for your own application testing.
Customizing Domino Backup mail notifications
By Daniel Nashed | 7/22/22 6:42 PM | Infrastructure - Notes / Domino | Added by Oliver Busse
Domino Backup offers to send e-mails depending on the status of your backup.
By default you are getting an e-mail in case of error or warning.
I am rarely getting error messages from my servers.
In this case here I updated my server to a new kernel and ZFS drivers failed to build.
Nomad Server 12.0.2 on Linux just works
By Daniel Nashed | 7/20/22 10:32 PM | Infrastructure - Notes / Domino | Added by Oliver Busse
The Nomad Server is a small component, you install on your Domino server.
And it is bundled with the Nomad Web files. So it is a all-in one server add-on solution.
The installation sounds more complicated then it is.
It's really simple to install. And I am thinking about making it an install option for the Domino community container image.
Why run Domino in a container today
By Daniel Nashed | 7/20/22 2:03 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
As my of you know, I am a big fan of running Domino and other applications in a container.
This can be a classical Docker/Podman deployment or K8s.
Containers might not be good for everyone. But a lot of software is available in a "Docker image", which can run in multiple environments.
Domino's main deployment model will not change to Docker.
Domino 12 Restore point in time
By Daniel Nashed | 7/15/22 1:23 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
Domino Backup can restore databases point in time!
For other backup applications this functionality is usually only available with archive transaction log.
But with circular translog or linear translog mode, the most current backup should have all the translogs available to recover point in time as well.
Domino Restore allows you to restore point in time in that case.
Even it is not guaranteed that the translogs are still there, this can be still a good configuration if you have servers without dramatic load.
Linux shell scripts: Difference between "set" and "env" -- fixed the Domino start script
By Daniel Nashed | 7/11/22 4:28 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
The Domino start script has always been using "set" to list the environment variables, before running the sever.
This information can be important to understand the environment passed to your Domino server at start-up.
It turns out that there had been a change over time, which causes much more information to be listed, then just the environment variables.
Domino ZFS Snapshot Backup
By Daniel Nashed | 6/20/22 2:05 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
ZFS is one of my favorite file-systems. And I posted before about using it as a backup target.
The integration is pretty simple with Domino backup, because it is a simple file backup.
Now that we have the new VSS Writer for Domino 12.0.2 on Windows, it is time to look into ZFS snapshots.
SafeLinx Nomad Server Community project?
By Daniel Nashed | 6/13/22 1:41 AM | Infrastructure - SafeLinx | Added by Oliver Busse
Wouldn't it be cool to have a SafeLinx Docker image with Nomad Web included with auto configuration?
Maybe having a docker-compose.yml with just some basic parameters to get SafeLinx and Nomad up and running?
Recovering a lost Domino server notes.ini quickly
By Daniel Nashed | 5/26/22 1:39 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
On Linux by default the notes.ini is in the data directory.
On Windows it is per default in the binary directory. You could move it to the data directory, which would make sense from backup point of view in many cases anyway.
But what if you have it in the program directory and install a new major version where you get rid of all your binaries as a best practice?
SHA512 is faster then SHA256
By Daniel Nashed | 5/13/22 6:12 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
Quite interesting results..
I have been looking into different hash algorithms to see the overhead today.
It turns out that SHA256 is the slowest option and SHA1 is the winner.
But it is interesting, that SHA384/SHA512 are also faster then SHA256