Technical

Everything in the same file apiVersion: v1 kind: Pod metadata: name: redispod spec: containers: - image: redis name: redisdb volumeMounts: - name: redis-data mountPath: /data ports: - containerPort: 6379 protocol: TCP volumes: - name: redis-data persistentVolumeClaim: claimName: redisdb-pvc --- apiVersion: v1 kind: PersistentVolume metadata: name: redis-pv spec: storageClassName: "" capacity: storage: 1Gi accessModes: - ReadWriteOnce hostPath: path: "/mnt/data" --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: redisdb-pvc spec: storageClassName: "" accessModes: - ReadWriteOnce resources: requests: storage: 1Gi If you deploy you will get something like

Learning a new syntax and the libraries is always a task, and the only way forward is gradual. In this blog I will try to create a small program to read a text file and will keep updating the program to include more sophistication

Must Read # https://blog.golang.org/laws-of-reflection https://golang.org/pkg/reflect/ https://golang.org/pkg/reflect/#Kind https://golang.org/pkg/reflect/#Value Package reflect implements run-time reflection, allowing a program to manipulate objects with arbitrary types. Reflection is a way to examine own structure. Let’s try and learn it via examples.

Interfaces are an interesting concept to learn in GOLANG. There are a tonne of informative articles and here is my contribution to it. In classical Object Oriented everything is treated in parlance of Object or Class like in Java or C++. In Go there is no class or inheritance.

Reference # https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/ https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/#example-use-cases Before we understand the tainting we should know about the Node Affinity. It the by the virtue of Affinity that pods attracts nodes as a preference or as a basic requirement.

A cheat sheet for Installing K8S cluster. Step1: # Check your OS? cat /etc/issue lsb_release -a Step 2: # Add the GPG key as described here

If you have worked with big projects you must have encountered pom.xml the central configuration for your projects that helps you build, test, and define your CI pipeline. There are other options that are available too, but that is for some other day.

Time to re-test how to create deployments. It has been sometime, I created some deployments so just checking on how much I remember. The quick way to always get started is to check the HELP.

In one of recent discussions with Mr. X I was asked about my approach to solving the Linux cookie problem A random quote Repository will be a text file Each quote will be segregated by a '%' Can be multiline. I always believe in incremental steps so here is what I could think of

Prime number is an age old problem. I am trying to solve it via streams. class PrimeCalculator { public int nth(int nth) { if (nth <= 0) throw new IllegalArgumentException(); // Will skip the nth - 1 // 2 to n-1 return IntStream.iterate(2, i -> i + 1).filter(PrimeCalculator::isPrime).skip(nth - 1).findFirst().getAsInt(); } public static boolean isPrime(int x) { // till half of x return IntStream.rangeClosed(2, (int) (x/2) ).dropWhile(i -> x % i != 0).findFirst().isEmpty(); } } Explanation # public int nth(int nth) { if (nth <= 0) throw new IllegalArgumentException(); // Will skip the nth - 1 // 2 to n-1 return IntStream.iterate(2, i -> i + 1).filter(PrimeCalculator::isPrime).skip(nth - 1).findFirst().getAsInt(); } The method nth(int) allows you to return the nth prime starting at 2, 3, 5....

Step by step: installing Docker, kubeadm, kubelet and kubectl on Ubuntu 18.04, initialising the control plane and joining worker nodes.

For my local K8S cluster of 4 nodes I wanted to get the resource usage metrics. From the official documentation of kubernetes.io These metrics can be accessed either directly by the user with the kubectl top command, or by a controller in the cluster, for example Horizontal Pod Autoscaler, to make decisions. https://kubernetes.io

In my K8S cluster when I was looking at component status I was observing an error. Please note componentstatus is deprecated in 1.19+ but the steps will still at least help out the OCD’s like me.

What good is an software engineer if he can’t keep revisiting the DS problems. It is like a pill for my laziness that keeps crawling ever so often. The problem I am talking about is mentioned in this cool portal exercism.io that I use almost on a weekly basis.

The utility package java.util.function has this functional interface Consumer. Each functional interface (Annotated with @FunctionalInterface) conceptually has a single method (abstract) called the functional method to which the lambda expression’s parameter and return types are matched against. Functional interfaces often represent abstract concepts like functions, actions, or predicates.

Java 8 streams: what map and flatMap do, with small examples of each.

Continuing from the last blog how about checking a pangram using streams. public class PangramChecker { private String allChars = "abcdefghijklmnopqrstuvwxyz"; public boolean isPangram(String input) { if (input.length() <= 0 || input.isBlank()){ return false; } return (allChars.chars().map(x -> (char) x).filter(x -> input.toLowerCase().indexOf(x) >= 0).count() == 26); } } All the for loops have been squeezed into a simple line

Stream is an new abstraction that lets one process data in a declarative way. A stream does not store data and, in that sense, is not a data structure. It also never modifies the underlying data source. The easiest way to learn a new API or a construct is by trying it. So here is an exercise that I tried to learn the Stream API. This is an exercise from exercism.io.

MAN Pages are one source of truth for all curious queries. man ps Name ps - report a snapshot of the current processes. It is used to display instant information about the active processes in the system.

The CPU operates in two modes in Linux User mode (non privileged mode) Kernel mode (System mode) When operating in Kernel mode it has un-restricted access to resources like memory and input output devices.

journald is used to query the systemd journal contents. Example 1 # Lets try and see the kubelet service logs in the systemd journal

Ping # I have used ping often but never to an extend that I could understand the complete range of options and power available at hand.

When two processes are executing concurrently, the exact order of execution of instructions is not deterministic. Without explicit synchronization instructions it is difficult to predict the outcome of these concurrent execution of process.

systemd is the replacement system for System V and with the primary goal of starting services in parallel. Remember init levels? GRUB: Grand Unified Boot Loader — /boot/grub/grub.conf Some key points

Helm is a dedicated package manager for Kubernetes. Although the Kubernetes objects are deployed in YAML’s there was a need to package and club them together and customise the deployment (not the K8S deployment).

OS version cat /etc/issue Ubuntu 18.04.2 LTS \n \l curl https://storage.googleapis.com/kubernetes-helm/helm-v2.12.3-linux-amd64.tar.gz > ./helm.tar.gz

A Kubernetes NetworkPolicy demo: deny all traffic in a namespace, then allow only labelled pods to reach nginx.

Every software cycle includes packaging; which essentially describes how the software written is packaged and shared. For the software built for Containers Helm is a popular choice of packaging the scripts.

I have been working on Kubernetes cluster and will soon plan to give a Kubernetes Certified Developer exam. In my quest, I will just record some commands that I found helpful or useful in this blog. It may be multi-part, as I will keep adding on a daily basis the commands that I will be learning.

POD # Smallest deployment unit you can manage in K8S, is a Pod. A pod may have one or more containers.

Always remember to refer the API and elements for the YAML structure API Overview — official K8S documentation The Kuberenetes documentation that has text descriptions and good explanatory examples.

A declarative way of maintaining the desired state. Continuing my example from the last blog --- # Pod deployment apiVersion: apps/v1 kind: Deployment metadata: name: go-server-deployment spec: replicas: 2 selector: matchLabels: app: my-go-server template: metadata: labels: app: my-go-server spec: containers: - name: go-server image: samarthya/epserver:2.0 resources: limits: memory: "256Mi" cpu: "1024m" env: - name: SERVERPORT value: "9090" ports: - containerPort: 9090 # Reference - https://kubernetes.io/docs/concepts/workloads/controllers/deployment/ In my deployment, I have two replicas configured, which can be increased to three using the command like the one below.

How can a pod communicate with the other pod? The answer to this question is through a network policy. Network Policy is a specification of how groups of pods are allowed to communicate with each other and other network endpoints.

In this blog we will be talking about Liveness, Readiness and Startup probes of a container. A Service in Kubernetes is an abstraction which defines a logical set of Pods and a policy by which to access them. — - Life without a service What is Kubelet? # Kubelet is a primary node agent that runs on every node. Works in term of PodSpec, that describes a POD (JSON or Yaml). Liveness Probe # It is used by kubelet to decide when to restart the container. periodSeconds defines when the kubelet check periodically. initialDelaySeconds specifies the initial wait before checking. I just created a single container pod that will run touch command to create a file and then it deletes that specific file after a delay, on which the livenessProbe is listening on and then once liveness fails we can check the container.

There are multiple Kubernetes objects we will be talking about in this blog. Mainly Services, Selectors and Labels. Service is an abstraction which defines a logical set of Pods and a policy by which to access them. A Service in Kubernetes is a REST object, similar to a Pod. Service tags the pods, logically grouping them using selectors. Pods have a lifecycle, they are born, they live, process and then they die. The pods (desired state) can be managed using deployment. A label is an identifying attribute of an object, and can be defined at the creation time, and subsequently modified and added at any time. Label selector is a grouping primitive in Kubernetes and allows client/user to identify a set of objects. There are two sets of selectors equality-based and set-based. An abstract way to expose an application running on a set of Pods as a network service. The set of Pods targeted by a Service is usually determined by a selector.

We looked at using the environment variables using the configmap in the last blog. The config map remains the same but now in YAML. apiVersion: v1 kind: ConfigMap metadata: name: happy-faces-mnt data: data.values: |- my.name=samarthya my.profession=engineer I will use this to mount it to a container and then see the contents of the file.

What is K8S? # To manage distributed containerised workloads, you use Kubernetes. It is resilient declarative configuration. It is open source platform. It allows you to bundle and run your application. From the official Kubernetes documentation. Kubernetes objects # K8S objects are the persistent entities in the K8S system and thee entities are used to represent the state of the cluster.

Someone asked me to define a simple docker pod for the service I wrote in GOLang here. I completely forgot about it, till today. So here is my first attempt at getting a docker image for the service.

The performance I was getting after deploying Elasticsearch 7.6.2 and Kibana in my local VM’s was not satisfactory, so I thought of deploying a front proxy Nginx in front and enabling static resources caching.

Learn by example # In this blog let’s learn how to create a bridge network. By definition - Bridge networks are usually used when your applications run in standalone containers that need to communicate

Timeseries is a data stream that is defined for a set of fields over a time period. Some examples may include, daily attendance register for an organization, wherein people punching in every-morning will be different, missing mostly for weekends or on planned and unplanned leaves.

I would move in stages, like a good learning guide always recommends.

PV and PVC: PersistentVolume and PersistentVolumeClaim # PV - Defines the structure for provisioning storage for use. They have a defined lifecycle, independent of any individual Pod that uses the PV.

Usually I use Centos, but this time I wanted to try out docker swarm and ubuntu was the preferred option due to the options and supporting blogs available.

Watcher # Watcher is an Elasticsearch feature that facilitates alerting and can be used to create actions based on conditions, which are periodically evaluated using queries on your data. There are two ways of creating a watcher

Security space is an ever evolving. It never stays stagnant; everyday you see a new exploit, a new hack. As an individual, you may have Antivirus installed on your system, Firewall - UP, ports blocked yet you are not free from threat

Aggregation works on a group of documents and can be categorized Metric Pipeline Bucket Matrix Metric Aggregation # Computes a single or multiple values

It is a short howto guide which shall talk about adding a LDAP realm to an Elasticsearch 7.6.2 cluster. Realms # It is a concept not alien to IDAM professionals. In simple words it is a system that allows you to authenticate and authorize an incoming request. It will define the entity USER, that may include at a bare minimum (not limited to)

In this blog we will try to setup a realm for elastic authentication. yum -y install openldap-clients openldap-servers mymachine>systemctl start slapd mymachine>systemctl enable slapd Created symlink from /etc/systemd/system/multi-user.target.wants/slapd.service to /usr/lib/systemd/system/slapd.service. mymachine>slappasswd New password: Re-enter new password: {SSHA}R5nPS+zItZsQrfsw+PIQSNnEiMpLcwLs mymachine>vi /etc/openldap/slapd.d/cn=config/olcDatabase={2}hdb.ldif # AUTO-GENERATED FILE - DO NOT EDIT!! Use ldapmodify. # CRC32 975a66fa dn: olcDatabase={2}hdb objectClass: olcDatabaseConfig objectClass: olcHdbConfig olcDatabase: {2}hdb olcDbDirectory: /var/lib/ldap olcSuffix: dc=my-domain,dc=com olcRootDN: cn=Manager,dc=my-domain,dc=com olcDbIndex: objectClass eq,pres olcDbIndex: ou,cn,mail,surname,givenname eq,pres,sub structuralObjectClass: olcHdbConfig entryUUID: 64a2a5de-3438-103a-8e85-c1c4cc3ef35f creatorsName: cn=config createTimestamp: 20200527073533Z entryCSN: 20200527073533.541185Z#000000#000#000000 modifiersName: cn=config modifyTimestamp: 20200527073533Z mymachine>firewall-cmd --add-service=ldap success mymachine>vi ldaproot.ldif dn: olcDatabase={0}config,cn=config changetype: modify add: olcRootPW olcRootPW: {SSHA}R5nPS+zItZsQrfsw+PIQSNnEiMpLcwLs mymachine>ldapadd -Y EXTERNAL -H ldapi:/// -f ldaproot.ldif SASL/EXTERNAL authentication started SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth SASL SSF: 0 modifying entry "olcDatabase={0}config,cn=config" mymachine>cp /usr/share/openldap-servers/DB_CONFIG.example /var/lib/ldap/DB_CONFIG mymachine>chown -R ldap:ldap /var/lib/ldap/DB_CONFIG mymachine>systemctl restart slapd mymachine>ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/openldap/schema/cosine.ldif SASL/EXTERNAL authentication started SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth SASL SSF: 0 adding new entry "cn=cosine,cn=schema,cn=config" mymachine>ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/openldap/schema/nis.ldif SASL/EXTERNAL authentication started SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth SASL SSF: 0 adding new entry "cn=nis,cn=schema,cn=config" mymachine>ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/openldap/schema/inetorgperson.ldif SASL/EXTERNAL authentication started SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth SASL SSF: 0 adding new entry "cn=inetorgperson,cn=schema,cn=config" vi ldapdomain.ldif dn: olcDatabase={1}monitor,cn=config changetype: modify replace: olcAccess olcAccess: {0}to * by dn.base="gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth" read by dn.base="cn=Manager,dc=samarthya,dc=com" read by * none dn: olcDatabase={2}hdb,cn=config changetype: modify replace: olcSuffix olcSuffix: dc=samarthya,dc=com dn: olcDatabase={2}hdb,cn=config changetype: modify replace: olcRootDN olcRootDN: cn=Manager,dc=samarthya,dc=com dn: olcDatabase={2}hdb,cn=config changetype: modify add: olcRootPW olcRootPW: {SSHA}R5nPS+zItZsQrfsw+PIQSNnEiMpLcwLs dn: olcDatabase={2}hdb,cn=config changetype: modify add: olcAccess olcAccess: {0}to attrs=userPassword,shadowLastChange by dn="cn=Manager,dc=samarthya,dc=com" write by anonymous auth by self write by * none olcAccess: {1}to dn.base="" by * read olcAccess: {2}to * by dn="cn=Manager,dc=samarthya,dc=com" write by * read mymachine>ldapmodify -Y EXTERNAL -H ldapi:/// -f ldapdomain.ldif SASL/EXTERNAL authentication started SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth SASL SSF: 0 modifying entry "olcDatabase={1}monitor,cn=config" modifying entry "olcDatabase={2}hdb,cn=config" modifying entry "olcDatabase={2}hdb,cn=config" modifying entry "olcDatabase={2}hdb,cn=config" modifying entry "olcDatabase={2}hdb,cn=config" mymachine>vi baseldapdomain.ldif dn: dc=samarthya,dc=com objectClass: top objectClass: dcObject objectclass: organization o: samarthya com dc: samarthya dn: cn=Manager,dc=samarthya,dc=com objectClass: organizationalRole cn: Manager description: Directory Manager dn: ou=People,dc=samarthya,dc=com objectClass: organizationalUnit ou: People dn: ou=Group,dc=samarthya,dc=com objectClass: organizationalUnit ou: Group mymachine>ldapadd -x -D cn=Manager,dc=samarthya,dc=com -W -f baseldapdomain.ldif Enter LDAP Password: adding new entry "dc=samarthya,dc=com" adding new entry "cn=Manager,dc=samarthya,dc=com" adding new entry "ou=People,dc=samarthya,dc=com" adding new entry "ou=Group,dc=samarthya,dc=com" dn: cn=Manager,ou=Group,dc=samarthya,dc=com objectClass: top objectClass: posixGroup gidNumber: 1009 ldapadd -x -W -D "cn=Manager,dc=samarthya,dc=com" -f ldapgroup.ldif Enter LDAP Password: adding new entry "cn=Manager,ou=Group,dc=samarthya,dc=com" mymachine>useradd samarthya mymachine>passwd samarthya Changing password for user samarthya. mymachine>ldapadd -x -D cn=Manager,dc=samarthya,dc=com -W -f ldauser.ldif Enter LDAP Password: adding new entry "uid=samarthya,ou=People,dc=samarthya,dc=com" dn: uid=samarthya,ou=People,dc=samarthya,dc=com objectClass: top objectClass: account objectClass: posixAccount objectClass: shadowAccount cn: samarthya uid: samarthya uidNumber: 1005 gidNumber: 1005 homeDirectory: /home/samarthya userPassword: {SSHA}R5nPS+zItZsQrfsw+PIQSNnEiMpLcwLs loginShell: /bin/bash gecos: samarthya shadowLastChange: 0 shadowMax: 0 shadowWarning: 0 Installing phpLdap # yum install php-ldap php-mbstring php-pear php-xml https://www.digitalocean.com/community/tutorials/how-to-install-linux-nginx-mysql-php-lemp-stack-on-centos-7

Kibana is an effective way of managing users; but as a programmer you always need an API. It helps to integrate with other applications you are building Quick and clean way without any cosmetics > GET /_security/_authenticate HTTP/1.1 > Host: myserver.elastic.node:9200 > Authorization: Basic ABCDEFGHIJKLMNOP== > User-Agent: insomnia/7.1.1 > Accept: */* < HTTP/1.1 200 OK < content-type: application/json; charset=UTF-8 < content-length: 236 * Received 236 B chunk * Connection #2 to host myserver.elastic.node left intact GET https://myserver.elastic.node:9200/_security/_authenticate I have added basic authentication to quickly check

GOLANG # Go is a powerful yet simple language. The nuances of assembling a workspace, sometimes are overwhelming, yet when it comes to simplicity it is very powerful.

Helpful links # http package How to write go code. First program Effective go GO webserver GO Packages Specification - Go JSON in Go blog Go command reference Using Go modules These are some of the helpful links I have used in the past for understanding Go and developing code.

Having practiced creating loads of Console based programs of golang, it is now time to try something new - A Web Service. In this blog, I will try and create a Web Service (RESTful) that will be built entirely on Go.

So continuing from the last blog where I wrote a crude version of SimpleCipher and was inclined on removing the duplicate code, here I am with the improved version. Handy links # [fmt](https://golang.org/pkg/fmt/) package [strings](https://golang.org/pkg/strings/) package. Defining types # While I compiled and unit tested code the general feeling was what can be optimized.

Hello back again, Golang is just a new syntax for me. Even though behind the scenes they developers have simplified many things, for a consumer like me writing a program is more about learning the new syntax and getting familiarized with the control and stack.

Text is a key data in most of the information that is ingested in the elasticsearch, and whether it is type text or keyword it requires careful consideration to choose which analyzer to use.

Painless is a simple, secure scripting language designed specifically for use with Elasticsearch. The official guide is available here.

Index modules are a complex subject; they are defined per index and control every aspect of an index.

Pipelines is a smooth way of processing some incoming data. In this blog, I will walk through some of the items I have tried my hands on. Pipeline # A pipeline is a definition of a series of processors that are to be executed in the same order as they are declared.

_reindex # Official documentation here. This API is meant for copying data from one index to another. It is especially useful when you are copying data from one index to another.

What is intriguing about elastic stack is that; it can do so many things that it simplifies the life of a security analyst. The uphill task is to learn the technology itself, but once you have mastered it, things are much easier.

The Rest API is the fundamental fabric of K8S. Every communication that happens between components, happens through the API invocations. Consequently, everything in the Kubernetes platform is treated as an API object and has a corresponding entry in the API.

It is always fun learning and trying out some new tech. In this blog, I will try and capture my experience of running my first pod using kubectl. If you wish to learn how to install a typical master, worker cluster of K8S I recommend going through this blog.

I will extrapolating based on the original documentation available here and based on some of my experience using this API. It copies documents from one index to another. Extracts the document source from the source index and indexes the documents into the destination index. You can copy all documents to the destination index, or reindex a subset of the documents.

Open specification for document fields for data ingested in Elasticsearch. It brings uniformity to certain common fields that are used across data sources. ECS defines two field levels Core fields: Represent the fields that are most common across all use cases. Extended fields: That are not part of the core and specific to cases. Field sets # Field sets are multiple groups of related fields defined by ECS. More information here.

If you are not aware of mapping, it might be a little confusing but I would try my best. Every document that gets indexed in elastic has fields, every field that is accessible has an associated type with it (Inverted Index) which helps elastic organize data for quick retrieval.

With the Corona-Lockdown in place “literally”, it was time to squeeze out some me-time to continue the quest for GO-Proficiency.

In this post I will try to enable security for the elasticsearch cluster. It would be simply penned as a sequence of steps you need to follow to enable security as part of the cluster.

If you have a production grade elastic cluster it is always recommended to do a regular backup as a standard practice. Snapshot is a backup of information, that can be be used to restore the last known good state of the system

In the previous blog we searched for a match\all, and sorted it by age. In this we will improve upon the query we are writing and look for other options.

What I have been doing is evaluating Elasticsearch. The indexing, searching (Phrase, Similarity, Structured, Full Text, Complex), clustering etc. In this blog I will just try and simplify the learning curve.

Just like Java or C++ there are standard libraries (packages) available with golang. The one that you might end up using the most often is strings. For this blog, I was writing another practice exercise that is meant to count number of words in an input string.

As a software professional, I have been working in the space of IAM, SSO, DRM for quite some time and from my experience I can say, the way Elastic has simplified the integration of external or internal security for access is a treat for fellow developers. It was interesting to experiment with the security settings, and control tenancy.

Data is key to any analytics, and if it can be organized, normalized and made available for correlated analysis, the results will be more concise. Elastic.co has a SIEM UI and the XPack security pack that helps the new age security analysts.

It was an interesting exercise, and I solved in three ways. More details available here @ exercism.io Introduction # RNA can be broken into three nucleotide sequences called codons, and then translated to a polypeptide like so:

Elastic Stack Monitoring Service is available with XPack only. GET _cluster/settings { “persistent” : { “xpack” : { “monitoring” : { “elasticsearch” : { “collection” : { “enabled” : “false” } }, “collection” : { “enabled” : “true” } } } }, “transient” : { } }

A pangram (Greek: παν γράμμα, pan gramma, "every letter") is a sentence using every letter of the alphabet at least once. The best known English pangram is: The quick brown fox jumps over the lazy dog.

An interface in Golang is a set of methods and a type.

All right, its been a busy week and finally some time at hand to write about something I have been doing.

Security is integral to every and all the tasks in a shared environment.

Funny thing about simple programs is that there are too many optimization options available that never lets you feel content.

* Header image from Elastic.co As we move towards the micro-service/container economy where there are heaps of products available in form of applications to be downloaded & installed on primarily two platforms Android and IOS, we just cant underline the security implications enough.