Project Graylog

In this project I am going to install and configure system monitoring software Graylog using salt.
Beginning the project by setting up master-slave architecture and testing states first with different guest slave.
When testing the states, master is guest os ubuntu and slave is guest os ubuntu.

First state is for mongodb.

State for mongodb.
Apparently had wrong syntax on init.sls. Fixed by adding “name:” in front of cmd.run functions.

Then I got error saying that signatures couldn’t be verified even though I had the key on init.sls. The problem seemed to be that importing the key was slower than installing a package, so I had to make two separate cmd.runs, and then it worked.

After mongodb I started installing elasticsearch which was a tricky one. When I tried to edit the conf file of elasticsearch, it kept displaying an error that said file not found. I spent a lot of time trying to debug the problem, but I couldn’t find solution so I moved to install graylog instead.

State for elasticsearch.
I checked thousands of times that elasticsearch.yml file was on place. I even deleted it and added it again.
After I was able to succesfully install graylog, I compared the rights of elasticsearchs and graylogs conf files. Apparently elasticsearch created a group which I wasn’t part of, so I had to edit the rights so I could read it. I enabled reading right for others.
Then it finally worked.
Conf file for elastic search. I added the very first line and one line at top of the file: cluster.name: graylog
State for graylog.
Graylog demanded generating hashed passwords on its configuration file, otherwise it wouldn’t start.
http_bind_address pointing to slaves IP. I assume this part could be done somehow using grains.items?
Adding the last line in order to send logs to graylog.

After all above steps were done, it was time to test if the graylog was functioning. I navigated to webbrowser typed in the slaves ip address and port 9000 which is used by graylog: 192:168.1.33:9000

It works! The default user name was admin, and password was set up when editing the configuration file.
This is the graylog interface. No logs is showing because there isn’t any inputs yet. Let’s create one.
Creating syslog udp input.
After creating the input, I generated some log files, and now they are appearing on graylog.
Advertisements

Installing LAMP stack using SALT stack (h6)

In this exercise I am going to install LAMP with a help of salt. My virtual ubuntu server at digital ocean will work as a master and give orders to guest os slave which is ubuntu server.
Master: Digital ocean droplet / Ubuntu server 18.04 /1GB memory / 25GB Disk
Slave: Guest OS Ubuntu server 18.04 on Windows 10 host PC

Beginning by cloning my git repository on guest ubuntu so I can use my shell script to make it a slave. After the script has run I accepted the key on my master. Now I have working master-slave architecture.

I wanted to install LAMP only by applying a highstate so everything would be fully automatized. To correctly do this without any bigger hassle on the ubuntu guest machine, I wanted to test every state before applying them so I manually tested every state first at another virtual machine.
Everything worked just fine except mysql.

Simple sls file to install apache2.
Work like a charm.
Php state was little bit more complex but it worked well.
Modified dir.conf to test out php on web.
Php and apache working, mysql to go.

I left mysql installation for last on purpose, because I remembered it might be a tricky one. I remembered there were something annoying with mysql passwords at the installation, and wondered what would be the easiest way to install it.
Fortunately I found article at terokarvinen.com which was very helpful:
http://terokarvinen.com/2018/mysql-automatic-install-with-salt-preseed-database-root-password
The thing was preseeding, answering questions on installation beforehand.

So this was the state for installing mysql and answering the questions beforehand. In addition to mysql pkgs, debconf-utils pkg was required.

At first when I applied that state, it froze completely. I got no feedback from salt master whether the state were correctly applied or not. I checked with salt-run jobs.active and saw that the job was still running but nothing happened. I am not sure what was going on there, but I had to quit for that day so I shut everything down.
Few days after that, I came back and changed the password on the sls file and tried to apply it again. It didn’t freeze this time, but I still got some errors.

Apparently the preseeding part succeeded, but mysql installation had some problems. That problem might be caused by faulty installation. So I removed the broken installation.
I navigated to updates directory and removed everything and after that ran apt-get update. After this, I applied the mysql state again at master, and now it fully succeeded! I could log in on mysql by using the password I had been set up before on the sls file.

Time for highstate.

First error occured. Fixed by changing names of the ID’s.
After fixing the previous error and 54 seconds LAMP was successfully installed on guest ubuntu minion.
Apache working on ubuntu minion.
Php working as well. I forgot to create a info.php test file on php.sls, so I had to manually add it to the state afterwards.
Mysql works too. No struggle with the password, I just typed in the password I had previously set on the mysql state. Now I have fully functioning LAMP installation which I may apply on any slave and have fully functioning webserver just in 54 seconds.

Salt on Windows (h5)

In this exercise I’m making my windows host desktop as a slave, and creating a new master available to public internet.
In my previous exercises my master has been a virtual machine which has no public ip address, which has been kinda boring, so I decided to create a virtual server at digital ocean. I made a virtual ubuntu server 18.04 which has a public IP address, 1GB RAM, 25GB SSD and is located at Frankfurt data center.

a)
Since my virtual server is publicly available to internet, it was safe to enable firewall. I used commands: sudo ufw enable & sudo ufw allow OpenSSH. Latter command is important because I might want to acces my server with ssh again.
The virtual server was very easy to set up, and I installed salt on it right away. I begun by installing both minion and master on the server to test if the master-slave architecture was working correctly, and it was.
After that I tried to make my windows host machine a slave, but I couldn’t succeed at first. Master didn’t recognize any keys when typing sudo salt-key. Then I remembered I had enabled firewall, so I had to make some rules for salt. I typed following commands: sudo ufw enable 4505/tcp 4506/tcp. Salt uses these two ports, and after allowing them master got request from windows slave to accept its key.

b) Using salt locally on windows slave

I tried to install advanced port scanner locally on windows, but got an error. Installing wireshark worked instead.
Windows IP Configuration.

I used some old virtualmachines in oracle virtualbox while collecting more slaves for my new master, but this wasn’t as easy and fast as I thought it would be. At first, it seemed to be an easy task to just change the masters address in /etc/salt/minion files. So I started four of my old virtualmachines and changed the addresses to be my new masters address. After doing this, I was able to accept all of keys on master except one.

When I checked the status of this minion, I saw that the log were full of same error.

I triple checked that I edited the minion file correctly and was certain that there were no typos. Still I was getting error which said: “The Salt Master has cached the public key for this node, this salt minion will wait for 10 seconds before attempting to re-authenticate”
I didn’t find any help from google, so I checked the file again and this time scrolled all the way down.

This is what I saw at the bottom. There were another master address! I should have checked the whole file…

Removing the extra master address from the minion file fixed the error message, but I still couldn’t connect slaves on my master. Fortunately, there is logs!

Because master has changed, so is the public key which minions use for authenticating. I had to remove the old masters public key, restart the minion and then it worked.

c)
I tried to locate application configuration files in windows, but I could find only few of them, and editing them didn’t seem to work. I am not familiar with windows’ configuration files, and would like to learn how to edit them using salt.

Vagrant & simple shell script (h4)

a)
In this exercise I’m going to write a simple shell script, that will make my machine a salt slave.
I dont have experience in shell scripting but since this script is quite simple I decided to give it a try:
#!/bin/bash
sudo apt install salt-minion
sudo nano /etc/salt/minion: “master: 192.168.190.128”
sudo systemctl restart salt-minion

This didn’t work. It just opened the file /etc/salt/minion but didn’t type in my masters ip. So I decided to look it up from the internet, and found article in stackoverflow which helped me to get forward.

I had to use “echo” instead of nano.
After running the shell script on slave, the key appeared on my master waiting to be accepted. My master in this case is Xubuntu running on VMware on my host Windows 10.

I thought it would be good idea to put the script on git so I could easily access it on a computer I want to make as a slave.
First I cloned my git srv repo to my slave: git clone https://github.com/arttugit/srv.git
Then added the script to git: git add testscript.sh
And finally making a commit: git add . git commit; git push; git pull

c)
Now it is time to install and try vagrant. My few first tries installing vagrant wasn’t so successful. I begun installing vagrant and virtualbox. After both packages were installed I headed to vagrant website to look for boxes. At this time I was trying to install vagrant on my guest machine xubuntu. I picked the most downloaded one, ubuntu server 14.04 LTS and entered following commands:

Everything seemed to work, until it stuck there and after five minutes it timed out.
I tried installing it again, but no luck. I even tried it with other guest machine.

After investigating this issue and finding information about it on the web, I found one post that suggested enabling the GUI so I could see an error message. So I enabled the GUI by editing the Vagrantfile.

After enabling the GUI this error message displayed. Then I realized it may be a little complicated to run virtualization within a virtualization.
Then I booted xubuntu live and did the exact same steps I did before inside virtualmachine. It seemed to be working.
After succesfully starting the VM, I was able to SSH in to it.

Pkg-file-service & salt states (h2, tiistain ryhm√§)

b)
Creating a salt state which enables Apaches users’ homepages. At first, installing everything manually on master: sudo salt ‘*’ pkg.install apache2
After that, enabling the userdir module: sudo a2enmod userdir
Then I created a directory named public_html, changed permissions (chmod 777 public_html) and created a file index.html. Then I could access the site via webrowser.

Let’s try to automate enabling user homepages on slaves by creating a state. I used
http://terokarvinen.com/2018/apache-user-homepages-automatically-salt-package-file-service-example as an example.
First error occured when I tried applying the state.

Error was fixed by removing one colon.

After fixing the error above, I applied the state again and this time it seemed to work.

At first, I hadn’t created a “default-index.html” file so I got this error. Everything else seemed to work as they should. I just had to manually create a public_html directory to set a user homepage.

After creating “default-index.html” file under the /srv/salt/apache/ directory, and applying the state again, apaches homepage was replaced by file I had just created.

c)
Next step was to enable php to function on users’ home directories as well. I installed php and made some modifications on apaches conf files. I did this manually on salt master, to make sure everything works before making a state. First I had to install php: sudo apt install php libapache2-mod-php after that, I made slight modifications to php7.2.conf file. I found a tutorial on ubuntus site how to enable php on user dir which helped me to get this working: https://wiki.ubuntu.com/UserDirectoryPHP
And after restarting apache daemon, php worked on my home directory.

After commenting the last five lines out, php worked on home directory.
I made a file called “info.php” which is a test file to test if php is working. File contains little piece of php code:
<?php
phpinfo();
?>

Then I wanted to make above steps to happen automatically. So I made a state.

State for enabling php on user directories.
Modified php.conf file.

Problems I encountered was few syntax errors when writing the state and wrong file name in source file. After fixing those errors, I created info.php file on slaves to test if it was working.

And it was.

d)
In this task, I created a state that would set up name-based virtual host on apache. I haven’t done virtual hosts on apache before, so I configured that manually first. I found a handy tutorial for doing that:
https://www.maketecheasier.com/name-based-virtualhost-apache/
So basically what I did was create directories for both of virtualhosts, make configuration files for them and tell apache to use them.

Oops, some errors. I had a typo on one of the conf files. Fortunately I didn’t broke anything.
State for virtual hosts.
State didn’t function as wanted.

e)
I couldn’t get example homepages work for new users. I was able to make a state which would create a index.html file for new users, but it didn’t work as I wanted.

I had to create a public_html directory and move the index.html file there to make it work.

Configuring Salt Master&Slave- architecture (H1)

Starting with salt stack. I haven’t used Linux that much yet, and I haven’t heard about salt stack before this course started. Controlling hundreds of slave computers does sound interesting for sure! I have read some get started-tutorials at docs.saltstack.com, and there is much to learn. Citing the tutorial: “You can get a general understanding of how Salt works by seeing it in action.” so let’s get started.

c)
At first I installed salt by using ubuntus apt packet manager: sudo apt install salt-master salt-minion
After the package installation, I configured the slave by editing the minion file: sudoedit /etc/salt/minion. I added two lines of text:

master: 192.168.190.128
id: arttu

Then i restarted the daemon: sudo systemctl restart salt-minion.service. In this case, master and slave was the same computer. Now I had to accept the slaves key on the master by typing sudo salt-key –accept-all which would then accept that computer as a slave. I also started another virtual machine to make it a slave for my master computer. I repeated the above steps on the another machine, and changed the id so I could identify my slaves. Then I would type in: sudo salt ‘*’ cmd.run ‘hostname -I’ to test the connection to my slaves. “*” refers to all slaves, so all slaves would return their hostname.


For now, I have only two slaves.
Executing remote commands.
Installing packages.

d)
I tried some salt states using examples posted here: https://github.com/joonaleppalahti/CCM/tree/master/salt/srv/salt
I used the firewall.sls as and example to apply firewall settings on my slaves. I began by creating a directory /srv/salt/. This directory contains instructions of the slave machines. I wanted to make firewall state so I created a file called firewall.sls.

The file looked like this.

After adding some firewall rules (sudo ufw allow 22,80,443,4505,4506/tcp), I tried out the state.
At first, I encountered some errors.

Error message after applying the state.

Apparently, there were incorrect syntax in my firewall.sls file. It was fixed by removing the space at the start of fourth line. And after that, it worked!

e)
With salt grains I am able to collect information about my slaves and master. Command sudo salt ‘*’ grains.items returns a lot of information about computers under my control, such as processor type, IP address, OS type, memory and other system information.

System information about salt-master.

f)
Time to test salt on windows, and make my host computer a slave. I downloaded and installed salt-minion from https://docs.saltstack.com/en/latest/topics/installation/windows.html on my windows 10 host machine. Installation was fast and simple, all I had to do was to type in my masters IP address, and set a hostname for windows slave. Then i accepted the key on my master.

To enable salt windows repositories, I need git. Using command sudo salt ‘*’ pkg.install git, I can do the installation easily to all computers. After git was installed, I updated windows repos.

After updating repos, I tried to install VLC player on all slaves.

Installing VLC from master to all slaves worked. Soon after running command shown above, a shortcut of VLC player appeared on my host Windows desktop! Maybe I should install more apps using my master. I recently reinstalled my windows, so I dont have many apps installed at the moment. So why dont I install few useful apps at the same time, from my guest machine with salt.

State for installing apps.
After applying the state on master, I could install three apps at the same time to Windows using Linux. Amazing.

Samba

In this excercise I configured SMB protocol (Server Message Block) on my Ubuntu server. Software package used is Samba.

Samba is easy to use, and when using desktop version of ubuntu its even easier. If you want to access file server, simply navigate to nautilus file manager in ubuntu desktop, select “other locations”, and connect to server using following syntax: “smb://<serversname>/

Configuring Samba on Ubuntu server

First I had to get Samba package using command: $sudo apt-get install samba
After succesfully installing samba package, I created a directory for samba: $mkdir /home/arttu/sambashare
Next step was to edit the configuration file of samba: $sudo nano /etc/samba/smb.conf

I added the commands in yellow text to the configuration file.

When the commands shown above has been typed in and the configuration saved, I restarted the Samba for those commands to take effect: $systemctl restart smbd.service
Samba doesn’t use system account login, so I had to make a user for samba and add password to it.

Adding password for user “testi”.

This is basically how to set up Samba. If you dont have firewall enabled on your server, you are good to go and you have a samba server up and running. If you have enabled firewall, then few rules needs to be added. Samba uses ports 139/tcp and 445/tcp, so we need to allow these ports.

Setting up few firewall rules for Samba.

After setting the rules in Ubuntu server, we need to map network drive in windows.

Mapping the network drive for Samba. “sambashare” is the directory in my ubunutu servers home directory.
After connecting, it asks for credentials which I gave earlier.

And now I have a working Samba server. I can access files from windows which are located in the “sambashare” directory on my ubuntu server.

Apache2 Excercise

1.
Installed Apache2 HTTP Server on my Ubuntu server. Command was: sudo apt-get install apache2.

If I wanted to reinstall apache2 i would type in: sudo apt purge apache2 .

Useful commands for using apache2:
apache2ctl start
apache2ctl restart
apache2ctl stop
apache2ctl fullstatus or
systemctl start apache2
systemctl restart apache2
systemctl stop apache2
systemctl status apache2

a2enmod



2.
/etc/apache2/apache2.conf is the main configuration file. mods-available contains all modules and mods-enabled contains modules which are currently in use. Modules can be enabled with a2enmod command. file “envvars” is default environment variables for apache2ctl.

3.
I created a www-page named “testing.html” in /var/www/html.
I can reach the site by typing my apache servers ip address to browsers search bar and /testing.html after the ip.

4.
Userdir module allows user-specific directories to be accessed using the http://example.com/~user/ syntax.

To enable it: a2enmod userdir
To disable it: a2dismod userdir



5.
Restarting the apache2 is done by typing in: systemctl restart apache2

6.
Public_html directory is for normal users to create their own web page.

10.
Copy file: default-ssl.conf > /sites-enabled/
Generating RSA key: openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/apache2/ssl/apache.key

Warning message from browser.

12.
I could get a free SSL certificate from Let’s Encrypt project, but first I would have to register a domain name, because Let’s Encrypt certificate authority will not issue certificates for a bare IP address.