Posted on Leave a comment

Syncthing = Dropbox & GDrive Alternative

Syncthing

Google Drive (GDrive) and other cloud storage alternatives such as Dropbox and Microsoft Ondrive all have the serious drawback of keeping one's information in a third party cloud repository. Privacy and security are generally compromised this way, even when paying for storage (as opposed to having an advertising model, which is worse in many ways).

Continue reading Syncthing = Dropbox & GDrive Alternative

Posted on Leave a comment

DNS Records and Services

First, there are two kinds of DNS records: those for client look, and those for a server.

Client Lookup

I don't trust Google DNS, though for a while it was the go to DNS, and easy to remember at 4.4.8.8 8.8.4.4 and 8.8.8.8. For privacy, for me, there are two options, with the first being just better: - dns.watch 84.200.69.80 / 84.200.70.40 - 1.1.1.1 / 1.0.0.1 If one wants some security (as a service), then Quad9 is worth a look.

DNS Services

There are several DNS services to choose from. Dyn and related companies is the worst. Free DNS services such as afraid.org and he.net are unreliable, or simply not reliably fast. It makes the most sense to go with a top-rated DNS service (highly available and fast resolve times), and pay for this service (though less is more when it comes to expenses). - DNSmadeEasy.com - Silly name, $30/year for 10 domains, fast and reliable. Generally in the top 10 of private resolvers. I've not found better/faster for cheaper.

DNS Records

NS Records

There are several records to worry about. The first are nameservers, which are put into the registrar database. This can be as few as two or as many as six (possibly more).

A Records

Depending on the DNS Server, these can have wildcards or not. Generally there are at least three A records to have: - Root domain - www subdomain - * wildcard For certain services, it is required to have a www. and also people mistype this, so it is best to have it as a domain, to have it on the SSL certificate, and to have a reroute from www. to the root domain.

CNAME Records

Usually only Bing Webmaster Tools requires a CNAME record. Otherwise these are generally worthless.

MX Records

These are for the mailserver. Usually a few are needed, one plus two backups. Gsuite has five records, but that is overkill. The top three make the most sense. Also, there are priority numbers, e.g, 1, 5, 10 to govern the round robbin-style resolving. - 1, aspmx.l.google.com. - 5, alt1.aspmx.l.google.com. - 5, alt2.aspmx.l.google.com.

TXT Records

TXT records are the go to place for every third party to put their info. Several examples of TXT Records include: - Yandex Webmaster Tools validation - Google Webmaster Tools/Analytics/GSuite/etc. validation - _acme-challenge records for DNS-based authentication for LetsEncrypt

PTR Records

PTR records are essentially a reverse so that an IP address is associated with a host.domain.tld. This is key for sending email.

DKIM, SPF, DMARC

These are all records for email security, at various levels. DKIM and DMARC are TXT records, and SPF can be TXT or specific SPF records, depending on the DNS service provider. - Setting up Gsuite DKIM, SPF, DMARC - Google on DMARC records - Test SPF and DKIM - Google on SPF - DKIM on Gsuite - Google: About DKIM

SPF Records

SPF looks like:

host.domain.com / "v=spf1 include:_spf.google.com ~all"

SPF are one of the earliest and easiest email records to set up for security, and specifically states which hosts can send email for the domain.

CAA Records

These records help tell SSL Cert providers which of those providers can generate a cert for the domain records. Each host needs two records: - Name (host), Type: iodef, Value: "mailto:address@domain.com" - Name (host), Type: issue, Value: "letsencrypt.org"

Posted on Leave a comment

PHP and MariaDB on Debian

Note: instructions for installing and configuring phpMyAdmin also included below.


Related Artices in Debian Services and Applications - Debian on AWS Lightsail - OpenVPN on Debian + UFW Firewall - Nginx and Letsencrypt on Debian - PHP & MariaDB on Debian - Grav CMS on Debian


As of December, 2018 there are decent performance gains with the latest PHP and MySQL (MariaDB, not Oracle) versions. These are: - PHP 7.3.0 released 06 Dec 2018 - Next PHP release 7.4 likely out December 2019 - MariaDB 10.3.11 released 20 Nov 2018 - Latest MariaDB release 10.4 is in release candidate status as of May, 2019. It would be good to do a new version along with PHP when it's next is released, say Dec 2019/Jan 2020.

PHP 7.3 outperforms PHP 7.2 and earlier versions on nearly all real-world web cms platforms. At the same time, MariaDB does indeed have performance enhancements which generally make it faster than the Oracle offering. For MariaDB the performance advantages have been apparent since at least MariaDB 10.1 vs. MySQL 5.7 back in 2014.

This is no surprise, being that MariaDB was founded and developed under the direction of the original MySQL founder. The main advantages technically are better thread management and defragmentation of the MariaDB than MySQL databases. In addition, a larger variety of engines are available under MariaDB including NoSQL (Cassandra).

Set up PHP Repository and Certs

sudo apt-get install apt-transport-https lsb-release ca-certificates
sudo wget -O /etc/apt/trusted.gpg.d/php.gpg https://packages.sury.org/php/apt.gpg
echo "deb https://packages.sury.org/php/ $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/php.list

Update and Install PHP

Currently this is the 7.3 branch

sudo apt-get update -y
sudo apt-get install -y php7.3
sudo apt-get install -y php7.3-cli php7.3-common php7.3-curl php7.3-fpm php7.3-gd php7.3-json php7.3-mbstring php7.3-opcache php7.3-readline php7.3-xml php7.3-intl php7.3-zip
php7.3-mysql

Update and Upgrade apt

sudo apt update -y
sudo apt upgrade -y

Verify php-fpm status

systemctl status php7.3-fpm.service

stop injected data into server returns

sed -i 's/;cgi.fix_pathinfo=1/cgi.fix_pathinfo=0/g' /etc/php/7.3/fpm/php.ini
systemctl restart php7.3-fpm.service

Edit php7.3 php-fpm conf file if needed, e.g., increase upload size variables.

nano /etc/php/7.3/fpm/php-fpm.conf

Make the following changes:

cgi.fix_pathinfo = 0
...
max_execution_time = 300
...
upload_max_filesize = 32M
...
post_max_size = 32M

MariaDB - Install cert manager, key, repository

currently 10.3

sudo apt-get install -y software-properties-common dirmngr
sudo apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 0xF1656F24C74CD1D8
sudo add-apt-repository 'deb [arch=amd64,i386,ppc64el] http://mirrors.dotsrc.org/mariadb/repo/10.3/debian stretch main'

Then perform update and install mariadb-server

sudo apt update -y
sudo apt-get install -y mariadb-server
sudo systemctl status mariadb

Enable auth socket

sudo nano /etc/mysql/mariadb.conf.d/50-server.cnf

Add plugin-load-add = auth_socket.so in the [mysqld] section. Then save and restart MariaDB.

sudo systemctl restart mariadb.service

Secure the database

sudo mysql_secure_installation

PhpMyAdmin on Debian

Provided that Nginx and LetsEncrypt SSL is installed and configured. It is time to install PhpMyAdmin

sudo apt-get update
sudo apt-get install -y phpmyadmin

Add a symlink from /usr/share/phpmyadmin to /var/www/html or whatever directory for whichever website

sudo ln -s /usr/share/phpmyadmin /var/www/html

Note for security through obscurity, rename the link

sudo mv /var/www/html/phpmyadmin pma

Install and enamble mcrypt in php, and restart php-fpm

sudo apt-get install -y mcrypt
sudo phpenmod mcrypt
sudo systemctl restart php7.3-fpm

Test to see if it works

https://host.domain.tld/pma/

Limit access to /pma/ by ip address, by editing the nginx configuration

nano /etc/nginx/sites-available/default

Add the following line to the top above server:

geo $admin { default 0; 203.150.176.16 1; }

And put a nested statement under \.php as per this StackOverflow answer

location ~ \.php$ {
    location ~ (/phpmyadmin/) {          # add this
        if ($admin = 0) { return 404; }  # add this
        ## fastcgi parameters            # duplicate these lines
    }                                    # add this
    ## fastcgi parameters ##
}
Posted on Leave a comment

Nginx and Letsencrypt SSL on Debian

It is a good idea to get PHP and MariaDB on Debian set up before Nginx (except the PhpMyAdmin which can come after).

Related Artices in Debian Services and Applications - Debian on AWS Lightsail - OpenVPN on Debian + UFW Firewall - Nginx and Letsencrypt on Debian - PHP & MariaDB on Debian

- Grav CMS on Debian

Install Nginx

Edit the /etc/apt/sources.list to add the Nginx repostitory

nano /etc/apt/sources.list

Add the following repository (currently for Debian 9/Stretch)

deb http://nginx.org/packages/mainline/debian/ stretch nginx

Download and install the key for the repository

wget https://nginx.org/keys/nginx_signing.key
sudo apt-key add nginx_signing.key

Remove nginx-common, update apt and install nginx

sudo apt-get remove -y nginx-common
sudo apt-get update -y
sudo apt-get install -y nginx

Systemd / Nginx Race Condition

There is a known race condition, with a workaround as follows:

mkdir /etc/systemd/system/nginx.service.d
printf "[Service]\nExecStartPost=/bin/sleep 0.1\n" > /etc/systemd/system/nginx.service.d/override.conf
systemctl daemon-reload

Edit /etc/nginx/sites-available/default

Note: these edits are not comprehensive, just to get certbot working. Uncomment the following lines:

listen 443 ssl default_server;
listen [::]:443 ssl default_server;
...
location / {
...
try_files $uri $uri/ =404;
}

Where it says server_name _; change _ to an appropriate fqdn that has an appropriate A record. Save and restart the nginx:

service nginx restart

Letsencrypt Certbot

sudo apt-get update
sudo apt-get install -y python-certbot-nginx certbot -t stretch-backports

Run letsencrypt (automatic)

certbot

Test access from a browser.

HSTS Preload

Browsers have a list of servers that require https/ssl. Add sites to the list. Two things are required: 80 to 443 redirect, and an hsts header. For the redirect, add this server configuration:

server {
        listen 80 default_server;
        listen [::]:80 default_server;
        server_name _;
        return 301 https://$host$request_uri;
}

For the HSTS header, this needs to be added to each server. Can simply be added after the listen 443 ssl; line:

add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;

Nginx Info

Nginx has become the standard for much of the web, for the basic standard reason it is not creaky old (though of course still lovable) Apache. However, before we get too far ahead of ourselves, let's recall exactly what we need to know about Nginx in order for it to work as well as Apache: - Installation - Configuration files - Support of SSL / LetsEncrypt - SFTP/SCP access to file system (and file rights + ownership) - Multiple virtual servers / directories - Mimetypes - Support for PHP - Threading - .htaccess and related

Nginx and Related Files and Directories

Standard or default files and directories as follows: - /etc/nginx - application directory - /etc/nginx/nginx.conf - main configuration file - /usr/share/nginx/html - default website root directory - noted as html in nginx.conf - /var/log/nginx/error.log - error log - /var/log/nginx/access.log - access log - /etc/nginx/mime.types - mime types - /etc/php.ini - php configuration file

Nginx / PHP-FPM Security Issues

There are significant issues with PHP-FPM in terms of keeping site caching partitioned when using multiple websites/virtual sites. Opcache should be turned off and individual users should be in charge of a different php-fpm process for each site. How to do this is not listed here (just yet).

Posted on Leave a comment

Gsuite Free Domain Alias Mailboxes

Google likes to remove functionality on free products to induce upselling. This is a common tactic in many software/SAS models. However, the cost of adopting Gsuite is very high, relative to free. Essentially a 5-10 pack of mailboxes with $5/month for the least expensive Gsuite paid option. That's $300-$600/year. What is sadly missing is a less expensive option. I don't mind paying money for valuable services, but an individual consumer who really only has family mailbox accounts, this is ridiculous pricing. As someone with multiple domains, here is how to get around this issue.

No Duplicate Mailboxes

The main problem comes when one wants to have mailboxes that have the same username, e.g., info@primary-domain.com and info@secondary-domain.com. Because added-on domains are always only aliases, only the primary domain is possible (e.g., info@primary-domain.com), and all subsequent domains with the same info@ are aliases of the underlying primary domain.

Steps to Support Duplicate Mailboxes

The work-around is as follows: - Create a unique mailbox such as secondary-domain@primary-domain.com. - After some amount of time (an hour at the most) the address info@secondary-domain.com will be added (provided info@primary-domain.com was already a primary or secondary mailbox address). - Log into secondary-domain@primary-domain.com and add info@secondary-domain.com as a second account. This will generate an email which will be sent to info@primary-domain.com. Verify access with the verification code. Set that info@secondary-domain.com as the default and configure the mailbox to always send email from that address. - Log into info@primary-domain.com and add a forwarding address of secondary-domain@primary-domain.com. This will generate a verification code emailed to secondary-domain@primary-domain.com. Verify this. - Next, create a new filter for incoming mail addressed to: info@secondary-domain.com and have it forward email to secondary-domain@primary-domain.com and also delete the email locally. The steps above will properly route and address mail so that the new mailbox will function properly using the normally disallowed duplicate username in the free version of Gsuite.

Endgame with Gsuite

Frankly I dislike Google and Gsuite. My use is only a holding action to not have to deal with email migration. The vast majority of time I no longer use Gsuite other than calendar and email, and also the use of those accounts for YouTube and Google Business Listings, and also the Analytics/Google Ads suite. Obviously there needs to be Google accounts, but they can be independent Gmail accounts rather than Gsuite accounts. At some point (in 2019), I'll migrate off and do self-hosting on mail and calendar, and therefore move YouTube, Business, Analytics over to Gmail accounts.

Posted on Leave a comment

Debian on AWS Lightsail

This is a setup of several items, starting with Debian 9 on Amazon AWS Lightsail. This has server basics and apt, and then follows with links to additional articles. In general, after several years of running CentOS on Linode, and then Amazon Linux AMI on EC2 and Lightsail, I find that Debian 9 is simply faster, just as secure, and at least slightly easier to use.

Continue reading Debian on AWS Lightsail

Posted on Leave a comment

Niche Search

There are several white label options for search and if one considers advertising a viable business model to engage in, then consider Bing or Infospacehttp://www.infospace.com/partners/. Consider the case of Izito and related country code tlds, as well as MonsterSpace, both are pure play partner deals with nothing technical in their own right, with traffic and profitability. The best numbers come out of www.ecosia.org, which publishes their financial reports. I guess if the alternative is a search engine that doesn't provide some kind of additional value, that would be good. For sheer scale it is important to do a general purpose search engine. And then also having some level of curated content overlaid that would be black-hat proof, as well as some kind of vetting of ads?

Posted on Leave a comment

Xiaomi – Brand on the Rise

In my household we have been introducing Xiaomi products for the past six months. Overwhelmingly positive, but not without a hiccup here and there. In general, there is an odd mixture of: quality, design, and value. I say odd because generally those things don't go together. Great price, good looks, and works well. Not perfect, but nothing is. Xiaomi may have much bigger rivals, especially in China, but they have such a strong combination that they are able to compete with, and in some cases beat out such giants as Samsung in markets like India.

Continue reading Xiaomi – Brand on the Rise

Posted on Leave a comment

Dropbox Cloud Storage and Sync

Dropbox is a cloud storage and sync service, with additional editors/apps, such as Paper and Showcase. For various reasons, those additional Dropbox apps are not useful for our use cases. However, storage and sync are excellent in and of themselves, and generally superior to Google Drive which is the only real alternative.

What Dropbox gets Wrong

One thing that is maddening about Dropbox is that when renaming a folder, all files and sub-folders within the folder are re-synchronized. This can be a huge undertaking (in terms of time required, not to mention wasted bandwidth. - Dropbox Rule #1: Try not to rename folders Another thing is the slow Microsoft Online Editors for Word and Excel. These can be very tedious to use, and there is more limited functionality than found in native editors for desktop operating systems. - Dropbox Rule #2: Use a native editor on Word and Excel documents when possible. Preview for Word and Excel do not support Indic Scripts (Fonts). This means that any Thai vowel, tone mark, or silent mark will not show properly in preview (but will when editing). This is a very odd limitation, and is based on a very poor preview functionality. In contrast, nearly all other editors support Indic Scripts (South and Southeast Asia-style fonts), with the only other known exception being

Backing up Multiple Folders with Symlinks

Besides what is in the main Dropbox folder (which can be some or all of the contents), there are times when folders in other locations are needed to be included in a backup. To do this, simple create symlinks (symbolic links) from the command line. Aliases created in the Finder do not work as symlinks, so the command line is needed (or some third party app, so unnecessary). The following example points to a folder on an SDCard:

ln -s /Volumes/jm-music/iTunes-Library ~/Desktop/Dropbox/iTunes-Library-symlink
  • ln = link
  • -s = symbolic
  • /Volumes/jm-music/iTunes-Library remote link destination
  • ~/Desktop/Dropbox/iTunes-Library-symlink local link location

Dropbox vs. Google Drive for Mobile Devices

Dropbox is one of the most widely available cloud storage providers in terms of support by third-party mobile apps. While Google Drive has increased its coverage, and Microsoft lags a bit behind, Dropbox is reliably the foremost access provider for cloud storage. As well, the Dropbox app can backup images/video from mobile devices automatically.

Dropbox Desktop Sync Performance

Dropbox is a much better application for synchronization of files, in terms of stability, reliability, and resource utilization (at least on OSX). Google Drive synchronization is a nightmare of processor utilization, hangs, and error messages.

Dropbox Security Audit in Four Steps

Storage in the Cloud does not magically remove the need for security, and especially that rare creature, the security audit. From a post over at Labnol, we learned how to do a Dropbox security audit, which is important for obvious reasons. However, this requires vigilance and a repeated review, something scheduled in your calendar. Note that the user interface at Dropbox changes over time so these steps need to be updated regularly. - Last updated 18 October 2017

Step 1 - Run the Security Checkup

Run the Dropbox security checkup which reviews devices/browser, connected apps, and suggests a password change, as well as review of two-step authentication settings.

Step 2 - Review Devices and Browsers

Check the [devices and browsers which access Dropbox](https://www.dropbox.com/account/security. Anything suspicious?

Step 3 - Review Connected Apps

Review the connected apps enabled to access Dropbox. Anything suspicious?

Step 4 - Review Available Space

Check the Dropbox plan and space used/available. After all, availability is one aspect of security.

Posted on Leave a comment

Dropbox Paper, Markdown, Sync

Dropbox Paper is a product I really want to like. For one thing, the promise of better editor is something long unfulfilled. And taking some design cues (or perhaps merely unrelated similarities), Medium did do something nice for the blogging environment. By extending it as essentially a wysiwyg Markdown+ editor, drag and drop-friendly, with handy JavaScript handles for visual editing, this is definitely an interesting project.

Markdown as a First Class Filetype

However, there are those of us who prefer something with Markdown as a first class document filetype, which could be seamlessly synchronized alongside other files, and edited with other editors. This is what the cloud editors do, after all, provide some level of editing of desktop-class documents, collaboratively, and those same files are generally available in the same binary format (via sync or import/export). Of course by Markdown we mean much more than the anemic initial (but no less necessary) initial Markdown spec. We like Markdown Extra as a more complete specification. With Dropbox Paper there is at least one additional feature from straightforward Markdown, the inclusion of images without having to know where they physically reside. Drag in and the image appears as a part of the document. This is similar to what Github does well with its Github-flavored Markdown. Clearly there is some kind of zip/archive file format behind the scenes, which we simply don't have access to, or perhaps a nasty rats nest of pointers in a database. The thing is, Paper doesn't have a public spec or source available. In other words, Paper documents only live in Paper, the Application (akin to Google Docs). This means Paper is not a first class filetype, and therein lies the rub.

Stable File Format is Key to Offline Sync, Editor Diversity

With its Office-in-the-Cloud, Microsoft actually preserves both the file format and allows a variety of editors (which is what preserving the file format enables). This in a superior way with Word and Excel documents, essentially round-tripping edits into synced files in the desktop or the cloud. Libre Office Online, Collabora CODE, and Collabora Online are similar to the Microsoft approach, with files not changing their basic structure. Of course one would expect Microsoft to take this approach of focus on file format, since it is what helped cement their leadership in editing applications. Own the format, own the tools. Google took a different approach (for scalability reasons, surely), and the ability to edit anywhere requires offline applications and the use of a browser (which means lock-in to the Google Suite editors). Dropbox Paper is less functional than Microsoft, Libre, and Google suites, but appears to be taking a Google approach, sad.

Scalability and the Requirements for State Management

A quick read of the EtherCalc story provide excellent insight into what it takes to maintain state in a connection-less multi-user environment. Essentially a copy of the document needs to be kept updated. That generally requires the same resources as on a workstation, and was likely a strong motivation for Google building from scratch simpler non-compatible file formats for Word and Excel documents, as well as a log of all change-sets (for version tracking). The same is said for the Pydio/LibreOffice Cloud offerings, namely that they take a bit of memory to get them to work, again due to the architectural requirements of real-time server-based state management.

Limitations of Paper as a First Class Editor

Since Dropbox already uses the Microsoft Online editors for Word and Excel documents (which work as advertised), Paper is a bit of an unwanted stepchild in terms of integration. Paper (which is both a file storage and a file editor, with web and mobile app versions) doesn't live within the Dropbox folder system, but rather has its own file system. This is awkward, for navigation, to say the least. Paper folders and files do not sync.

Offline Editing with Mobile, but not Web

With the IOS or Android App, Paper files can be edited offline, akin to Google Suite, but without the ability to do offline editing with the web app. Paper files can be exported in Markdown and Word document formats, but there is no ability to import Paper files, one has to copy/paste. This is trouble if someone has a lot of Markdown files already at hand. It seems clear that this use case is fairly well ignored.

Copy/Paste and AutoCorrect in Dropbox Paper

With copy/paste, another glaring problem comes to light, which is the required transformation of single and double quotes into their fancy quotes equivalents. This is a non-starter for people working with text that needs to remain inviolable in their originally intended ASCII characters. There are obvious work-arounds for the well-known ASCII and Unicode Quotation Marks problem, and a handy visual JavaScript replacement is one that would keep the underlying text unchanged, but that is not the path which Dropbox took on this editor. Rather like Microsoft Word and Google Docs -- but unlike them without the ability to turn off the AutoCorrect options -- characters are automatically replaced, not leaving a trace of what they were formerly.

Mo' Paper, Mo' Problems

Intimated above is that what went wrong is a lack of open source of the file format that Dropbox Paper uses, which gives it severe technical limitations in terms of portability, offline editing, file synchronization, and a clear separation between interface and file format. This approach is one which Google has also embraced, and Google's moderate success in the face of such technical limitations should not be a signal of the weakness of such limitations. Rather one should look at the years and years of massive resources poured into the cloud editing project, which still cannot do proper offline file synchronization, and which has allowed Microsoft to compete effectively after a very long delay in entering the market with cloud editors.

Conclusion: Ignore Dropbox Paper

For the particular use case mentioned above, requiring file format and content integrity, file synchronization, diversity of editors, and the like, the solution is to simply ignore Dropbox Paper. The product is not for me or others with my same requirements. Fair enough. For our needs: - Dropbox for file synchronization - Stackedit version 5 for web-based editing (with access to the Dropbox file system) - Editorial for IOS is a great plain text editor supporting Markdown and Fountain (screenplay formatting), and also includes workflow scripting with python. There's a book on doing workflow with Editorial. - Atom editor Updated needs (18-Sep-2018): - Google Drive (Gsuite) + Insync for Linux - for files only, no online editors - Text - Chrome Extension, Excellent - Caret - Chrome Extension, Also Excellent - Atom editor - Native Application, Cross-Platform - Libre Office - Clunky but Functional