Developer's Note

// shall I write some keywords here to boost search engine ranking?

Saturday, January 15, 2022

Command to Compare Data Files: comm

`diff` is great command to compare the changes among 2 versions of a file, but for comparing data file and extract data from it, `comm` command is more suitable for the job.

For demo purpose, we will prepare 2 sorted data file as below:

left.txt:


123
456
789

right.txt:


789
888
999

`comm` command work by suppressing lines that exist in one of the file, or both file, where -1 suppress lines unique to the left file, -2 suppress lines unique to the right file, and -3 suppress lines appear in both files.

if no options being specify, it will display data in 3 columns:

> comm left.txt right.txt
123
456
                789
        888
        999

To show only lines that unique to left file, we shall suppress the line unique to right file, and line that appear in both files:

> comm -23 left.txt right.txt
123
456

For showing lines that only appear on the right, we will surpress line unique to the left file, and those appear in both files:

> comm -13 left.txt right.txt
888
999

To extract lines that appear in both files, just suppress line that unique to left and right file:

> comm -12 left.txt right.txt
789

And to exlcude only line that appear on the left, command below will show data in 2 column, one of it is the line commons for both files:

> comm -1 left.txt right.txt
        789
888
999

We believe by now you shall be able to use `comm` command for comparing data file effectively. Thanks for reading.

Monday, August 09, 2021

Linux Scripts to Create Table Partition for Next Month Automatically

 In this article we are going to show you a method of create new monthly partition for MySQL table without stored procedure, with the REORGANIZE PARTITION from MySQL/MariaDB.

(Click here to skip the article and straight to the scripts.)

For demo purpose, below are the DML of the table to be partitioned, and the ALTER statement to start with 2 partitions: "p_old" and "p_future". Please change the p_old partition date to the first of next month.


Below is the template of ALTER statement to REORGANIZE PARTITION "p_future" into a partition of next month ('p_202109' for example, as of writing), and 'p_future' for records after next month.


And here is the shell script that did some simple string substitute based on the template, and execute it on your MySQL/MariaDB server. Please do not forget to change the username, password and database name to match you database server.


Finally, you may run this each month to create a new partition for next month with command:

./create_partition.sh

If you are to schedule it via crontab, ensure you modify the script to use full path for the template file.

Thank you.

Monday, August 26, 2019

Offload your SSD (on Windows) and SD Card (on Linux)

As many of us aware, both SSD and SD Card wearing out upon write operation. So to extend life span of the storage, we may look into reduce the write operation.

(With wear leveling, most modern SSD have lifespan that is long enough where you might want to skip this post. But no harm to know something new, right ?)

For SSD, you may consider putting the Windows's TEMP folder into a RAM disk. RAM disk, as the name implies, it is a virtual drive that reserve and utilise the space on your RAM. There are many RAM disk software out there, and I'm using ImDisk's RamDisk Configuration.

You can define the space of the RAM disk, and set the TEMP folders to the RAM Disk with this tool:

(ImDisk RamDisk Configuration)


For SD Card that store your OS (i.e. Raspberry Pi), we will put the `/var/log` on RAM with log2ram. This will make the logs stay in RAM and only flush to storage once a day by default (configurable).

After done installation and reboot your Raspberry Pi, you shall see the screen below:

(log2ram: new mounting point after installation and restart)
This shall extend the lifespan of your SD Card.

Thanks for reading.

Friday, August 16, 2019

Get 30GB of Samba Storage on Cloud

OK, it is less than 30GB to be exact, as the OS took some space too.

The whole idea started from the offer from Google Cloud with an "always free" plan which include 1 f1-micro instance with 30GB standard persistent storage. (Anyway, you might still get charge for egress too)

So if you already have some running VPS or cloud instance, you may just use that too.

However, as you might experienced, most cloud provider have strong focus on network security. And the steps to define firewall rules, etc are different among providers.

Fortunately, ZeroTier come into rescue. We can use ZeroTier to link our home PC with the cloud instance just like there are in the same LAN (or Global Area Network, as advertise by ZeroTier).

So here go the list of guide that I had followed, hope you enjoy it:


Additional Ideas:

  • If you are have Windows 10 Pro at home, you may also consider to use NFS too.
  • You may experiment on mount object storage to the VM via FUSE for additional storage. (Do share with me the result and experience)

Wednesday, April 29, 2015

Chrome unable to load Google sites

Recently I had been facing problem loading Google sites such as its search engine, Gmail, Youtube, etc.


It is either take extra ordinary long time to load, or just failed with `ERR_QUIC_PROTOCOL_ERROR`.

I did a search and found I am not alone, and below are the solution [1]:

  1. Open a new tab in Chrome, and type `chrome://flags/` into address bar, then press Enter.
  2. Search for `Experimental QUIC protocol`
  3. Disable it.
  4. Restart Chrome. Done.
It had been about a week, and the error never happen again.

[1]: https://productforums.google.com/forum/#!topic/chrome/xKFfhDvhjU8