Tag Archives: Scripting

Computing, DevOps, Knowledge & Reference, Open Source, Productivity

Tiny linux disk usage reporting script – dux.bash

Leave a comment

I decided to write a very small disk usage reporting script that provides some extra information than does just using ‘du‘ directly.  The script of course uses du, among other command line tools and parsing commands to generate the little report.  It suits my needs for the moment. Hopefully others out there will also find it useful.

Firstly, here is the script output to give you an idea of how to execute it, what it does, and how it looks:

$ ./dux.bash /home/jbl /home/jbl
Building extended du reports for /home/jbl in /home/jbl ...
[du extended report]:
13G     ./Personal
4.7G    ./DATA
2.4G    ./Pictures
1.4G    ./Downloads
Total (GB): 21.5
 
373M    ./core
260M    ./tmp
73M     ./game-saves
37M     ./new-music
33M     ./new-books
32M     ./vim-env
24M     ./random-tools
15M     ./stuff
Total (MB): 847

The script takes two arguments, the directory you want to analyze, and the directory where you want to store the reports.

As you will see, the script provides me an at-a-glance look at the following characteristics of a particular directory structure:

  • Grouping and separation of larger (GB sized) directories from smaller (MB sized) directories.
  • Directories sorted by size in each group.
  • Total sizes for each group of files and directories.

With this output I can see clearly which directories are causing the most contention, and how much of an impact they have compared to other directories.

The script is very crude, and probably needs some work and error correction (accounting for files off or root, etc.)   It also creates some temporary text files (used to construct the report), which is the reason for the second argument to the script.  However for now it’s doing what I need, so I figure it’s worth sharing.  Here it is:

#!/bin/bash
echo "Building extended du reports for $1 in $2 ...";
cd $1
du -sh $1/* > $2/du-output.txt
cat $2/du-output.txt | egrep '([0-9][0-9]M)' > ~jbl/du-output-MB.txt
cat $2/du-output.txt | egrep '[0-9]G'> ~jbl/du-output-GB.txt
cat $2/du-output-MB.txt | sort -hr > $2/du-output-MB-sorted.txt
cat $2/du-output-GB.txt | sort -hr > $2/du-output-GB-sorted.txt
echo '[du extended report]:';
cat $2/du-output-GB-sorted.txt
echo -ne "Total (GB): " && cat ~jbl/du-output-GB-sorted.txt | perl -pe 's/^(\d+\.+\d+|\d+)\w*.*/$1/g' | paste -sd+ | bc
echo ""
cat $2/du-output-MB-sorted.txt
echo -ne "Total (MB): " && cat ~jbl/du-output-MB-sorted.txt | perl -pe 's/^(\d+\.+\d+|\d+)\w*.*/$1/g' | paste -sd+ | bc

I’m not sure what more I will need from it going forward, so it may not get much love in the way of improvements.  Since it would be nice to have it on each system I’m using, I may convert it to a single script that has no need to generate temporary text files.  

Nuff said!  Hopefully you find either the script, or portions of it useful in some way!

For convenience, I’ve also added this script as a public gist so you can download dux.bash from github.

Cheers!

Computing, Knowledge & Reference, Open Source, Perl

Examples of recursion, in Perl, Ruby, and Bash

Leave a comment

This article is in response to the following question posted in the Perl community group on  LinkedIn:

I’m new to PERL and trying to understand recursive subroutines. Can someone please explain with an example (other than the factorial ;) ) step by step, how it works? Thanks in Advance.

Below, are some very simplified code examples in Perl, Ruby, and Bash.

A listing of the files used in these examples:

blopez@blopez-K56CM ~/hello_scripts 
$ tree .
 ├── hello.pl
 ├── hello.rb
 └── hello.sh
0 directories, 3 files
blopez@blopez-K56CM ~/hello_scripts $


Recursion example using Perl:
– How the Perl script is executed, and it’s output:

blopez@blopez-K56CM ~/hello_scripts $ perl hello.pl "How's it going!"
How's it going!
How's it going!
How's it going!
^C
blopez@blopez-K56CM ~/hello_scripts $

– The Perl recursion code:

#!/usr/bin/env perl
use Modern::Perl;
my $status_update = $ARGV[0]; # get script argument
 
sub hello_world
{
    my $status_update = shift; # get function argument
    say "$status_update";
    sleep 1; # sleep, or eventually crash your system
    &hello_world( $status_update ); # execute myself with argument
}
 
&hello_world( $status_update ); # execute function with argument


Recursion example using Ruby:
– How the Ruby script is executed:

blopez@blopez-K56CM ~/hello_scripts 
$ ruby hello.rb "Doing great!"
Doing great!
Doing great!
Doing great!
^Chello.rb:7:in `sleep': Interrupt
    from hello.rb:7:in `hello_world'
    from hello.rb:8:in `hello_world'
    from hello.rb:8:in `hello_world'
    from hello.rb:11:in `'
blopez@blopez-K56CM ~/hello_scripts $

Note: In Ruby’s case, stopping the script with CTRL-C returns a bit more debugging information.

– The Ruby recursion code:

#!/usr/bin/env ruby
status = ARGV[0] # get script argument
 
def hello_world( status ) # define function, and get script argument
    puts status
    sleep 1 # sleep, or potentially crash your system
    return hello_world status # execute myself with argument
end
 
hello_world status # execute function with argument

Recursion example using Bash:

– How the Bash script is executed:

blopez@blopez-K56CM ~/hello_scripts $ bash hello.sh "..nice talking to you."
..nice talking to you.
..nice talking to you.
..nice talking to you.
^C
blopez@blopez-K56CM ~/hello_scripts $

– The Bash recursion code:

#!/usr/bin/env bash
 
mystatus=$1 # get script argument
 
hello_world() {
    mystatus=$1 # get function argument
    echo "${mystatus}"
    sleep 1 # breath between executions, or crash your system
    hello_world "${mystatus}" # execute myself with argument
}
 
hello_world "${mystatus}" # execute function with argument
Computing

Back to Basics – Very Simple Log Monitoring with Perl

Leave a comment

multitail_apache

There are many many tools out there which allow you to monitor and view your system or networking logs in several different ways.  Sometimes though, you may find yourself looking for a specific feature that none of these tools currently provide.  Whenever your goals are very specific, and you don’t want to use a big feature-full program to accomplish a simple task, you may want to consider writing your own tool.

Below is a simple Perl program I wrote which does just that. All the requirements of the program are within the script itself (using the __DATA__ handle at the bottom of the file). The only thing you may need to install on your system to get this to work is the File::Tail CPAN package.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

#!/usr/bin/perl -w
 
use strict;
use File::Tail;
 
my @patterns = <data>;
my $file = File::Tail-&gt;new ("/var/log/syslog");
while ( defined(my $line=$file-&gt;read) )
{
    my $match = &amp;filter($line);
    if ( $match eq "no" )
    {
        print $line;
    }
}
 
sub filter ()
{
    my $line = $_[0];
    my $match = "no";
    foreach my $test (@patterns)
    {
        chomp($test);
        if ( $line =~ m/$test/ )
        {
            $match = "yes";
        }
    }
    return $match;
}
 
__DATA__
PROTO=UDP SPT=67 DPT=68
ACCEPT IN=br0 OUT=vlan1 src=192.168.0.111.*PROTO=TCP.*DPT=80
ACCEPT IN=br0 OUT=vlan1 src=192.168.0.102.*PROTO=TCP.*DPT=80
ACCEPT IN=br0 OUT=vlan1 src=192.168.0.111.*PROTO=TCP.*DPT=443
ACCEPT IN=br0 OUT=vlan1 src=192.168.0.102.*PROTO=TCP.*DPT=443
ACCEPT IN=vlan1 OUT=br0.*DST=192.168.0.101.*PROTO=UDP.*DPT=1755
ACCEPT IN=vlan1 OUT=br0.*DST=192.168.0.101.*PROTO=TCP.*DPT=1755
ACCEPT IN=br0 OUT=vlan1 src=10.100.0.1.*PROTO=UDP.*DPT=53
JBLLNXWKS dhclient
 
__END__
</data>

This program will monitor the end of the file (like the Unix ‘tail’ command) and check for new log entries. When it detects new lines in the log, it will filter those lines with the patterns defined at the end of the script (under __DATA__) and display anything it detects in the logs except those filter lines.

You’ll probably notice that the filter lines are regular expressions, which makes this script more powerful than doing filtering by simple full-string comparison.

Aside from simply printing the output to STDOUT, you could use regular expressions to pop pieces of each line into an array or hash, in order to do calculations, such as how many entries had a source IP of X, or destination port of Y, etc.

Its definitely a good thing to keep in mind that whatever software you could possibly need is already out there on the internet, and possibly open source. However, its also good to keep in mind that YOU can create a tool yourself to accomplish your specific task; all it takes is a little self-confidence, effort, and patience.