TCP/IP Over USB

Setup
------

Android phone, rooted.
Linux host.
USB 2.0 Compliant connection between the above two devices.

Goal
-----

A client browser program running on the android phone should be able to load the home page from the web server running on the Linux host.

More information coming soon.

Session logging in Linux

Most of you must be familiar with the "who" command on linux, it lists the users logged in currently on the machine and shows a lot more information related to where they are logged in from etc. There are a lot more options the command supports including the usual "who am i" and "who mom likes" versions.
Basically what "who" does is parse a couple of files which are used for session logging, this post aims to explain these files and the structure of the records in these files. Typically on linux systems two files are used for session logging :

/var/run/utmp : Stores user session related information, a record is appended at each login and then wiped out on logout.
/var/log/wtmp : Stores a trail of the login/logout information and other such logging information for the system. The same record which is pushed to utmp is pushed here on login and then the same record with the "user" field zeroed in is pushed on logout.

The paths to the files can change and are available more generically as _PATH_UTMP and _PATH_WTMP defined in /usr/include/paths.h. Applications should use these definitions rather than hard coding the paths.

Linux for compatibility provides both the utmp and utmpx API for dealing with these log files, the utmpx API is an extension(and a parallel API) of the utmp API and was created in System V Release 4. Linux however does not create parallel utmpx and wtmpx files, all the information is available in the above mentioned two files only.

The records stored in each of the files is of the type "struct utmpx" defined in /usr/include/bits/utmpx.h as :

Note that to get some of the types for the fields in the structure it is important to declare _GNU_SOURCE in your program.

Each of the lines in "utmp" and "wtmp" files contains records of the above format. As I mentioned before Linux supports both the old utmp API and the utmpx API, if the utmp API is used then the records are read in as "struct utmp" which is defined in /usr/include/bits/utmp.h. The main difference between the two APIs is that the former has some re-entrant versions for functions which the latter(utmpx) does not. The various defined function calls are : 

One field which requires some attention above is the type field. Records in the files can be of various types,  : 

EMPTY - Invalid accounting information

RUN_LVL - Indicates a change in run-level during boot or shutdown (requires definition of _GNU_SOURCE)

BOOT_TIME - Contains the system boot time in ut_tv field. 

NEW_TIME - Contains the new time after a system clock change, recorded in the ut_tv field.

OLD_TIME - Contains the old time before a system clock change, similar to the NEW_TIME type record.

INIT_PROCESS - Signifies a process which has been spawned by the INIT process.
LOGIN_PROCESS - Record for a process like "login"
USER_PROCESS - Signifies a user session process started by the login program.
DEAD_PROCESS - Identifies a process that has exited (occurs on logout)

RUN_LVL and BOOT_TIME type records are written by "init" and these records are written to both the "utmp" file and the "wtmp" file.

Using the utmpx API you can read and write entries to these files directly. The good thing is that you need not directly open/close these files, calling the function "setutxent()" opens the file or rewinds the file pointer if already open. Similarly when we are done reading/writing, we can close the file using "endutxent()".
The functions available in the API to deal with the "utmp" file can be divided into three categories :

1. Getting entire records  using the getutxent() function which returns the entire record starting from the current file pointer location.
       struct utmpx * getutxent(void);
2. Searching for records based on given parameters, the parameters are passed in a struct utmpx pointer.
       struct utmpx * getutxid(const struct utmpx *ut);
       struct utmpx * getutxline(const struct utmpx *ut);
3. For putting a record to the file
       struct utmpx * pututxline(const struct utmpx *ut);

Functions of the type getutx* return a pointer to the utmpx record read or NULL if EOF is reached. The returned pointer points to a static area and can be cached on certain systems, that is if the search parameters match the result in the pointer returned by a previous call, then the same contents might be returned again. It is a good idea to zero the static area pointed to by this pointer. Also the pututxline() function may internally call the getutx* set of functions, but this will not affect the contents of the static area that the getutx* set of functions return a pointer to. The pututxline() functions returns a pointer to the record passed as the argument  on success and NULL on failure.
By default all the getutx* functions work on the "utmp" file, we can change this (for example to the "wtmp" file) using the following function :

int utmpxname (const char *file); 

Note that this function does not open the file and thus will not report any errors pertaining to invalid paths, such errors will show up only on the consequent calls to the other getutx* functions or the setutxent() function.
Also since updates to the "wtmp" files are always just simple appends (remember that records are never removed from this file), this can be achieved using the following wrapper call which opens the file, writes the record and then closes it :

void updwtmpx (char *wtmpx_file, struct utmpx *ut);

Some systems might instead provide the updwtmp(char *, struct utmp *) or the logwtmp (const char *, const char *, const char *) functions for updating the wtmp file.

There is another file on linux (and some unix implementations) which provides useful session information, this is "/var/log/lastlog" which contains logs of the logout time for users. The records in this file are of type "struct lastlog"  defined in /usr/include/bits/utmp.h (search for lastlog). This information is sometimes used to display the last time when you logged into a machine on login.

Here is some sample code which reads the "utmp" file and lists the user name on each record. The code does not use the API calls but reads the records directly from the file, (this is not the preferred way of doing it though, the code is just a proof-of-concept).





Apart from "who", there are other commands like "last" which display information extracted from the utmp/wtmp files.

All the above information (and more to come) is a result of the time I have spending with the "The Linux Programming Interface". Chapter 40 (Login Accounting) in the book provides quite a lot more details and sample code listings which use the utmpx API and do a lot more than the above sample code.

Memory at each node on a NUMA machine using libnuma (linux only)

For the past few days I have been occupied with trying to understand how NUMA works and what are related implications and challenges for system design. Linux supports more than a few memory location policies on a NUMA supported machine and a corresponding user space library to deal with these policies, migration of memory, thread location etc among other things. The library for most part seems to get information from the files in /proc/self and in /sys/devices/system/node. The latter location in the sys filesystem has a multitude of information pertaining to the memory on the system.
Here is a simple program I wrote toady which prints the memory located at each "defined" memory node on the NUMA system, this program uses the numa library libnuma. It is a rather trivial piece of code. My next task at hand (once I am done with the semester finals) would be to play around with my parallel implementation of QuickSort using pthreads and see if I can improve performance by manually playing with locality of memory and thread execution.

I did hit upon one important piece of information which I did not know before, that my laptop (DELL E6410) is NUMA compatible but has a single memory node with all the 8 Gigs of memory. This might as well be the reason why the parallel QuickSort worked faster on my laptop than on some of the department machines with distributed memory banks and faster truly multithreaded cores.

EDIT :

if you have the numactl package installed, you can also list numa related hardware information using the following command :

numactl --hardware

looking at the information you can tell that it is the same as what you would see under the various entries in the /sys/devices/system/node directory.

Quick everyday scripting in bash

From my experience, mostly in grading courses and everyday tasks on my linux box, the most common requirements for scripting deal with iterating over a given set of entities and performing some task, or iterating over a given folder's contents and performing some task. Here are some sample cases and solutions to performing the same quickly. (I use bash so I am sure all the following work in bash).

Case 1 : 


You have a program X and you want to verify that it runs correctly (lets assume running correctly means the same as the program X exiting with success) over, say 100 iterations.

Solution

for i in `seq 100`
do
    ./X [ args for X]
    if [ $? -ne 0 ]; then
        echo "X failed in iteration $i"
        exit 1
    fi
done

The construct `...` runs the command inside and is replaced by the output of that command. the "basename" command gives you the last literal in a path.
The special variable $? stores the exit code of the previous program that was run in the shell script, this is typically what you return at the end of the main() function in a C program. A return code of 0 stands for success.
Note that the above snippet will work assuming that you are in the directory which contains the file (program) X.

Case 2 : 

A folder X has contents a, b, c, d, e, f, g , each of which is a folder. You want to create a file [parent_folder_name].txt in each of those folders.

for file in X/*
do
    touch $file/$(basename $file).txt
done

The construct $(...) runs the command inside and is replaced by the output of that command. the "basename" command gives you the last literal in a path. so

$ basename X/a
a
$ basename X/b
b

A slight modification to case 2, if X has both files and directories and you want to run the operation only for directories then you can modify the above snippet to check if the entry is a file/directory and act accordingly :


for file in X/*
do
    if [ -d $file ]; then
        touch $file/$(basename $file).txt
    fi
done

Note that the above two snippets (for Case 2) will work assuming that your current working directory is the parent of directory X.

Case 3 :

Execute a command via ssh on a set of remote machines, whose hostnames are in lexicographic order. For example the CS department here at purdue has the following set of machines : 

sslab01-sslab21 and suppose I want to run the command "who" on each one of them.

for i in `seq 21`

do

    if [ $i -lt 10 ]; then

        echo "Running 'who' on sslab0$i"

        ssh sslab0$i "who"

    else

        echo "Running 'who' on sslab$i"

        ssh sslab$i "who"

    fi

done

The if conditions in the above snippet are required because, by default the "seq"  does not output numbers of equal width, that is , it does not pad zeros at the beginning. However if you pass the "-w" option to seq, you can simplify the above snippet to : 

for i in `seq -w 21`

do

    echo "Running 'who' on sslab$i"

    ssh sslab$i "who"

done

The "-w" option ensures that `seq` prints its output in fixed width, padding the number with zeroes when ever necessary.

I will edit the post and add more common scenarios as I come across new ones, but all the above snippets are short and once you get hold of things, you can actually type them in on the prompt whenever you want rather than storing them in a script file and running that file. 

Adding Scripts to the PATH

However if you do want to run them as a script (from a file), it is a better option to do the following : 

1. if  ! [ -d ~/bin ]; then mkdir ~/bin; fi

2. touch ~/bin/script_I_run_often

3. Add the snippet to the script

4. chmod +x ~/bin/script_I_run_often

5. add "export PATH=$PATH:~/bin" to ~/.bashrc

Now you can just run the script as : 

$ script_I_run_often

Other Special Variables

Some other special variables which are very useful in everyday scripting are : 

$# : the number of arguments passed to the script, you can check this variable to see if your script got the correct number of arguments or not 

$1, $2, $3 ... : are the arguments passed to the script

$$ : Pid of the current process

NOTE : None of the information mentioned above is "new" or "novel", the post is just meant to serve as a compilation of some quick bash tricks. I am sure many other such compilations exist on the internet.

[Fixed] Blue Tint on Youtube videos (flash problem) on Centos 6.x

I have the adobe flash plugin from the adobe repository with the following version installed on my laptop running Centos  :


flash-plugin.x86_64                    11.2.202.233-release    @adobe-linux-x86_64

Since the last update, the videos on Youtube have mostly had a blue tint and less often a pale orange shade. After a couple of days of not trying to fix it (thanks to the hectic semester) I guessed this was a problem with flash or something to do with how flash uses the GPU, since those are the only two pieces of proprietary software on my laptop (and thus would be expected to be fixed rather slow for a problem so evident). 

Finally, a little googling around showed this post on the Arch Linux forums : https://bbs.archlinux.org/viewtopic.php?pid=1084648

I just followed whatever was mentioned there, and the flash plugin looks far more stable now. Here are the steps I followed on my Centos 6.x machine : 

1. Created the file /etc/adobe/mms.cfg and added the following contents : 

#Hardware video decoding

EnableLinuxHWVideoDecode=1

OverrideGPUValidation=true

2. Added the following line at the beginning of  /etc/X11/xinit/xinitrc-common

export VDPAU_NVIDIA_NO_OVERLAY=1

You will need root permissions for both of the above, an alternative for the second step can be adding the same line to the user specific xinitrc file (~/.xinitrc). I logged out and logged in again after the changes, the blue tint is gone and the flash plugin itself is far more stable than before. As suggested in the thread on the arch linux forums, it might be a good idea to try a free alternative to the adobe flash player plugin if possible.

PS : 

1. I run Centos 6.x on my Dell E6410, which has a NVIDIA NVS 3100M gpu.

2. It is just amazing the thread which showed up was on the arch linux forums, Arch Linux continues to amaze me.