Session logging in Linux

Most of you must be familiar with the "who" command on linux, it lists the users logged in currently on the machine and shows a lot more information related to where they are logged in from etc. There are a lot more options the command supports including the usual "who am i" and "who mom likes" versions.
Basically what "who" does is parse a couple of files which are used for session logging, this post aims to explain these files and the structure of the records in these files. Typically on linux systems two files are used for session logging :

/var/run/utmp : Stores user session related information, a record is appended at each login and then wiped out on logout.
/var/log/wtmp : Stores a trail of the login/logout information and other such logging information for the system. The same record which is pushed to utmp is pushed here on login and then the same record with the "user" field zeroed in is pushed on logout.

The paths to the files can change and are available more generically as _PATH_UTMP and _PATH_WTMP defined in /usr/include/paths.h. Applications should use these definitions rather than hard coding the paths.

Linux for compatibility provides both the utmp and utmpx API for dealing with these log files, the utmpx API is an extension(and a parallel API) of the utmp API and was created in System V Release 4. Linux however does not create parallel utmpx and wtmpx files, all the information is available in the above mentioned two files only.

The records stored in each of the files is of the type "struct utmpx" defined in /usr/include/bits/utmpx.h as :


/* The structure describing an entry in the user accounting database.  */
struct utmpx
{
            short int ut_type;            /* Type of login.  */
            __pid_t ut_pid;               /* Process ID of login process.  */
            char ut_line[__UT_LINESIZE];  /* Devicename.  */
            char ut_id[4];                /* Inittab ID. */
            char ut_user[__UT_NAMESIZE];  /* Username.  */
            char ut_host[__UT_HOSTSIZE];  /* Hostname for remote login.  */
            struct __exit_status ut_exit; /* Exit status of a process marked as DEAD_PROCESS.  */


          /* The fields ut_session and ut_tv must be the same size when compiled
             32- and 64-bit.  This allows files and shared memory to be shared
             between 32- and 64-bit applications.  */
          #if __WORDSIZE == 64 && defined __WORDSIZE_COMPAT32
                      __int32_t ut_session;         /* Session ID, used for windowing.  */
                      struct
                      {
                                  __int32_t tv_sec;           /* Seconds.  */
                                  __int32_t tv_usec;          /* Microseconds.  */
                      } ut_tv;                      /* Time entry was made.  */
          #else
                      long int ut_session;          /* Session ID, used for windowing.  */
                      struct timeval ut_tv;         /* Time entry was made.  */
          #endif
            __int32_t ut_addr_v6[4];      /* Internet address of remote host.  */
            char __unused[20];            /* Reserved for future use.  */
};

Note that to get some of the types for the fields in the structure it is important to declare _GNU_SOURCE in your program.

Each of the lines in "utmp" and "wtmp" files contains records of the above format. As I mentioned before Linux supports both the old utmp API and the utmpx API, if the utmp API is used then the records are read in as "struct utmp" which is defined in /usr/include/bits/utmp.h. The main difference between the two APIs is that the former has some re-entrant versions for functions which the latter(utmpx) does not. The various defined function calls are : 

One field which requires some attention above is the type field. Records in the files can be of various types,  : 

EMPTY - Invalid accounting information

RUN_LVL - Indicates a change in run-level during boot or shutdown (requires definition of _GNU_SOURCE)

BOOT_TIME - Contains the system boot time in ut_tv field. 

NEW_TIME - Contains the new time after a system clock change, recorded in the ut_tv field.

OLD_TIME - Contains the old time before a system clock change, similar to the NEW_TIME type record.

INIT_PROCESS - Signifies a process which has been spawned by the INIT process.
LOGIN_PROCESS - Record for a process like "login"
USER_PROCESS - Signifies a user session process started by the login program.
DEAD_PROCESS - Identifies a process that has exited (occurs on logout)

RUN_LVL and BOOT_TIME type records are written by "init" and these records are written to both the "utmp" file and the "wtmp" file.

Using the utmpx API you can read and write entries to these files directly. The good thing is that you need not directly open/close these files, calling the function "setutxent()" opens the file or rewinds the file pointer if already open. Similarly when we are done reading/writing, we can close the file using "endutxent()".
The functions available in the API to deal with the "utmp" file can be divided into three categories :

1. Getting entire records  using the getutxent() function which returns the entire record starting from the current file pointer location.
       struct utmpx * getutxent(void);
2. Searching for records based on given parameters, the parameters are passed in a struct utmpx pointer.
       struct utmpx * getutxid(const struct utmpx *ut);
       struct utmpx * getutxline(const struct utmpx *ut);
3. For putting a record to the file
       struct utmpx * pututxline(const struct utmpx *ut);

Functions of the type getutx* return a pointer to the utmpx record read or NULL if EOF is reached. The returned pointer points to a static area and can be cached on certain systems, that is if the search parameters match the result in the pointer returned by a previous call, then the same contents might be returned again. It is a good idea to zero the static area pointed to by this pointer. Also the pututxline() function may internally call the getutx* set of functions, but this will not affect the contents of the static area that the getutx* set of functions return a pointer to. The pututxline() functions returns a pointer to the record passed as the argument  on success and NULL on failure.
By default all the getutx* functions work on the "utmp" file, we can change this (for example to the "wtmp" file) using the following function :

int utmpxname (const char *file); 

Note that this function does not open the file and thus will not report any errors pertaining to invalid paths, such errors will show up only on the consequent calls to the other getutx* functions or the setutxent() function.
Also since updates to the "wtmp" files are always just simple appends (remember that records are never removed from this file), this can be achieved using the following wrapper call which opens the file, writes the record and then closes it :

void updwtmpx (char *wtmpx_file, struct utmpx *ut);

Some systems might instead provide the updwtmp(char *, struct utmp *) or the logwtmp (const char *, const char *, const char *) functions for updating the wtmp file.

There is another file on linux (and some unix implementations) which provides useful session information, this is "/var/log/lastlog" which contains logs of the logout time for users. The records in this file are of type "struct lastlog"  defined in /usr/include/bits/utmp.h (search for lastlog). This information is sometimes used to display the last time when you logged into a machine on login.

Here is some sample code which reads the "utmp" file and lists the user name on each record. The code does not use the API calls but reads the records directly from the file, (this is not the preferred way of doing it though, the code is just a proof-of-concept).



#include <utmpx.h>
#include <paths.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
int main(int argc, char **argv){
        struct utmpx data;
        FILE *log_file = fopen(_PATH_UTMP, "r");
        memset(&data, 0, sizeof(struct utmpx));


        while(fread(&data, sizeof(struct utmpx), 1, log_file) == 1){
                printf("Read a record , User : %s\n", data.ut_user);
        }
        fclose(log_file);
        return EXIT_SUCCESS;
}

Apart from "who", there are other commands like "last" which display information extracted from the utmp/wtmp files.

All the above information (and more to come) is a result of the time I have spending with the "The Linux Programming Interface". Chapter 40 (Login Accounting) in the book provides quite a lot more details and sample code listings which use the utmpx API and do a lot more than the above sample code.

Memory at each node on a NUMA machine using libnuma (linux only)

For the past few days I have been occupied with trying to understand how NUMA works and what are related implications and challenges for system design. Linux supports more than a few memory location policies on a NUMA supported machine and a corresponding user space library to deal with these policies, migration of memory, thread location etc among other things. The library for most part seems to get information from the files in /proc/self and in /sys/devices/system/node. The latter location in the sys filesystem has a multitude of information pertaining to the memory on the system.
Here is a simple program I wrote toady which prints the memory located at each "defined" memory node on the NUMA system, this program uses the numa library libnuma. It is a rather trivial piece of code. My next task at hand (once I am done with the semester finals) would be to play around with my parallel implementation of QuickSort using pthreads and see if I can improve performance by manually playing with locality of memory and thread execution.



#include <numa.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char **argv){

        long free;
        int num_cpus = 4;
        long node_size = -1L;
        int i;

        if(numa_available() == -1){
                printf("ERROR : Numa not available on the system\n");
                return EXIT_FAILURE;
        }
        num_cpus = numa_num_configured_cpus();
        if(numa_num_configured_nodes() != num_cpus){
                printf("INFO : The number of thread nodes [%d] is different from the number of thread CPUs [%d]\n",
                                numa_num_configured_nodes(),
                                num_cpus);
        }
        else{
                printf("INFO : The number of thread nodes and thread CPUs is [%d]\n",
                        num_cpus);
        }
        for(i=0;i<num_cpus; i++){
                node_size = numa_node_size(i, &free);
                printf("INFO : CPU [%d] : Memory Available [%ld] MB, Free Memory [%ld] MB\n",
                                i,
                                node_size/(1024*1024),
                                free/(1024*1024));
        }

        return EXIT_SUCCESS;
}

I did hit upon one important piece of information which I did not know before, that my laptop (DELL E6410) is NUMA compatible but has a single memory node with all the 8 Gigs of memory. This might as well be the reason why the parallel QuickSort worked faster on my laptop than on some of the department machines with distributed memory banks and faster truly multithreaded cores.

EDIT :

if you have the numactl package installed, you can also list numa related hardware information using the following command :

numactl --hardware

looking at the information you can tell that it is the same as what you would see under the various entries in the /sys/devices/system/node directory.

Quick everyday scripting in bash

From my experience, mostly in grading courses and everyday tasks on my linux box, the most common requirements for scripting deal with iterating over a given set of entities and performing some task, or iterating over a given folder's contents and performing some task. Here are some sample cases and solutions to performing the same quickly. (I use bash so I am sure all the following work in bash).

Case 1 : 


You have a program X and you want to verify that it runs correctly (lets assume running correctly means the same as the program X exiting with success) over, say 100 iterations.

Solution

for i in `seq 100`
do
    ./X [ args for X]
    if [ $? -ne 0 ]; then
        echo "X failed in iteration $i"
        exit 1
    fi
done

The construct `...` runs the command inside and is replaced by the output of that command. the "basename" command gives you the last literal in a path.
The special variable $? stores the exit code of the previous program that was run in the shell script, this is typically what you return at the end of the main() function in a C program. A return code of 0 stands for success.
Note that the above snippet will work assuming that you are in the directory which contains the file (program) X.

Case 2 : 

A folder X has contents a, b, c, d, e, f, g , each of which is a folder. You want to create a file [parent_folder_name].txt in each of those folders.

for file in X/*
do
    touch $file/$(basename $file).txt
done

The construct $(...) runs the command inside and is replaced by the output of that command. the "basename" command gives you the last literal in a path. so

$ basename X/a
a
$ basename X/b
b

A slight modification to case 2, if X has both files and directories and you want to run the operation only for directories then you can modify the above snippet to check if the entry is a file/directory and act accordingly :


for file in X/*
do
    if [ -d $file ]; then
        touch $file/$(basename $file).txt
    fi
done

Note that the above two snippets (for Case 2) will work assuming that your current working directory is the parent of directory X.

Case 3 :

Execute a command via ssh on a set of remote machines, whose hostnames are in lexicographic order. For example the CS department here at purdue has the following set of machines : 

sslab01-sslab21 and suppose I want to run the command "who" on each one of them.

for i in `seq 21`

do

    if [ $i -lt 10 ]; then

        echo "Running 'who' on sslab0$i"

        ssh sslab0$i "who"

    else

        echo "Running 'who' on sslab$i"

        ssh sslab$i "who"

    fi

done

The if conditions in the above snippet are required because, by default the "seq"  does not output numbers of equal width, that is , it does not pad zeros at the beginning. However if you pass the "-w" option to seq, you can simplify the above snippet to : 

for i in `seq -w 21`

do

    echo "Running 'who' on sslab$i"

    ssh sslab$i "who"

done

The "-w" option ensures that `seq` prints its output in fixed width, padding the number with zeroes when ever necessary.

I will edit the post and add more common scenarios as I come across new ones, but all the above snippets are short and once you get hold of things, you can actually type them in on the prompt whenever you want rather than storing them in a script file and running that file. 

Adding Scripts to the PATH

However if you do want to run them as a script (from a file), it is a better option to do the following : 

1. if  ! [ -d ~/bin ]; then mkdir ~/bin; fi

2. touch ~/bin/script_I_run_often

3. Add the snippet to the script

4. chmod +x ~/bin/script_I_run_often

5. add "export PATH=$PATH:~/bin" to ~/.bashrc

Now you can just run the script as : 

$ script_I_run_often

Other Special Variables

Some other special variables which are very useful in everyday scripting are : 

$# : the number of arguments passed to the script, you can check this variable to see if your script got the correct number of arguments or not 

$1, $2, $3 ... : are the arguments passed to the script

$$ : Pid of the current process

NOTE : None of the information mentioned above is "new" or "novel", the post is just meant to serve as a compilation of some quick bash tricks. I am sure many other such compilations exist on the internet.

[Fixed] Blue Tint on Youtube videos (flash problem) on Centos 6.x

I have the adobe flash plugin from the adobe repository with the following version installed on my laptop running Centos  :


flash-plugin.x86_64                    11.2.202.233-release    @adobe-linux-x86_64

Since the last update, the videos on Youtube have mostly had a blue tint and less often a pale orange shade. After a couple of days of not trying to fix it (thanks to the hectic semester) I guessed this was a problem with flash or something to do with how flash uses the GPU, since those are the only two pieces of proprietary software on my laptop (and thus would be expected to be fixed rather slow for a problem so evident). 

Finally, a little googling around showed this post on the Arch Linux forums : https://bbs.archlinux.org/viewtopic.php?pid=1084648

I just followed whatever was mentioned there, and the flash plugin looks far more stable now. Here are the steps I followed on my Centos 6.x machine : 

1. Created the file /etc/adobe/mms.cfg and added the following contents : 

#Hardware video decoding

EnableLinuxHWVideoDecode=1

OverrideGPUValidation=true

2. Added the following line at the beginning of  /etc/X11/xinit/xinitrc-common

export VDPAU_NVIDIA_NO_OVERLAY=1

You will need root permissions for both of the above, an alternative for the second step can be adding the same line to the user specific xinitrc file (~/.xinitrc). I logged out and logged in again after the changes, the blue tint is gone and the flash plugin itself is far more stable than before. As suggested in the thread on the arch linux forums, it might be a good idea to try a free alternative to the adobe flash player plugin if possible.

PS : 

1. I run Centos 6.x on my Dell E6410, which has a NVIDIA NVS 3100M gpu.

2. It is just amazing the thread which showed up was on the arch linux forums, Arch Linux continues to amaze me.

Converting a space delimited sentence into a sequence of null terminated strings

The title says it all, you are given a string which consists of words separated by spaces, here is a snippet of code for converting the same into a sequence of null terminated strings and then later printing them. Returning the indices in the original string where the words begin is the key here :



int * break_by_spaces(char * arg){
        if(strlen(arg) <= 0 || (strlen(arg) ==1 && *arg == ' ')){
                return NULL;
        }
        int * retval = malloc((strlen(arg) -1)*sizeof(int));
        int * temp = retval;
        int counter = 0;
        if(retval == NULL){
                perror("malloc");
                exit(EXIT_FAILURE);
        }
        memset(retval, 0, (strlen(arg) -1)*sizeof(int));
        *temp = counter;
        arg++;
        counter++;
        while(*arg){
                if(*arg == ' '){
                        *(++temp) = counter+1;
                        *arg = 0;
                }
                arg++;
                counter++;
        }
        *(++temp) = -1;
        return retval;
}


Printing the strings is pretty trivial as :

int * string_indices= break_by_spaces("A String of words");

for(i=0; *(string_indices + i) != -1; i++){

                fprintf(stdout, "[%d] %d %s \n", i, *(string_indices + i), (char *)(argv[1] + *(string_indices + i)));

}

Quick code, picked up the hack from a certain shell command parsing code in Xinu.

Intents and Activities in Android

The android UI is built around the concept of Activities. Each screen you see when you use an android app is an activity. And as usual, each activity has a layout that in programming methodology can only be accessed by the thread which set/created the layout. Android however has better method for creating activities and getting back information from them than purely language oriented methods (like global variables, shared objects etc). Android defines the notion of an intent, which can be used to give some information to an Activity and also get back information from an activity as an intent. The general code flow looks like the following :

create an intent and start the activity :

 final int result = 1;
Intent intent = new Intent(VoipChat.this, UserSettings.class);
startActivityForResult(intent, result);

Add code in the event handler when an activity returns ( you need to check for the specific return code which is the same as result)

   public void onActivityResult(int requestCode, int resultCode, Intent data) {
                 if (resultCode == Activity.RESULT_OK && requestCode == 1) {
                                    //Do some work here
                   }
   }

In the activity class which is started, make sure you put the necessary data in a new intent before calling the finish() method. The code in UserSettings.java for example would look something like :

   Bundle bundle = new Bundle();
Log.d("USERSETTINGS", "Returning : "+userName.getText().toString());
bundle.putString("UserName", userName.getText().toString());
data.putExtras(bundle);
this.setResult(RESULT_OK, data);
finish();

Accordingly the code inside the if block above would look something like :
                 
   String result = data.getExtras().getString("UserName");

note that data is the intent variable in the arguments for the onActivityResult() method.

NOTE : This blog post contains very specific information about the android sdk and included libraries. Unlike most of the other posts on this blog, it may not be of interest to the general readers.

Disabling auto indentation for code pasting in vim

If you are a vim user and have automatic indentation enabled, then you have sure been through the trouble of trying to paste indented code and messing up the indentation completely. There is however a fix for this , you can temporarily disable code indentation when you are pasting text. Add the following to your vim configuration file (mostly ~/.vimrc) :

set pastetoggle=<key sequence>

You can set whatever key sequence you want and if you do not have any other mapping for that key sequence, you just need to hit the key sequence and then "i" for insert and paste the text. Insert will generally show you the mode, for example if you have paste mode enabled, you will see

-- INSERT (paste) --

You can disable the paste mode using the same sequence as above. On my machine I have the sequence  set to <leader><C-p>, (my leader key is bound to the default ",")

set pastetoggle=<leader><C-p>

So hitting, "," and then Ctrl-p toggles the mode on vim on my machine.

Per host username configuration for ssh

Using ssh to access many remote servers is a common scenario and more often the user names on these servers might be different from the one on the local machine and/or different among themselves. Ssh allows one to set the default username to use per host. You need to edit either the user specific configuration file (~/.ssh/config) or the system wide configuration file (/etc/ssh/ssh_config) and add the following :

1     Host <hostname>
2          User <username>


You can add as many entries as you want like above and the next time you can just do

$ ssh <hostname>

and ssh will try to log you in with the default username specified for that host in the configuration file.
More information on ssh can be found on the ssh man page (man ssh) and more information about the ssh configuration options can be found on the ssh_config man page(man ssh_config). (BTW, setting up passphrase-less ssh authentication is suggested if you connect to the remote servers very often and if security is not of high priority).

Letting it go ...

No system is efficient, there are memory leaks, there are application freezes, and some never work at all, you cannot keep a system up for ever, even the most resilient of servers have to be rebooted once in a while.
It so happens in life too, you need to let it go, to forget all the things that you have seen, heard and been through so far, this is not a foolish blog post with the constant onslaught of 'a' someone's thoughts in the hindsight of the mind. It is rather a realization, once in a while, you need to let it go, discard all the burden that has come as part of the memories that so form the foundation of our (everyone's) existence. The multitude of neurons in the head will take a few days if not weeks to flush the changes through to the deepest reaches of the synapses, but you have to start it somewhere and let it go, start afresh. I feel that need now more than ever, but this of all things in the world is something you cannot verify, still you just have to go with the feeling that your heart asked your mind to let it go and the grey blob paid heed, you will however never know for sure that it did or not until another day when all the melancholy is triggered [again] by that one memory you have tried to let go time and over  !

Converting paths on a linux filesystem to WINE relative paths

Wine is a windows software emulator for Linux. Wine generally installs itself to ~/.wine and the C:\ Drive for the corresponding emulated windows environment is ~/.wine/drive_c/ . Wine sees the actually filesystem as mounted under a emulated Z:\ drive, so for example /mnt on the linux filesystem becomes Z:\mnt. To automatically run windows programs which require files from the linux filesystem, it is nifty if the linux filesystem path for a file can be converted to the emulated version starting with Z:\ , here is a small snippet of code which does that for the current working directory :

$ echo Z:`pwd` | sed 's/\//\\/g'

It gets the absolute path for the current working directory and converts all forward slashes ('/') to backward slashes ('\') and prepends the drive letter (Z:) to it. Now adapting the above snippet to get the wine path for a normal file is quite easy, and is left for those who are interested